Introduction

ReadStor is a simple CLI for exporting user-generated data from Apple Books. The goal of this project is to facilitate data-migration from Apple Books to any other platform. Currently, Apple Books provides no simple way to do this. Exporting is possible but not ideal and often times truncates long annotations.

Version 0.1.x contained the core functionality: (1) save all annotations and notes as JSON (2) render them via a custom (or the default) template using the Tera syntax or (3) backup the current Apple Books databases.

Note that this repository is a heavy work-in-progress and things are bound to change.

Quickstart

The following is an overview of some of the more common options.

Default Directories

Running readstor with no arguments uses the following default directories:

• databases directory: ~/Library/Containers/com.apple.iBooksX/Data/Documents
• output directory: ~/.readstor

We can change the output directory using the --output-directory option:

readstor [COMMAND] --output-directory "/path/to/output"

And change the databases directory using the --databases-directory option:

readstor [COMMAND] --databases-directory "/path/to/databases"

Filter

Filters can be used to run commands on a subset of annotations. Note that all filters are case-insensitive.

We can filter annotations where the book's title is exactly the art spirit:

readstor [COMMAND] --filter "=title:the art spirit"
# '=' operator: exact -------^

Or, filter annotations where the book's author contains henri:

readstor [COMMAND] --filter "author:henri"
# no operator: contains ----^

Or, filter annotations that contain the tag #star:

readstor [COMMAND] --filter "tag:#star" --extract-tags

We can also combine filters:

readstor [COMMAND]                   \
    --filter "=title:the art spirit" \
    --filter "tag:#star"             \
    --extract-tags

Custom Templates

A custom templates directory can be declared by using the --templates-directory option. A template directory can contain any number of templates structured in any way, therefore template groups are used to name them and define relations between them.

Render a single template group found inside a custom templates directory:

readstor render                                \
    --templates-directory "/path/to/templates" \
    --template-group "my-template-group"

In this example, my-template-group refers not to any file or directory name but rather the group declared within any of the templates found inside the /path/to/templates directory.

For example /path/to/templates/my-template.jinja2:

<!-- readstor
group: my-template-group
context: book
structure: flat
extension: md
-->

# {{ book.author }} - {{ book.title }}

{% for annotation in annotations -%}
{{ annotation.body }}

{% endfor %}

Pre- and Post-processing

We can run pre- and post-processors on the data to modify it before completing the command. Pre-processors are run on the raw data while post-processors are run on the output data. Therefore pre-processors apply only to the render and export commands while post-processors only apply to the render command.

Generate easily editable text by normalizing whitespace, converting any "smart" Unicode symbols to ASCII and hard-wrapping the text to 100 characters.

readstor render            \
    --normalize-whitespace \
    --ascii-symbols        \
    --trim-blocks          \
    --wrap-text 100

See normalize whitespace, ascii symbols, trim blocks, wrap text for more information.

Complete Examples

Render

Renders all annotations containing the #star tag using a template named my-template-group found inside the /path/to/templates directory. Runs the pre-processes: ascii-symbols, extract-tags and normalize-whitespace, and the post-processes trim-blocks and wrap-text. The resulting files are output to /path/to/output.

readstor render                                \
    --output-directory "/path/to/output"       \
    --templates-directory "/path/to/templates" \
    --template-group "my-template-group"       \
    --filter "tag:#star"                       \
    --ascii-symbols                            \
    --extract-tags                             \
    --normalize-whitespace                     \
    --trim-blocks                              \
    --wrap-text 100

Export

Exports all annotations. Runs the pre-processes: ascii-symbols, extract-tags and normalize-whitespace. The resulting files are output to /path/to/output using a custom directory template.

readstor export                                                  \
    --output-directory "/path/to/output"                         \
    --directory-template "{{ book.title }} by {{ book.author }}" \
    --ascii-symbols                                              \
    --extract-tags                                               \
    --normalize-whitespace

Back-up

Backs-up macOS's Apple Books databases to /path/to/output using a custom directory template.

readstor backup                          \
    --output-directory "/path/to/output" \
    --directory-template "{{ now | date(format='%Y-%m-%d') }}-v{{ version }}"

Commands

render

Render data via templates.

See Templates for a full guide on creating templates.

See Pre-process, Post-process and Render options for available options.

export

Export data as JSON.

See Pre-process options for available options.

Outputs using the following structure:

[output-directory]
 ├── [author-title]
 │    ├── book.json
 │    └── annotations.json
 │
 ├── [author-title]
 │    └── ...
 └── ...

Example output structure:

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── annotations.json
 │   └── book.json
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── annotations.json
 │   └── book.json
 └── Robert Henri - The Art Spirit
     ├── annotations.json
     └── book.json

Example book.json:

{
  "title": "The Art Spirit",
  "author": "Robert Henri",
  "tags": [],
  "metadata": {
    "id": "1969AF0ECA8AE4965029A34316813924",
    "last_opened": "2021-11-02T18:27:04.781938076Z"
  },
  "slugs": {
    "title": "the-art-spirit",
    "author": "robert-henri"
  }
}

Example annotations.json:

[
  {
    "body": "We are not here to do what has already been done.",
    "style": "purple",
    "notes": "",
    "tags": [],
    "metadata": {
      "id": "C932CE69-8584-4555-834C-797DF84E6825",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:12:50.826642036Z",
      "modified": "2021-11-02T18:12:51.831905841Z",
      "location": "6.18.4.2.20.2.1:0",
      "epubcfi": "epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)",
      "slugs": {
        "created": "2021-11-02-181250",
        "modified": "2021-11-02-181250"
      }
    }
  },
  {
    "body": "The object of painting a picture...",
    "style": "yellow",
    "notes": "",
    "tags": ["#artist", "#being"],
    "metadata": {
      "id": "3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:13:25.905355930Z",
      "modified": "2021-11-02T18:14:12.444134950Z",
      "location": "6.24.4.2.296.2.1:0",
      "epubcfi": "epubcfi(/6/24[Part09_Split3]!/4/2/296/2,/1:0,/7:257)",
      "slugs": {
        "created": "2021-11-02-181325",
        "modified": "2021-11-02-181325"
      }
    }
  },
  {
    "body": "Of course it is not easy to go one’s road...",
    "style": "blue",
    "notes": "",
    "tags": [],
    "metadata": {
      "id": "9D1B71B1-895C-446F-A03F-50C01146F532",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:04:45.184863090Z",
      "modified": "2021-11-02T18:12:30.355533123Z",
      "location": "6.26.4.2.446.2.1:0",
      "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/446/2/1,:0,:679)",
      "slugs": {
        "created": "2021-11-02-180445",
        "modified": "2021-11-02-180445"
      }
    }
  },
  {
    "body": "Do not let the fact that things are not made for you...",
    "style": "green",
    "notes": "",
    "tags": ["#inspiration"],
    "metadata": {
      "id": "4620564A-0B64-4099-B5D6-6C9116A03AFF",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:15:10.700510978Z",
      "modified": "2021-11-02T18:15:20.879488945Z",
      "location": "6.26.4.2.636.2.1:0",
      "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/636/2/1,:0,:166)",
      "slugs": {
        "created": "2021-11-02-181510",
        "modified": "2021-11-02-181510"
      }
    }
  }
]

This export was run with the --extract-tags option.

backup

Back-up macOS's Apple Books databases.

Outputs using the following structure:

[output-directory]
 └── [YYYY-MM-DD-HHMMSS-VERSION]
      ├── AEAnnotation
      │   ├── AEAnnotation*.sqlite
      │   └── ...
      └── BKLibrary
          ├── BKLibrary*.sqlite
          └── ...

Example output:

[output-directory]
 └── 2022-10-09-152506-v4.4-5177
     ├── AEAnnotation
     │   ├── AEAnnotation_v10312011_1727_local.sqlite
     │   ├── AEAnnotation_v10312011_1727_local.sqlite-shm
     │   └── AEAnnotation_v10312011_1727_local.sqlite-wal
     └── BKLibrary
         ├── BKLibrary-1-091020131601.sqlite
         ├── BKLibrary-1-091020131601.sqlite-shm
         └── BKLibrary-1-091020131601.sqlite-wal

Options

Each of the three currently available Commands has its own pipeline for processing Apple Books' data before writing it to disk. And by extension, each pipeline has its own set of applicable options.

The pipelines and options for these three Commands are as follows:

╭─────────╮           Export Pipeline
│ Global* │          ╭────────╮ ╭─────────────╮ ╔════════╗
╰────┬────╯        ┌─┤ Filter ├─┤ Pre-process ├─╢ export ╟────────────────────┐
     │             │ ╰────────╯ ╰─────────────╯ ╚════════╝                    │
     │             │                                                          │
     │             │                                                          │
     │             │  Render Pipeline                                         │
     │             │ ╭───────────╮                                            │
     │             │ │ Templates ├──────────────┐                             │
     │             │ ╰───────────╯              │                             │
     │ ┌╌╌╌╌╌╌╌╌╌┐ │ ╭────────╮ ╭─────────────╮ │ ╔════════╗ ╭──────────────╮ │ ┌╌╌╌╌╌╌╌┐
     █─┤ Extract ├─┴─┤ Filter ├─┤ Pre-process ├─┴─╢ render ╟─┤ Post-process ├─┼─┤ Write │
     │ └╌╌╌╌╌╌╌╌╌┘   ╰────────╯ ╰─────────────╯   ╚════════╝ ╰──────────────╯ │ └╌╌╌╌╌╌╌┘
     │                                                                        │
     │                                                                        │
     │  Backup Pipeline                                                       │
     │ ╔════════╗                                                             │
     └─╢ backup ╟─────────────────────────────────────────────────────────────┘
       ╚════════╝

Global

The following options affect all Commands.

--output-directory <PATH>

Set the output directory for all Commands.

Default: ~/.readstor.

--databases-directory <PATH>

Set the directory containing macOS's Apple Books databases

Default: ~/Library/Containers/com.apple.iBooksX/Data/Documents

The databases directory should contain the databases for macOS's Apple Books. These databases are: AEAnnotation*.sqlite and BKLibrary*.sqlite. The directory should follow the following structure:

[databases-directory]
 │
 ├── AEAnnotation
 │   ├── AEAnnotation*.sqlite
 │   └── ...
 │
 ├── BKLibrary
 │   ├── BKLibrary*.sqlite
 │   └── ...
 └── ...

This can be useful when running ReadStor on databases backed-up with the backup command. Note that the backup command produces an output structure identical to this. So backing up and extracting data would require little effort.

--plists-directory <PATH>

Set the directory containing iOS's Apple Books plists

Experimental! Extracting data from Apple Books for iOS hasn't been tested as thoroughly as its macOS counterpart. Please submit an issue if you run into any.

The plists directory should contain the Books.plist and com.apple.ibooks-sync.plist. The directory should follow the following structure:

[plists-directory]
 │
 ├── Books.plist
 ├── com.apple.ibooks-sync.plist
 └── ...

See iOS - Library Location and iOS - Access Library on how to retrieve these files.

--force

Run even if Apple Books is currently running.

--quiet

Silence output messages.

Render

The following options affect only the render commands.

--templates-directory <PATH>

Set a custom templates directory.

See the default templates for fully working examples.

--template-group <GROUP>

Render specified Template Groups.

Passing non-existent Template Groups will return an error.

Multiple Template Groups can be passed using the following syntax.

readstor
    # ...
    --template-group basic
    --template-group using-backlinks
    # ..

Export

The following options affect only the export commands.

--directory-template <TEMPLATE>

Set the output directory template.

For example, using the default template, the non-rendered ouput structure would look like the following:

[ouput-directory]
 ├── {{ book.author }} - {{ book.title }}
 │    ├── book.json
 │    └── annotations.json
 │
 ├── {{ book.author }} - {{ book.title }}
 │    └── ...
 └── ...

And when rendered, the ouput structure would result in the following:

[ouput-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── annotations.json
 │   └── book.json
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── annotations.json
 │   └── book.json
 └── Robert Henri - The Art Spirit
     ├── annotations.json
     └── book.json

--overwrite-existing

Overwrite existing files.

By default, exising files are skipped.

Backup

The following options affect only the backup commands.

--directory-template <TEMPLATE>

Set the output directory template

Note that an escaping backslash \ was required to nest a pipe | inside a markdown table. In other words, the default value does not contain a backslash.

For example, using the default template, the non-rendered ouput structure would look like the following:

[ouput-directory]
 └── {{ now | date(format='%Y-%m-%d-%H%M%S') }}-{{ version }}
      ├── AEAnnotation
      │   ├── AEAnnotation*.sqlite
      │   └── ...
      └── BKLibrary
          ├── BKLibrary*.sqlite
          └── ...

And when rendered, the ouput structure would result in the following:

[ouput-directory]
 └── 2022-10-09-152506-v4.4-5177
     ├── AEAnnotation
     │   ├── AEAnnotation_v10312011_1727_local.sqlite
     │   ├── AEAnnotation_v10312011_1727_local.sqlite-shm
     │   └── AEAnnotation_v10312011_1727_local.sqlite-wal
     └── BKLibrary
         ├── BKLibrary-1-091020131601.sqlite
         ├── BKLibrary-1-091020131601.sqlite-shm
         └── BKLibrary-1-091020131601.sqlite-wal

Backup Context

Filter

The following options affect only the render and export commands.

--filter <[OP]{FIELD}:{QUERY}>

Filter books/annotations before outputting.

Filtering allows you to specify, to a certain degree, which books and/or annotations to output. Currently, this is available for the export and render commands.

For example, this filter would only render annotations where its respective book's title is exactly the art spirit AND their tags contain the #star tag.

readstor render                      \
    --filter "=title:the art spirit" \
    --filter "tag:#star"             \
    --extract-tags

This filter would export annotations where its respective book's author contains the string krishnamurti AND their tags contain either #star or #love.

readstor export                   \
    --filter "author:krishnmurti" \
    --filter "?tag:#star #love"   \
    --extract-tags

Note that filters are case-insensitive.

Filter Results

After all the filters are run, a confirmation prompt is shown with a brief summary of the filtered down books/annotations.

readstor render                      \
    --filter "=title:the art spirit" \
    --filter "tag:#star"             \
    --extract-tags
...
   ----------------------------------------------------------------
   Found 9 annotations from 2 books:
    • Think on These Things by Krishnamurti
    • The Art Spirit by Robert Henri
   ----------------------------------------------------------------
   Continue? [y/N]: █

This prompt can be auto-confirmed by passing the --auto-confirm-filter flag.

Filter Syntax

A filter consists of three parts: an optional operator, a field and a query. The syntax structure is as follows:

[operator]{field}:{query}

For example, breaking down the command from above:

readstor render
    --extract-tags
    --filter "=title:the art spirit"
              │└──┬┘ └───────────┬┘
              │   │              │
              │   │              └────────── query: the art spirit
              │   └───────────────────────── field: title
              └────────────────────────── operator: = (exact)
    --filter "tag:#star"
              └┬┘ └──┬┘
               │     │
               │     └────────────────────── query: #star
               └──────────────────────────── field: tag
                                operator (default): ? (any)

Operator

The operator token determines how matching will be handled against the query.

When a filter is processed, the query is split on its spaces to create its component queries. For example, the input string the art sprit turns into three parts: the, art and spirit, and depending on the operator these three parts are handled differently in order to determine if an annotation is filtered out or not.

Note that when searching for an exact match in the tags field i.e. =tags:[query], the query remains split and the set of tags in the query is compared to those in each annotation.

Field

The field token determines which field to run the filter on.

Currently, only three fields are supported:

Query

The query string determines what will be searched in the specified field. A query is a space delineated set of words where each word can potentially be a separate search term depending on the specified operator.

--auto-confirm-filter

Auto-confirm Filter Results.

Pre-process

The following options affect only the render and export commands.

--extract-tags

Extract #tags from annotation.notes.

All matches are removed from annotation.notes and placed into annotation.tags.

Tags must start with a hash symbol # followed by a letter [a-zA-Z] and then a series of any characters. A tag ends when a space or another # is encountered.

--normalize-whitespace

Normalize whitespace in annotation.body.

Trims whitespace and replaces all line-breaks with two consecutive line-breaks: \n\n.

--ascii-all

Convert all Unicode characters to ASCII.

All Unicode characters found in book.title, book.author and annotation.body are converted to ASCII.

--ascii-symbols

Convert "smart" Unicode symbols to ASCII.

"Smart" Unicode symbols found in book.title, book.author and annotation.body are converted to ASCII.

Characters and transliterations taken from:

• Daring Fireball - SmartyPants
• Python-Markdown - SmartyPants

Post-process

The following options affect only the render command.

--trim-blocks

Trim any blocks left after rendering.

Currently, this is a very naive implementation that mimics what tera might do if/when it adds trim_blocks. It is by no means smart and will just normalize whitespace regardless of what the template requested.

--wrap-text <WIDTH>

Wrap text to a maximum character width.

Maximum line length is not guaranteed as long words are not broken if their length exceeds the maximum. Hyphenation is not used, however, existing hyphen can be split on to insert a line-break.

This will naively wrap all the text inside a rendered file regardless its structure. Use with caution! Extremely low values may cause unexpected results. Values above 80 or so are recommended.

Templates

ReadStor's most powerful feature is the templating interface it provides for Apple Books. Templates can be designed to output almost any kind of text-based file format including: Markdown, HTML, CSV. PDFs are possible as well but would require an extra step to convert an HTML file to the final PDF.

See the default templates for fully working examples.

An Example Template

The following is an example template along with its expected output and output structure. In fact, it's almost identical to the default template that comes with ReadStor.

Template Syntax

Tera, a Jinja-flavored templating language, provides a rich set of tools for building a wide range of template complexities. The following is a brief intro to the syntax.

Values can be accessed by placing the desired attribute between double curly-braces:

{{ book.author }}

A list of items can be iterated over using a for loop:

{% for annotation in annotations %} ... {% endfor %}

Conditional behavior can be achieved by using if/else statements:

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif %}

Values can be modified with filters. Here, a list of tags is concatenated with a space as a delimiter.

{{ annotation.tags | join(sep=" ") }}

And finally comments can be added like so:

{# Hi! I'm a comment! #}

See Tera's documentation to learn more about these and more advanced features, including template inheritance, the include tag, macros and the full list of available filters.

Template

Templates consist of two main sections, the configuration block written in YAML (and some Tera) and the template body written in Tera.

The configuration for this example template describes that this template, grouped under the name my-vault, will render a single file for each book and place them all directly into the output directory. Each output filename will follow the pattern of Author - Title.md.

<!-- readstor
group: my-vault
context: book
structure: flat
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
-->

---
title: {{ book.title }}
author: {{ book.author }}
last-opened: {{ book.metadata.last_opened | date(format="%Y-%m-%dT%H:%M") }}
---

# {{ book.author }} - {{ book.title }}

{% for annotation in annotations -%}

---

{{ annotation.body }}

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif %}

{% endfor %}

Output Structure

The output structure is primarily determined by the structure and context keys. With the structure set to flat all the output files will be placed inside the output directory with no structure. With the context set to book, a single file will be created for each book and, if the template body requests, will contain all its respective annotations. See Context Modes and Structure Modes for more information.

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit.md

Output Rendered File

Krishnamurti - Think on These Things.md

---
title: Think on These Things
author: Krishnamurti
last-opened: 2021-11-02T18:30
---

# Krishnamurti - Think on These Things

---

Do you know what intelligence is? It is the capacity, surely, to think freely,
without fear, without a formula, so that you begin to discover for yourself what
is real, what is true; but if you are frightened you will never be intelligent.
Any form of ambition, spiritual or mundane, breeds anxiety, fear; therefore
ambition does not help to bring about a mind that is clear, simple, direct, and
hence intelligent.

tags: #education #intelligence #ambition #fear

---

To find out is not to come to a conclusion. I don’t know if you see the
difference. The moment you come to a conclusion as to what intelligence is, you
cease to be intelligent. That is what most of the older people have done: they
have come to conclusions. Therefore they have ceased to be intelligent. So you
have found out one thing right off: that an intelligent mind is one which is
constantly learning, never concluding.

tags: #learning #intelligence

---

The deeper the mind penetrates its own thought processes, the more clearly it
understands that all forms of thinking are conditioned; therefore the mind is
spontaneously very still—which does not mean that it is asleep. On the contrary,
the mind is then extraordinarily alert, no longer being drugged by mantrams, by
the repetition of words, or shaped by discipline. This state of silent alertness
is also part of awareness; and if you go into it still more deeply you will find
that there is no division between the person who is aware and the object of
which he is aware.

tags: #thinking

Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md

---
title: "Surely You're Joking, Mr. Feynman!"
author: Richard P. Feynman
last-opened: 2021-11-02T18:27
---

# Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"

---

After the dinner we went off into another room, where there were different
conversations going on. There was a Princess Somebody of Denmark sitting at a
table with a number of people around her, and I saw an empty chair at their
table and sat down.

She turned to me and said, “Oh! You’re one of the Nobel-Prize-winners. In what
field did you do your work?”

“In physics,” I said.

“Oh. Well, nobody knows anything about that, so I guess we can’t talk about it.”

“On the contrary,” I answered. “It’s because somebody knows something about it
that we can’t talk about physics. It’s the things that nobody knows anything
about that we can discuss. We can talk about the weather; we can talk about
social problems; we can talk about psychology; we can talk about international
finance—gold transfers we can’t talk about, because those are understood—so it’s
the subject that nobody knows anything about that we can all talk about!”

I don’t know how they do it. There’s a way of forming ice on the surface of the
face, and she did it! She turned to talk to somebody else.

Robert Henri - The Art Spirit.md

---
title: The Art Spirit
author: Robert Henri
last-opened: 2021-11-02T18:27
---

# Robert Henri - The Art Spirit

---

We are not here to do what has already been done.

---

The object of painting a picture is not to make a picture—however unreasonable
this may sound. The picture, if a picture results, is a by-product and may be
useful, valuable, interesting as a sign of what has past. The object, which is
back of every true work of art, is the attainment of a state of being, a state
of high functioning, a more than ordinary moment of existence. In such moments
activity is inevitable, and whether this activity is with brush, pen, chisel, or
tongue, its result is but a by-product of the state, a trace, the footprint of
the state.

tags: #artist #being

---

Of course it is not easy to go one’s road. Because of our education we
continually get off our track, but the fight is a good one and there is joy in
it if there is any success at all. After all, the goal is not making art. It is
living a life. Those who live their lives will leave the stuff that is really
art. Art is a result. It is the trace of those who have led their lives. It is
interesting to us because we read the struggle and the degree of success the man
made in his struggle to live. The great question is: “What is worth while?” The
majority of people have failed to ask themselves seriously enough, and have
failed to try seriously enough to answer this question.

---

Do not let the fact that things are not made for you, that conditions are not as
they should be, stop you. Go on anyway. Everything depends on those who go on
anyway.

tags: #inspiration

Configuration

A template's configuration describes both what the template expects to render and how the output structure and naming should be. Every template must start with a configuration block.

Overview

The configuration starts off as a basic HTML comment tag...

<!-- -->

... with the word readstor...

<!-- readstor -->

...and a new line.

<!-- readstor

-->

The YAML configuration is then placed inside the tag. For example:

<!-- readstor
group: my-vault
context: book
structure: nested
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"
-->

...

Note that the final rendered output file will not include its template's configuration. Additionally, if the configuration ended with trailing line-breaks, a single one of them is removed. This allows for some extra whitespace while working with a template without affecting final rendered output.

A quick rundown of each configuration key:

Template Groups

Groups are used to identify multiple templates that are intended to be part of a single output. Conversely, they also provide a way to separate unrelated templates rendered to the same output directory. The most common use case would be when using a pair of templates where one is used to render a book and the other to render each of its annotations separately.

Group names are sanitized to make sure they interact well with the file system. See String Sanitization for more information.

Grouping is triggered when one or more templates are set to either the flat-grouped or the nested-grouped Structure Mode. The output files are placed within a directory named after the group.

For example, if two templates share these values in their configurations:

group: my-vault
structure: flat-grouped

The output will be:

[output-directory]
 ├── my-vault
 │   ├── 2021-11-02-180445-the-art-spirit.md
 │   ├── 2021-11-02-181250-the-art-spirit.md
 │   ├── 2021-11-02-181325-the-art-spirit.md
 │   ├── 2021-11-02-181510-the-art-spirit.md
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   ├── Krishnamurti - Think on These Things.md
 │   ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 │   └── Robert Henri - The Art Spirit.md
 │
 ├── [other-group]
 │    └── ...
 └── ...

And, if two templates share these values in their configurations:

group: my-vault
structure: nested-grouped

The output will be:

[output-directory]
 ├── my-vault
 │   ├── Krishnamurti - Think on These Things
 │   │   ├── 2021-11-02-182319-think-on-these-things.md
 │   │   ├── 2021-11-02-182426-think-on-these-things.md
 │   │   ├── 2021-11-02-182543-think-on-these-things.md
 │   │   ├── 2021-11-02-182648-think-on-these-things.md
 │   │   ├── 2021-11-02-182805-think-on-these-things.md
 │   │   └── Krishnamurti - Think on These Things.md
 │   ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 │   └── Robert Henri - The Art Spirit
 │       ├── 2021-11-02-180445-the-art-spirit.md
 │       ├── 2021-11-02-181250-the-art-spirit.md
 │       ├── 2021-11-02-181325-the-art-spirit.md
 │       ├── 2021-11-02-181510-the-art-spirit.md
 │       └── Robert Henri - The Art Spirit.md`
 │
 ├── [other-group]
 │    └── ...
 └── ...

Context Modes

At render time, each template is injected with a "context", in other words, the data it will render. ReadStor provides two different context modes: book and annotation. The context mode dictates not just the data within the context but also changes the number of output files. See A Note On Output Structure for more information.

The Book Context

Note that annotations is plural in the book context.

When selected, a single file is rendered out from a context containing the data from a single book and its annotations. For example, represented here in YAML:

book:
  title: The Art Spirit
  author: Robert Henri
  metadata:
    id: 1969AF0ECA8AE4965029A34316813924
    last_opened: 2021-11-02T18:27:04.781938076Z
  slugs:
    title: the-art-spirit
    author: robert-henri
annotations:
  - body: We are not here to do what has already been done.
    style: purple
    notes: ""
    tags: []
    metadata:
      id: C932CE69-8584-4555-834C-797DF84E6825
      book_id: 1969AF0ECA8AE4965029A34316813924
      created: 2021-11-02T18:12:50.826642036Z
      modified: 2021-11-02T18:12:51.831905841Z
      location: 6.18.4.2.20.2.1:0
      epubcfi: epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)
      slugs:
        created: 2021-11-02-181250
        modified: 2021-11-02-181250
  - body: The object of painting a picture...
    style: yellow
    notes: ""
    tags:
      - "#artist"
      - "#being"
    metadata:
      id: 3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C
      book_id: 1969AF0ECA8AE4965029A34316813924
      created: 2021-11-02T18:13:25.905355930Z
      modified: 2021-11-02T18:14:12.444134950Z
      location: 6.24.4.2.296.2.1:0
      epubcfi: epubcfi(/6/24[Part09_Split3]!/4/2/296/2,/1:0,/7:257)
      slugs:
        created: 2021-11-02-181325
        modified: 2021-11-02-181325
  # ...
names:
  book: Robert Henri - The Art Spirit.md
  annotations:
    C932CE69-8584-4555-834C-797DF84E6825: 2021-11-02-181250-the-art-spirit.md
    3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C: 2021-11-02-181325-the-art-spirit.md
    # ...
  directory: Robert Henri - The Art Spirit

See Context Reference - Book for more information.

The Annotation Context

When selected, multiple files are rendered out from a context containing the data from a single annotation and its respective book. For example, represented here in YAML:

book:
  title: The Art Spirit
  author: Robert Henri
  metadata:
    id: 1969AF0ECA8AE4965029A34316813924
    last_opened: 2021-11-02T18:27:04.781938076Z
  slugs:
    title: the-art-spirit
    author: robert-henri
annotation:
  body: We are not here to do what has already been done.
  style: purple
  notes: ""
  tags: []
  metadata:
    id: C932CE69-8584-4555-834C-797DF84E6825
    book_id: 1969AF0ECA8AE4965029A34316813924
    created: 2021-11-02T18:12:50.826642036Z
    modified: 2021-11-02T18:12:51.831905841Z
    location: 6.18.4.2.20.2.1:0
    epubcfi: epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)
    slugs:
      created: 2021-11-02-181250
      modified: 2021-11-02-181250
names:
  book: Robert Henri - The Art Spirit.md
  annotations:
    C932CE69-8584-4555-834C-797DF84E6825: 2021-11-02-181250-the-art-spirit.md
  directory: Robert Henri - The Art Spirit

See Context Reference - Annotation for more information.

A Note On Output Structure

When selecting a context mode it's important to understand how the output files will look. The following is a quick visualization.

When the context mode is set to book, a single file for each book is rendered out from the template. Using the following configuration keys:

context: book
structure: nested

The output structure is as follows:

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit
     └── Robert Henri - The Art Spirit.md`

And when the context mode is set to annotation, multiple files are rendered out, one for each annotation within a book. Using the following configuration keys:

context: annotation
structure: nested

The output structure is as follows:

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   └── 2021-11-02-182805-think-on-these-things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   └── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 └── Robert Henri - The Art Spirit
     ├── 2021-11-02-180445-the-art-spirit.md
     ├── 2021-11-02-181250-the-art-spirit.md
     ├── 2021-11-02-181325-the-art-spirit.md
     └── 2021-11-02-181510-the-art-spirit.md

When both templates are rendered to the same directory, we get a complete output with a book and each of its annotations all rendered out to their own file.

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit
     ├── 2021-11-02-180445-the-art-spirit.md
     ├── 2021-11-02-181250-the-art-spirit.md
     ├── 2021-11-02-181325-the-art-spirit.md
     ├── 2021-11-02-181510-the-art-spirit.md
     └── Robert Henri - The Art Spirit.md

Structure Modes

The structure mode determines how the output directories and files are structured. ReadStor provides four structure modes: flat, flat-grouped, nested and nested-grouped.

Flat Mode

structure: flat

When selected, the template is rendered to the output directory without any structure. All the files are placed directly within the output directory. No additional directories are created.

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit.md
 └── ...

Flat & Grouped Mode

group: my-vault
structure: flat-grouped

When selected, the template is rendered to the output directory and placed inside a directory named after its group. This is useful if multiple template groups are being rendered to the same directory.

[output-directory]
 └── my-vault
     ├── Krishnamurti - Think on These Things.md
     ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
     ├── Robert Henri - The Art Spirit.md
     └── ...

Nested Mode

structure: nested

When selected, the template is rendered to the output directory and placed inside a directory named after the names.directory key. This is useful if a template group contains multiple templates.

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit
 │   ├── 2021-11-02-180445-the-art-spirit.md
 │   ├── 2021-11-02-181250-the-art-spirit.md
 │   ├── 2021-11-02-181325-the-art-spirit.md
 │   ├── 2021-11-02-181510-the-art-spirit.md
 │   └── Robert Henri - The Art Spirit.md`
 └── ...

Nested & Grouped Mode

group: my-vault
structure: nested-grouped

When selected, the template is rendered to the output directory and placed inside a directory named after its group and another named after the names.directory key. This is useful if multiple template groups are being rendered to the same directory and if a template group contains multiple templates.

[output-directory]
 └── my-vault
     ├── Krishnamurti - Think on These Things
     │   ├── 2021-11-02-182319-think-on-these-things.md
     │   ├── 2021-11-02-182426-think-on-these-things.md
     │   ├── 2021-11-02-182543-think-on-these-things.md
     │   ├── 2021-11-02-182648-think-on-these-things.md
     │   ├── 2021-11-02-182805-think-on-these-things.md
     │   └── Krishnamurti - Think on These Things.md
     ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
     │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
     │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
     └── Robert Henri - The Art Spirit
         ├── 2021-11-02-180445-the-art-spirit.md
         ├── 2021-11-02-181250-the-art-spirit.md
         ├── 2021-11-02-181325-the-art-spirit.md
         ├── 2021-11-02-181510-the-art-spirit.md
         └── Robert Henri - The Art Spirit.md

Extension

Defines the template's output file extension. Note that this will be appended to all the output filenames with the format [filename].[extension].

Names

Output files and directory names can be customized using the same Tera syntax. ReadStor will inject a different context into each names during render time and set the template's output file/directory name to the resulting string.

The rendered book, annotation and directory names are sanitized to make sure they interact well with the file system. See String Sanitization for more information.

Additionally, all the rendered values from these keys are available within the template's context under the names variable regardless of the current context mode. See ContextReference - Names for more information.

Note that the template's context matters when setting names.

For example, if the template's context is set to book:

group: my-vault
structure: flat
context: book # <- Here!
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"

ReadStor will generate a single file for each book and name them using the rendered result of names.book. The resulting output would be:

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit.md
 └── ...

However, if the context is changed to annotation:

group: my-vault
structure: flat
context: annotation # <- Here!
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"

ReadStor will now generate a single file for each annotation in each book and name them using the rendered result of names.annotation. The resulting output would be:

[output-directory]
 ├── 2021-11-02-180445-the-art-spirit.md
 ├── 2021-11-02-181250-the-art-spirit.md
 ├── 2021-11-02-181325-the-art-spirit.md
 ├── 2021-11-02-181510-the-art-spirit.md
 ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 ├── 2021-11-02-182319-think-on-these-things.md
 ├── 2021-11-02-182426-think-on-these-things.md
 ├── 2021-11-02-182543-think-on-these-things.md
 ├── 2021-11-02-182648-think-on-these-things.md
 ├── 2021-11-02-182805-think-on-these-things.md
 └── ...

Book Names

Defines the filename template to use when the parent template's context mode is set to book. This template only has access to the book context when it's rendered.

Annotation Names

Defines the filename template to use when the parent template's context mode is set to annotation. This template has access to the book and annotation context when its rendered.

Directory Names

Defines the directory name template to use when the parent template's structure mode is set to nested or the nested-grouped. This template only has access to the book context when its rendered.

Limitations

Why does a single template have both a names.book and names.annotation key?

This is mainly the result of a current limitation with ReadStor. Templates that are part of the same group have no relation internally. ReadStor just renders each template it finds. If multiple templates share the same value for group then they end up in the same directory when the structure mode is set to grouped or nested-grouped.

As a result of this limitation, if a template requires awareness of its group's sibling names i.e. the value defined in the names field, they must be defined in each template. This results in some duplication across templates. This should hopefully be resolved in future versions of ReadStor.

For more information on how to generate and use names see Context Reference - Names and Backlinks.

Partial Templates

When working with larger templates it can be helpful to separate them into smaller building blocks. One way to do this is to create partial templates and include them in other templates.

Partial templates differ in two important ways from their normal template counterparts.

1. They must begin with an underscore _ as this is currently the only indicator of whether a template is partial or not.

2. They require no configuration block as they inherit their configuration from the template that includes them.

See the using-partials templates for a fully working example.

Including a Partial Template

Partial templates are included using the include tag:

{% include "_my-partial-template.md" %}

Where "_my-partial-template.md" is a path relative to the templates directory. Therefore, this would be pointing to a template at the root of the templates directory. If we wanted to organize our templates into different directories we would have to add the directory names for any directories between the including template and the templates directory.

For example, with the following structure:

[templates-directory]
 └── my-vault-templates
     ├── _annotation.md
     ├── _book.md
     └── template.md

We would use the following include tags:

{% include "my-vault-templates/_book.jinja2" %}
{% include "my-vault-templates/_annotation.jinja2" %}

See the documentation for Tera's include tag for more information on its features and limitations.

Backlinks

Generating backlinks for a Zettelkasten-like note-taking experience is relatively easy with ReadStor. It requires two separate templates: One for rendering the book and one for rendering each of its annotations.

See the using-backlinks templates for a fully working example.

Template Configuration

First, let's define the configurations for our two templates. They should be almost identical to one another.

• group is set to the same value across the two templates. This makes sure that if we select a grouped Structure Mode, the templates will be rendered to the same directory.

• context is set to book for the book template and annotation for the annotation template.

• structure can be set to any of the four modes, however, nested-grouped feels the most appropriate as it would place each book into its own directory and place them all under a directory named after the value for the group key. See Structure Modes for more information.

• extension is set to md as the template will be outputting Markdown.

• names can be set to anything as long as the values are identical between the two templates. It might seem odd to see the values for names duplicated across the two templates. Shouldn't the book template define names.book and the annotation template define names.annotation? Ideally, yes. This need for duplication is the result of a current limitation of ReadStor, therefore they must be identical, so the backlinks are correctly generated.

Book Template Configuration

group: my-vault
context: book # <- The only difference!
structure: nested-grouped
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

Annotation Template Configuration

group: my-vault
context: annotation # <- The only difference!
structure: nested-grouped
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

Template Body

With our configuration all set up, we can now use the names object, which contains all the rendered Names, to link between our rendered output files.

See Context Reference - Names for more information.

Book Template Body

The names.annotations object is a list of dictionaries, where each dictionary refers to a rendered annotation file and contains its filename along with metadata about its respective annotation. This allows us to link back to the annotation and sort the links based of off different criteria. In the example below, Tera's sort filter is used with the attribute argument and the location attribute.

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations | sort(attribute="location")-%}
![[{{ name.filename }}]]
{% endfor %}

Alternatively we can use the names.directory variable to access the rendered name of the parent directory. This value is only available if the Structure Mode is set to nested or nested-grouped.

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations -%}
![[{{ name.directory }}/{{ name.filename }}]]
{% endfor %}

Annotation Template Body

Finally, using the names.book variable we're able to link back to the source book.

# [[{{ names.book }}]]

{{ annotation.body }}

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif -%}

Output Structure

[output-directory]
 ├── my-vault
 │   └── Robert Henri - The Art Spirit
 │       ├── 2021-11-02-180445-the-art-spirit.md
 │       ├── 2021-11-02-181250-the-art-spirit.md
 │       ├── 2021-11-02-181325-the-art-spirit.md
 │       ├── 2021-11-02-181510-the-art-spirit.md
 │       └── Robert Henri - The Art Spirit.md
 │
 ├── [group]
 │    └── ...
 └── ...

Output Rendered Files

Robert Henri - The Art Spirit.md

# Robert Henri - The Art Spirit

![[2021-11-02-180445-the-art-spirit.md]]
![[2021-11-02-181250-the-art-spirit.md]]
![[2021-11-02-181325-the-art-spirit.md]]
![[2021-11-02-181510-the-art-spirit.md]]

2021-11-02-180445-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

Of course it is not easy to go one’s road. Because of our education we
continually get off our track, but the fight is a good one and there is joy in
it if there is any success at all. After all, the goal is not making art. It is
living a life. Those who live their lives will leave the stuff that is really
art. Art is a result. It is the trace of those who have led their lives. It is
interesting to us because we read the struggle and the degree of success the
man made in his struggle to live. The great question is: “What is worth while?”
The majority of people have failed to ask themselves seriously enough, and have
failed to try seriously enough to answer this question.

2021-11-02-181250-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

We are not here to do what has already been done.

2021-11-02-181325-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

The object of painting a picture is not to make a picture—however unreasonable
this may sound. The picture, if a picture results, is a by-product and may be
useful, valuable, interesting as a sign of what has past. The object, which is
back of every true work of art, is the attainment of a state of being, a state
of high functioning, a more than ordinary moment of existence. In such moments
activity is inevitable, and whether this activity is with brush, pen, chisel,
or tongue, its result is but a by-product of the state, a trace, the footprint
of the state.

tags: #artist #being

2021-11-02-181510-the-art-spirit.m

# [[Robert Henri - The Art Spirit.md]]

Do not let the fact that things are not made for you, that conditions are not
as they should be, stop you. Go on anyway. Everything depends on those who go
on anyway.

tags: #inspiration

String Sanitization

To avoid any unexpected behavior, certain user-provided strings are sanitized before they are used for naming files or directories. Characters are either removed or replaced with an underscore _.

This following values are sanitized:

• Template Group set in the group config key.
• The rendered values from Names set in the names config key.

Context Reference

Every template is injected with a "context" i.e. the data currently available to rendering. ReadStor injects three different objects into every template context: book, annotation (or annotations depending on the Context Mode) and names.

Book

A single book object is injected into all template contexts regardless of the template's Context Mode.

Template Fields - Book

Example Data - Book

{
  "title": "The Art Spirit",
  "author": "Robert Henri",
  "tags": ["#artist", "#being", "#inspiration"],
  "metadata": {
    "id": "1969AF0ECA8AE4965029A34316813924",
    "last_opened": "2021-11-02T18:27:04.781938076Z"
  },
  "slugs": {
    "title": "the-art-spirit",
    "author": "robert-henri"
  }
}

Example Template - Book

---
title: {{ book.title }}
author: {{ book.author }}
id: {{ book.metadata.id }}
last-opened: {{ book.metadata.last_opened | date(format="%Y-%m-%d-%H:%M") }}
---

Here Tera's date filter is used to format a datetime object into a human-readable date.

Annotation

The number of annotation objects injected into a template depends on the template's Context Mode. When the Context Mode is set to annotation, a single annotation object is injected into the template's context. When the Context Mode is set to book, multiple annotation objects, under the name annotations, are injected into the template's context.

Template Fields - Annotation

Example Data - Annotation

{
  "body": "Of course it is not easy to go one’s road...",
  "style": "blue",
  "notes": "",
  "tags": [],
  "metadata": {
    "id": "9D1B71B1-895C-446F-A03F-50C01146F532",
    "book_id": "1969AF0ECA8AE4965029A34316813924",
    "created": "2021-11-02T18:04:45.184863090Z",
    "modified": "2021-11-02T18:12:30.355533123Z",
    "location": "6.26.4.2.446.2.1:0",
    "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/446/2/1,:0,:679)",
    "slugs": {
      "created": "2021-11-02-180445",
      "modified": "2021-11-02-180445"
    }
  }
}

Example Template - Annotation

{% for annotation in annotations -%}

---

{{ annotation.body }}

{%- if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif %}

{% endfor %}

Here Tera's join filter is used to join an array of items into a space-separated string.

Names

A single names object is injected into all template contexts regardless of the template's Context Mode. These contain all the file and directory names rendered from the names key in the template's config. See Names for more information.

Template Fields - Names

The names.annotations object is a list of dictionaries, where each dictionary refers to a rendered annotation file and contains its filename along with metadata about its respective annotation. Each dictionary consists of the following attributes:

These attributes allow the sorting of the names.annotations list using Tera's sort filter. See Backlinks for example usage.

Example Data - Names

With the following names configuration:

names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

{
  "book": "Robert Henri - The Art Spirit.md",
  "annotations": [
    {
      "filename": "2021-11-02-181510-the-art-spirit.md",
      "created": "2021-11-02T18:15:10.700510978Z",
      "modified": "2021-11-02T18:15:20.879488945Z",
      "location": "6.26.4.2.636.2.1:0"
    },
    {
      "filename": "2021-11-02-180445-the-art-spirit.md",
      "created": "2021-11-02T18:04:45.184863090Z",
      "modified": "2021-11-02T18:12:30.355533123Z",
      "location": "6.26.4.2.446.2.1:0"
    },
    {
      "filename": "2021-11-02-181325-the-art-spirit.md",
      "created": "2021-11-02T18:13:25.905355930Z",
      "modified": "2021-11-02T18:14:12.444134950Z",
      "location": "6.24.4.2.296.2.1:0"
    },
    {
      "filename": "2021-11-02-181250-the-art-spirit.md",
      "created": "2021-11-02T18:12:50.826642036Z",
      "modified": "2021-11-02T18:12:51.831905841Z",
      "location": "6.18.4.2.20.2.1:0"
    }
  ],
  "directory": "Robert Henri - The Art Spirit"
}

Example Template - Names

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations | sort(attribute="location") -%}
![[{{ name.filename }}]]
{% endfor %}

Apple Books

The following section assumes that Apple Books syncing with iCloud Drive is disabled!

macOS

macOS - Library Location

The following information assumes that Apple Books syncing with iCloud Drive is disabled!

Apple Books for macOS stores its data in two directories within ~/Library/Containers/.

Assets

EPUBs, PDFs, Audiobooks, etc. are stored in:

~/Library/Containers/com.apple.BKAgentService

Databases

The books and annotations databases are stored in:

~/Library/Containers/com.apple.iBooksX

The books database is located at:

~/Library/Containers/com.apple.iBooksX/Data/Documents/BKLibrary/BKLibrary***.sqlite

The annotations database is located at:

~/Library/Containers/com.apple.iBooksX/Data/Documents/AEAnnotation/AEAnnotation***.sqlite

Note that the database names will vary therefore *** is used in the filenames here.

macOS - Archive/Restore Library

Complete archive and restore scripts are available in scripts/apple-books.

What To Archive/Restore?

Apple Books for macOS stores its data in two locations:

~/Library/Containers/com.apple.BKAgentService
~/Library/Containers/com.apple.iBooksX

See macOS - Library Location for more information.

However, archiving and restoring only these directories might miss some metadata. Searching ~/Library/Containers for anything that contains Books yields some other directories:

find ~/Library/Containers -type d -name "*Books*"

./com.apple.BKAgentService/Data/Documents/iBooks
./com.apple.BKAgentService/Data/Documents/iBooks/Books
./com.apple.iBooksX
./com.apple.iBooksX/Data/Documents/BCCloudKit-iBooks
./com.apple.iBooksX/Data/Documents/BCCloudAsset-iBooks
./com.apple.iBooksX/Data/Documents/BKBookstore
./com.apple.iBooksX/Data/Library/Caches/com.apple.iBooksX
./com.apple.iBooksX.CacheDelete
./com.apple.iBooksX.DiskSpaceEfficiency
./com.apple.iBooksX.SharingExtension
./com.apple.iBooksX-SecureUserDefaults

Using BK as a proxy for Books yields no additional directories:

find ~/Library/Containers -type d -name "*BK*"

./com.apple.BKAgentService
./com.apple.iBooksX/Data/Documents/BKSeriesDatabase
./com.apple.iBooksX/Data/Documents/BKLibraryDataSourceDevelopment
./com.apple.iBooksX/Data/Documents/BKLibrary
./com.apple.iBooksX/Data/Documents/BKBookstore

So it's safe to assume any directory starting with com.apple.Books or com.apple.BK is important. Therefore, these two globs along with Apple Books' Group Container should be used for archiving and restoring:

"~/Library/Containers/com.apple.BK*"
"~/Library/Containers/com.apple.iBooks*"
"~/Library/Group Containers/group.com.apple.iBooks"

Important Note

Archiving/restoring will work only if the path to the username has not changed since the library was archived. Doing a few searches shows that the username has been hard-coded into some files. This is most evident in the BKLibrary-*.sqlite file which contains absolute paths to library files.

For example:

find "~/Library/Containers/com.apple.BKAgentService" \
    -type f -a -exec grep -l --exclude=\*.{htm,html,xhtml} $USER {} +

.../.com.apple.containermanagerd.metadata.plist
.../Container.plist
.../Data/Documents/iBooks/Books/Books.plist

find "~/Library/Containers/com.apple.iBooksX*" \
    -type f -a -exec grep -l --exclude=\*.{htm,html,xhtml} $USER {} +

.../com.apple.iBooksX/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX/Container.plist
.../com.apple.iBooksX/Data/Documents/BKLibrary/BKLibrary-1-091020131601.sqlite-wal
.../com.apple.iBooksX/Data/Documents/BKLibrary/BKLibrary-1-091020131601.sqlite
.../com.apple.iBooksX-SecureUserDefaults/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX-SecureUserDefaults/Container.plist
.../com.apple.iBooksX.CacheDelete/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.CacheDelete/Container.plist
.../com.apple.iBooksX.DiskSpaceEfficiency/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.DiskSpaceEfficiency/Container.plist
.../com.apple.iBooksX.SharingExtension/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.SharingExtension/Container.plist

Archive Library

The following snippet is taken from scripts/apple-books/macos-archive.sh.

Archiving the library is as simple as running two rsync commands. This should save all the relevant Apple Books data and metadata to a single directory. Make sure to replace [PATH-TO-ARCHIVE] with a valid path to said directory.

rsync                                            \
    --archive                                    \
    --extended-attributes                        \
    "$HOME/Library/Containers/com.apple.BK*"     \
    "$HOME/Library/Containers/com.apple.iBooks*" \
    "[PATH-TO-ARCHIVE]/Containers"

rsync                                                       \
    --archive                                               \
    --extended-attributes                                   \
    "$HOME/Library/Group Containers/group.com.apple.iBooks" \
    "[PATH-TO-ARCHIVE]/Group Containers"

For example, if [PATH-TO-ARCHIVE] is:

~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6

Our rsync commands would be:

rsync                                            \
    --archive                                    \
    --extended-attributes                        \
    "$HOME/Library/Containers/com.apple.BK*"     \
    "$HOME/Library/Containers/com.apple.iBooks*" \
    "~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Containers"

rsync                                                       \
    --archive                                               \
    --extended-attributes                                   \
    "$HOME/Library/Group Containers/group.com.apple.iBooks" \
    "~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Group Containers"

And the resulting archive would resemble:

~/archives
  └── 2022-10-08--apple-books-v4.4-5177--macos-v12.6
      ├── Containers
      │   ├── com.apple.BKAgentService
      │   ├── com.apple.iBooks.BooksNotificationContentExtension
      │   ├── com.apple.iBooks.engagementExtension
      │   ├── com.apple.iBooks.iBooksSpotlightExtension
      │   ├── com.apple.iBooksX
      │   ├── com.apple.iBooksX-SecureUserDefaults
      │   ├── com.apple.iBooksX.BooksThumbnail
      │   ├── com.apple.iBooksX.CacheDelete
      │   ├── com.apple.iBooksX.DiskSpaceEfficiency
      │   └── com.apple.iBooksX.SharingExtension
      └── Group Containers
          └── group.com.apple.iBooks

Restore Library

The following snippets are taken from scripts/apple-books/macos-restore.sh.

When restoring a library make sure that the current version of Apple Books for macOS and the version the archive was created from are identical. Not doing so could lead to unexpected results.

Restoring the library takes an extra step. First we need to clear out the current Apple Books library. We can delete all the library files and directories by using the paths we determined from What To Archive/Restore?.

rm -rf "$HOME/Library/Containers/com.apple.BK*"
rm -rf "$HOME/Library/Containers/com.apple.iBooks*"
rm -rf "$HOME/Library/Group Containers/group.com.apple.iBooks"

Finally, we can run the reverse rsync commands and restore the archive we previously made. Make sure to replace [PATH-TO-ARCHIVE] with a valid path.

rsync                               \
    --archive                       \
    --extended-attributes           \
    "[PATH-TO-ARCHIVE]/Containers/" \
    "$HOME/Library/Containers/"

rsync                                     \
    --archive                             \
    --extended-attributes                 \
    "[PATH-TO-ARCHIVE]/Group Containers/" \
    "$HOME/Library/Group Containers/"

The trailing forward-slash after Containers and Group Containers here is important. It tells rsync to move the archive directory's contents into the target. Otherwise, it would move the archive directory into the target.

For example, if [PATH-TO-ARCHIVE] is:

~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6

Our rsync command would be:

rsync                                                                       \
    --archive                                                               \
    --extended-attributes                                                   \
    "~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Containers/" \
    "$HOME/Library/Containers/"  # Note the forward-slash! --------------^

rsync                                                                             \
    --archive                                                                     \
    --extended-attributes                                                         \
    "~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Group Containers/" \
    "$HOME/Library/Group Containers/"  # Note the forward-slash! --------------^

iOS

iOS - Library Location

The following information assumes that Apple Books syncing with iCloud Drive is disabled!

Apple Books for iOS stores its data in the Books directory on an iOS device.

[ios-device]
 │
 ├── Books
 │   ├── Managed/
 │   ├── MetadataStore/
 │   ├── Purchases/
 │   ├── Sync/
 │   ├── Books.plist ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ Books data
 │   ├── com.apple.ibooks-sync.plist ╌╌╌╌╌╌╌╌╌╌ Annotations data
 │   ├── 376FAA7E4CF81729.epub ╌╌╌╌╌╌╌╌╌╌╌╌╌┐
 │   ├── 669FEE1FFBB29D81.epub              ├╌╌ EPUBs
 │   ├── F788455723912C6D.epub ╌╌╌╌╌╌╌╌╌╌╌╌╌┘
 │   └── ...
 └── ...

Both the EPUBs and plists are stored in:

[ios-device]/Books

The books plist is located at:

[ios-device]/Books/Books.plist

The annotations plist is located at:

[ios-device]/Books/com.apple.ibooks-sync.plist

See iOS - Access Library for more information.

iOS - Access Library

There are roughly two different methods for accessing iOS's Apple Books library. A (possibly-paid) third party application or manually mounting the device with macFUSE / ifuse.

Access via Third-party Applications

There are a number of third-party applications that grant access to an iOS device's filesystem. For the most part, they all expose the same directories, however, each might differ slightly in how they are represented in the GUI.

The following is a non-exhaustive list of paid software and the library's location within them:

iMazing - https://imazing.com

iExplorer - https://macroplant.com/iexplorer

Access via macFUSE / ifuse

The free and open-source option requires a bit of work during installation but is effortless to run once complete.

Note that this has only been tested on Apple Silicon running macOS Ventura.

Requirements

1. Install Homebrew. See here for installation instructions.

2. Install macFUSE. See here for installation instructions as macFUSE requires enabling support for third party kernel extensions and will require a restart.

3. Install ifuse

   brew install gromgit/fuse/ifuse-mac
   

   Running brew install ifuse is broken as of 2023-02-05.

Mount the Device

First we create a mount point. This can be anywhere and named anything.

mkdir /tmp/my-device

Mount a Single Device

If only a single device is connected, it can be mounted by simply running:

ifuse /tmp/my-device

Mount Multiple Devices

If there are multiple devices connected, a specific one can be mounted using its serial number. Assuming the device is an iPhone, run the following command:

system_profiler SPUSBDataType -detailLevel mini | grep -e iPhone -e Serial

Something similar to the following will print out. The 24-character string is the device's serial number.

iPhone:
  Serial Number: XXXXXXXXXXXXXXXXXXXXXXXX
  Serial Number: 000000000000

To convert it to a UDID add a hyphen after the eight character.

XXXXXXXX-XXXXXXXXXXXXXXXX

Finally, to mount the device, pass it to the command after the --udid flag to mount it:

ifuse /tmp/my-device --udid XXXXXXXX-XXXXXXXXXXXXXXXX

Troubleshooting

ifuse might complain: Invalid device UDID specified, length needs to be 40 characters. Starting with the iPhone X, Apple changed UDIDs to use 24 bytes > and a dash (-) instead of the old 40-byte format.

If you get Failed to connect to lockdownd service on the device. Try again. If it still fails try rebooting your device. ensure that your device is connected, and isn't displaying a "Trust this computer" dialog. You'll need to approve that first. If you then get ERROR: Device 000000000000000000000000 returned unhandled error code -13 you'll need to disconnect and reconnect the device.

via https://reincubate.com/support/how-to/mount-iphone-files/

Sources

• https://github.com/RhetTbull/osxphotos/issues/745
• https://reincubate.com/support/how-to/mount-iphone-files/

iOS - Archive/Restore Library

Archiving and restoring iOS's Apple Books library is fairly straightforward. All the relevant data exists in a single folder. A simple drag/drop, copy/paste or zip/unzip should suffice.

When restoring a library make sure that the current version of iOS and the version the archive was created from are identical. Not doing so could lead to unexpected results.

See iOS - Library Location and iOS - Access Library for more information.