Introduction

ReadStor is a simple CLI for exporting user-generated data from Apple Books. The goal of this project is to facilitate data-migration from Apple Books to any other platform. Currently, Apple Books provides no simple way to do this. Exporting is possible but not ideal and often times truncates long annotations.

Version 0.1.x contained the core functionality: (1) save all annotations and notes as JSON (2) render them via a custom (or the default) template using the Tera syntax or (3) backup the current Apple Books databases.

Note that this repository is a heavy work-in-progress and things are bound to change.

Commands

render

Render Apple Books' data via templates.

See Templates for a full guide on creating templates.

See Pre-process, Post-process and Render options for available options.

export

Export Apple Books' data as JSON.

See Pre-process options for available options.

Outputs using the following structure:

[ouput-directory]
 ├── [author-title]
 │    ├── book.json
 │    └── annotations.json
 │
 ├── [author-title]
 │    └── ...
 └── ...

Example output structure:

[ouput-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── annotations.json
 │   └── book.json
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── annotations.json
 │   └── book.json
 └── Robert Henri - The Art Spirit
     ├── annotations.json
     └── book.json

Example book.json:

{
  "title": "The Art Spirit",
  "author": "Robert Henri",
  "tags": [],
  "metadata": {
    "id": "1969AF0ECA8AE4965029A34316813924",
    "last_opened": "2021-11-02T18:27:04.781938076Z"
  },
  "slugs": {
    "title": "the-art-spirit",
    "author": "robert-henri"
  }
}

Example annotations.json:

[
  {
    "body": "We are not here to do what has already been done.",
    "style": "purple",
    "notes": "",
    "tags": [],
    "metadata": {
      "id": "C932CE69-8584-4555-834C-797DF84E6825",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:12:50.826642036Z",
      "modified": "2021-11-02T18:12:51.831905841Z",
      "location": "6.18.4.2.20.2.1:0",
      "epubcfi": "epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)",
      "slugs": {
        "created": "2021-11-02-181250",
        "modified": "2021-11-02-181250"
      }
    }
  },
  {
    "body": "The object of painting a picture...",
    "style": "yellow",
    "notes": "",
    "tags": ["#artist", "#being"],
    "metadata": {
      "id": "3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:13:25.905355930Z",
      "modified": "2021-11-02T18:14:12.444134950Z",
      "location": "6.24.4.2.296.2.1:0",
      "epubcfi": "epubcfi(/6/24[Part09_Split3]!/4/2/296/2,/1:0,/7:257)",
      "slugs": {
        "created": "2021-11-02-181325",
        "modified": "2021-11-02-181325"
      }
    }
  },
  {
    "body": "Of course it is not easy to go one’s road...",
    "style": "blue",
    "notes": "",
    "tags": [],
    "metadata": {
      "id": "9D1B71B1-895C-446F-A03F-50C01146F532",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:04:45.184863090Z",
      "modified": "2021-11-02T18:12:30.355533123Z",
      "location": "6.26.4.2.446.2.1:0",
      "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/446/2/1,:0,:679)",
      "slugs": {
        "created": "2021-11-02-180445",
        "modified": "2021-11-02-180445"
      }
    }
  },
  {
    "body": "Do not let the fact that things are not made for you...",
    "style": "green",
    "notes": "",
    "tags": ["#inspiration"],
    "metadata": {
      "id": "4620564A-0B64-4099-B5D6-6C9116A03AFF",
      "book_id": "1969AF0ECA8AE4965029A34316813924",
      "created": "2021-11-02T18:15:10.700510978Z",
      "modified": "2021-11-02T18:15:20.879488945Z",
      "location": "6.26.4.2.636.2.1:0",
      "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/636/2/1,:0,:166)",
      "slugs": {
        "created": "2021-11-02-181510",
        "modified": "2021-11-02-181510"
      }
    }
  }
]

This export was run with the --extract-tags option.

backup

Back-up Apple Books' databases.

Outputs using the following structure:

[ouput-directory]
 └── [YYYY-MM-DD-HHMMSS-VERSION]
      ├── AEAnnotation
      │   ├── AEAnnotation*.sqlite
      │   └── ...
      └── BKLibrary
          ├── BKLibrary*.sqlite
          └── ...

Example output:

[ouput-directory]
 └── 2022-10-09-152506-v4.4-5177
     ├── AEAnnotation
     │   ├── AEAnnotation_v10312011_1727_local.sqlite
     │   ├── AEAnnotation_v10312011_1727_local.sqlite-shm
     │   └── AEAnnotation_v10312011_1727_local.sqlite-wal
     └── BKLibrary
         ├── BKLibrary-1-091020131601.sqlite
         ├── BKLibrary-1-091020131601.sqlite-shm
         └── BKLibrary-1-091020131601.sqlite-wal

Options

Each of the three currently available Commands has its own pipeline for processing Apple Books' data before writing it to disk. And by extension, each pipeline has its own set of applicable options.

The pipelines and options for these three Commands are as follows:

╭─────────╮           Export Pipeline
│ Global* │          ╭────────╮ ╭─────────────╮ ╔════════╗
╰────┬────╯        ┌─┤ Filter ├─┤ Pre-process ├─╢ export ╟────────────────────┐
     │             │ ╰────────╯ ╰─────────────╯ ╚════════╝                    │
     │             │                                                          │
     │             │                                                          │
     │             │  Render Pipeline                                         │
     │             │ ╭───────────╮                                            │
     │             │ │ Templates ├──────────────┐                             │
     │             │ ╰───────────╯              │                             │
     │ ┌╌╌╌╌╌╌╌╌╌┐ │ ╭────────╮ ╭─────────────╮ │ ╔════════╗ ╭──────────────╮ │ ┌╌╌╌╌╌╌╌┐
     █─┤ Extract ├─┴─┤ Filter ├─┤ Pre-process ├─┴─╢ render ╟─┤ Post-process ├─┼─┤ Write │
     │ └╌╌╌╌╌╌╌╌╌┘   ╰────────╯ ╰─────────────╯   ╚════════╝ ╰──────────────╯ │ └╌╌╌╌╌╌╌┘
     │                                                                        │
     │                                                                        │
     │  Backup Pipeline                                                       │
     │ ╔════════╗                                                             │
     └─╢ backup ╟─────────────────────────────────────────────────────────────┘
       ╚════════╝

Global

The following options affect all Commands.

--output-directory <PATH>

Set the output directory for all Commands. Defaults to ~/.readstor.

--databases-directory <PATH>

Set a custom databases directory.

This can be useful when running ReadStor on databases backed-up with the backup command. The output structure the backup command creates is identical to the required databases directory structure.

The databases directory should contain the following structure:

[databases-directory]
 │
 ├── AEAnnotation
 │   ├── AEAnnotation*.sqlite
 │   └── ...
 │
 ├── BKLibrary
 │   ├── BKLibrary*.sqlite
 │   └── ...
 └── ...

--force

Run even if Apple Books is currently running.

--quiet

Silence output messages.

Render

The following options affect only the render commands.

--templates-directory <PATH>

Set a custom templates directory.

See the default templates for fully working examples.

--template-group <GROUP>

Render specified Template Groups.

Passing nonexistent Template Groups will return an error.

Multiple Template Groups can be passed using the following syntax.

readstor \
    # ...
    --template-group basic
    --template-group using-backlinks
    # ..

Export

The following options affect only the export commands.

--directory-template <TEMPLATE>

Set the output directory template.

--overwrite-existing

Overwrite existing files.

By default, exising files are left as is. With this flag, existing files are overwritten if they are re-exported.

Backup

The following options affect only the backup commands.

--directory-template <TEMPLATE>

Set the output directory template

Note that an escaping backslash \ was required to nest a pipe | inside a markdown table. In other words, the default value does not contain a backslash.

Backup Context

Filter

The following options affect only the render and export commands.

--filter <[OP]{FIELD}:{QUERY}>

Filter books/annotations before outputting.

Filtering allows you to specify, to a certain degree, which books and/or annotations to output. Currently, this is available for the export and render commands.

For example, this filter would only render annotations where its respective book's title is exactly the art spirit AND their tags contain the #star tag.

readstor render \
    --extract-tags \
    --filter "=title:the art spirit" \
    --filter "tag:#star"

This filter would export annotations where its respective book's author contains the string krishnamurti AND their tags contain either #star or #love.

readstor export \
    --extract-tags \
    --filter "author:krishnmurti" \
    --filter "?tag:#star #love"

Note that filters are case-insensitive.

Filter Results

After all the filters are run, a confirmation prompt is shown with a brief summary of the filtered down books/annotations.

$ readstor render \
    --extract-tags \
    --filter "=title:the art spirit" \
    --filter "tag:#star"
...
   ----------------------------------------------------------------
   Found 9 annotations from 2 books:
    • Think on These Things by Krishnamurti
    • The Art Spirit by Robert Henri
   ----------------------------------------------------------------
   Continue? [y/N]: █

This prompt can be auto-confirmed by passing the --auto-confirm-filter flag.

Filter Syntax

A filter consists of three parts: an optional operator, a field and a query. The syntax structure is as follows:

[operator]{field}:{query}

For example, looking at part of the command from above, we can see the three district parts of a filter:

readstor render \
    --extract-tags \
    --filter "=title:the art spirit" \
              │└──┬┘ └───────────┬┘
              │   │              │
              │   │              └────────── query: the art spirit
              │   └───────────────────────── field: title
              └────────────────────────── operator: = (exact)
    --filter "tag:#star"
              └┬┘ └──┬┘
               │     │
               │     └────────────────────── query: #star
               └──────────────────────────── field: tag
                                operator (default): ? (any)

Operator

The operator token determines how matching will be handled against the query.

When a filter is processed, the query is split on its spaces to create its component queries. For example, the input string the art sprit turns into three parts: the, art and spirit, and depending on the operator these three parts are handled differently in order to determine if an annotation is filtered out or not.

Note that when searching for an exact match in the tags field i.e. =tags:[query], the query remains split and the set of tags in the query is compared to those in each annotation.

Field

The field token determines which field to run the filter on.

Currently, only three fields are supported:

Query

The query string determines what will be searched in the specified field. A query is a space delineated set of words where each word can potentially be a separate search term depending on the specified operator.

--auto-confirm-filter

Auto-confirm Filter Results.

Pre-process

The following options affect only the render and export commands.

--extract-tags

Extract #tags from annotation.notes.

All matches are removed from annotation.notes and placed into annotation.tags. Additionally, all #tags within a book are compiled and placed them into book.tags.

Tags must start with a hash symbol # followed by a letter [a-zA-Z].

--normalize-whitespace

Normalize whitespace in annotation.body.

Trims whitespace and replaces all line-breaks with two consecutive line-breaks: \n\n.

--ascii-all

Convert all Unicode characters to ASCII.

All Unicode characters found in book.title, book.author and annotation.body are converted to ASCII.

--ascii-symbols

Convert "smart" Unicode symbols to ASCII.

"Smart" Unicode symbols found in book.title, book.author and annotation.body are converted to ASCII.

Characters and transliterations taken from:

• Daring Fireball - SmartyPants
• Python-Markdown - SmartyPants

Post-process

The following options affect only the render command.

--trim-blocks

Trim any blocks left after rendering.

Currently, this is a very naive implementation that mimics what tera might do if/when it adds trim_blocks. It is by no means smart and will just normalize whitespace regardless of what the template requested.

--wrap-text

Wrap text to a maximum character width.

Maximum line length is not guaranteed as long words are not broken if their length exceeds the maximum. Hyphenation is not used, however, existing hyphen can be split on to insert a line-break.

This will naively wrap all the text inside a rendered file regardless its structure. Use with caution! Extremely low values may cause unexpected results. Values above 80 or so are recommended.

Templates

ReadStor's most powerful feature is the templating interface it provides for Apple Books. Templates can be designed to output almost any kind of text-based file format including: Markdown, HTML, CSV. PDFs are possible as well but would require an extra step to convert an HTML file to the final PDF.

See the default templates for fully working examples.

An Example Template

The following is an example template along with its expected output and output structure. In fact, it's almost identical to the default template that comes with ReadStor.

Template Syntax

Tera, a Jinja-flavored templating language, provides a rich set of tools for building a wide range of template complexities. The following is a brief intro to the syntax.

Values can be accessed by placing the desired attribute between double curly-braces:

{{ book.author }}

A list of items can be iterated over using a for loop:

{% for annotation in annotations %} ... {% endfor %}

Conditional behavior can be achieved by using if/else statements:

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif %}

Values can be modified with filters. Here, a list of tags is concatenated with a space as a delimiter.

{{ annotation.tags | join(sep=" ") }}

And finally comments can be added like so:

{# Hi! I'm a comment! #}

See Tera's documentation to learn more about these and more advanced features, including template inheritance, the include tag, macros and the full list of available filters.

Template

Templates consist of two main sections, the configuration block written in YAML (and some Tera) and the template body written in Tera.

The configuration for this example template describes that this template, grouped under the name my-vault, will render a single file for each book and place them all directly into the output directory. Each output filename will follow the pattern of Author - Title.md.

<!-- readstor
group: my-vault
context: book
structure: flat
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
-->

---
title: {{ book.title }}
author: {{ book.author }}
last-opened: {{ book.metadata.last_opened | date(format="%Y-%m-%dT%H:%M") }}
---

# {{ book.author }} - {{ book.title }}

{% for annotation in annotations -%}

---

{{ annotation.body }}

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif %}

{% endfor %}

Output Structure

The output structure is primarily determined by the structure and context keys. With the structure set to flat all the output files will be placed inside the output directory with no structure. With the context set to book, a single file will be created for each book and, if the template body requests, will contain all its respective annotations. See Context Modes and Structure Modes for more information.

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit.md

Output Rendered File

Krishnamurti - Think on These Things.md

---
title: Think on These Things
author: Krishnamurti
last-opened: 2021-11-02T18:30
---

# Krishnamurti - Think on These Things

---

Do you know what intelligence is? It is the capacity, surely, to think freely,
without fear, without a formula, so that you begin to discover for yourself what
is real, what is true; but if you are frightened you will never be intelligent.
Any form of ambition, spiritual or mundane, breeds anxiety, fear; therefore
ambition does not help to bring about a mind that is clear, simple, direct, and
hence intelligent.

tags: #education #intelligence #ambition #fear

---

To find out is not to come to a conclusion. I don’t know if you see the
difference. The moment you come to a conclusion as to what intelligence is, you
cease to be intelligent. That is what most of the older people have done: they
have come to conclusions. Therefore they have ceased to be intelligent. So you
have found out one thing right off: that an intelligent mind is one which is
constantly learning, never concluding.

tags: #learning #intelligence

---

The deeper the mind penetrates its own thought processes, the more clearly it
understands that all forms of thinking are conditioned; therefore the mind is
spontaneously very still—which does not mean that it is asleep. On the contrary,
the mind is then extraordinarily alert, no longer being drugged by mantrams, by
the repetition of words, or shaped by discipline. This state of silent alertness
is also part of awareness; and if you go into it still more deeply you will find
that there is no division between the person who is aware and the object of
which he is aware.

tags: #thinking

Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md

---
title: "Surely You're Joking, Mr. Feynman!"
author: Richard P. Feynman
last-opened: 2021-11-02T18:27
---

# Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"

---

After the dinner we went off into another room, where there were different
conversations going on. There was a Princess Somebody of Denmark sitting at a
table with a number of people around her, and I saw an empty chair at their
table and sat down.

She turned to me and said, “Oh! You’re one of the Nobel-Prize-winners. In what
field did you do your work?”

“In physics,” I said.

“Oh. Well, nobody knows anything about that, so I guess we can’t talk about it.”

“On the contrary,” I answered. “It’s because somebody knows something about it
that we can’t talk about physics. It’s the things that nobody knows anything
about that we can discuss. We can talk about the weather; we can talk about
social problems; we can talk about psychology; we can talk about international
finance—gold transfers we can’t talk about, because those are understood—so it’s
the subject that nobody knows anything about that we can all talk about!”

I don’t know how they do it. There’s a way of forming ice on the surface of the
face, and she did it! She turned to talk to somebody else.

Robert Henri - The Art Spirit.md

---
title: The Art Spirit
author: Robert Henri
last-opened: 2021-11-02T18:27
---

# Robert Henri - The Art Spirit

---

We are not here to do what has already been done.

---

The object of painting a picture is not to make a picture—however unreasonable
this may sound. The picture, if a picture results, is a by-product and may be
useful, valuable, interesting as a sign of what has past. The object, which is
back of every true work of art, is the attainment of a state of being, a state
of high functioning, a more than ordinary moment of existence. In such moments
activity is inevitable, and whether this activity is with brush, pen, chisel, or
tongue, its result is but a by-product of the state, a trace, the footprint of
the state.

tags: #artist #being

---

Of course it is not easy to go one’s road. Because of our education we
continually get off our track, but the fight is a good one and there is joy in
it if there is any success at all. After all, the goal is not making art. It is
living a life. Those who live their lives will leave the stuff that is really
art. Art is a result. It is the trace of those who have led their lives. It is
interesting to us because we read the struggle and the degree of success the man
made in his struggle to live. The great question is: “What is worth while?” The
majority of people have failed to ask themselves seriously enough, and have
failed to try seriously enough to answer this question.

---

Do not let the fact that things are not made for you, that conditions are not as
they should be, stop you. Go on anyway. Everything depends on those who go on
anyway.

tags: #inspiration

Configuration

A template's configuration describes both what the template expects to render and how the output structure and naming should be. Every template must start with a configuration block.

Overview

The configuration starts off as a basic HTML comment tag...

<!-- -->

... with the word readstor...

<!-- readstor -->

...and a new line.

<!-- readstor

-->

The YAML configuration is then placed inside the tag. For example:

<!-- readstor
group: my-vault
context: book
structure: nested
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"
-->

...

Note that the final rendered output file will not include its template's configuration. Additionally, if the configuration ended with trailing line-breaks, a single one of them is removed. This allows for some extra whitespace while working with a template without affecting final rendered output.

A quick rundown of each configuration key:

Template Groups

Groups are used to identify multiple templates that are intended to be part of a single output. Conversely, they also provide a way to separate unrelated templates rendered to the same output directory. The most common use case would be when using a pair of templates where one is used to render a book and the other to render each of its annotations separately.

Group names are sanitized to make sure they interact well with the file system. See String Sanitization for more information.

Grouping is triggered when one or more templates are set to either the flat-grouped or the nested-grouped Structure Mode. The output files are placed within a directory named after the group.

For example, if two templates share these values in their configurations:

group: my-vault
structure: flat-grouped

The output will be:

[output-directory]
 ├── my-vault
 │   ├── 2021-11-02-180445-the-art-spirit.md
 │   ├── 2021-11-02-181250-the-art-spirit.md
 │   ├── 2021-11-02-181325-the-art-spirit.md
 │   ├── 2021-11-02-181510-the-art-spirit.md
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   ├── Krishnamurti - Think on These Things.md
 │   ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 │   └── Robert Henri - The Art Spirit.md
 │
 ├── [other-group]
 │    └── ...
 └── ...

And, if two templates share these values in their configurations:

group: my-vault
structure: nested-grouped

The output will be:

[output-directory]
 ├── my-vault
 │   ├── Krishnamurti - Think on These Things
 │   │   ├── 2021-11-02-182319-think-on-these-things.md
 │   │   ├── 2021-11-02-182426-think-on-these-things.md
 │   │   ├── 2021-11-02-182543-think-on-these-things.md
 │   │   ├── 2021-11-02-182648-think-on-these-things.md
 │   │   ├── 2021-11-02-182805-think-on-these-things.md
 │   │   └── Krishnamurti - Think on These Things.md
 │   ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 │   └── Robert Henri - The Art Spirit
 │       ├── 2021-11-02-180445-the-art-spirit.md
 │       ├── 2021-11-02-181250-the-art-spirit.md
 │       ├── 2021-11-02-181325-the-art-spirit.md
 │       ├── 2021-11-02-181510-the-art-spirit.md
 │       └── Robert Henri - The Art Spirit.md`
 │
 ├── [other-group]
 │    └── ...
 └── ...

Context Modes

At render time, each template is injected with a "context", in other words, the data it will render. ReadStor provides two different context modes: book and annotation. The context mode dictates not just the data within the context but also changes the number of output files. See A Note On Output Structure for more information.

The Book Context

Note that annotations is plural in the book context.

When selected, a single file is rendered out from a context containing the data from a single book and its annotations. For example, represented here in YAML:

book:
  title: The Art Spirit
  author: Robert Henri
  metadata:
    id: 1969AF0ECA8AE4965029A34316813924
    last_opened: 2021-11-02T18:27:04.781938076Z
  slugs:
    title: the-art-spirit
    author: robert-henri
annotations:
  - body: We are not here to do what has already been done.
    style: purple
    notes: ""
    tags: []
    metadata:
      id: C932CE69-8584-4555-834C-797DF84E6825
      book_id: 1969AF0ECA8AE4965029A34316813924
      created: 2021-11-02T18:12:50.826642036Z
      modified: 2021-11-02T18:12:51.831905841Z
      location: 6.18.4.2.20.2.1:0
      epubcfi: epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)
      slugs:
        created: 2021-11-02-181250
        modified: 2021-11-02-181250
  - body: The object of painting a picture...
    style: yellow
    notes: ""
    tags:
      - "#artist"
      - "#being"
    metadata:
      id: 3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C
      book_id: 1969AF0ECA8AE4965029A34316813924
      created: 2021-11-02T18:13:25.905355930Z
      modified: 2021-11-02T18:14:12.444134950Z
      location: 6.24.4.2.296.2.1:0
      epubcfi: epubcfi(/6/24[Part09_Split3]!/4/2/296/2,/1:0,/7:257)
      slugs:
        created: 2021-11-02-181325
        modified: 2021-11-02-181325
  # ...
names:
  book: Robert Henri - The Art Spirit.md
  annotations:
    C932CE69-8584-4555-834C-797DF84E6825: 2021-11-02-181250-the-art-spirit.md
    3FCC630A-55E6-4D6F-8E8F-DAD7C4E20A1C: 2021-11-02-181325-the-art-spirit.md
    # ...
  directory: Robert Henri - The Art Spirit

See Context Reference - Book for more information.

The Annotation Context

When selected, multiple files are rendered out from a context containing the data from a single annotation and its respective book. For example, represented here in YAML:

book:
  title: The Art Spirit
  author: Robert Henri
  metadata:
    id: 1969AF0ECA8AE4965029A34316813924
    last_opened: 2021-11-02T18:27:04.781938076Z
  slugs:
    title: the-art-spirit
    author: robert-henri
annotation:
  body: We are not here to do what has already been done.
  style: purple
  notes: ""
  tags: []
  metadata:
    id: C932CE69-8584-4555-834C-797DF84E6825
    book_id: 1969AF0ECA8AE4965029A34316813924
    created: 2021-11-02T18:12:50.826642036Z
    modified: 2021-11-02T18:12:51.831905841Z
    location: 6.18.4.2.20.2.1:0
    epubcfi: epubcfi(/6/18[Part09_Split0]!/4/2/20/2/1,:0,:49)
    slugs:
      created: 2021-11-02-181250
      modified: 2021-11-02-181250
names:
  book: Robert Henri - The Art Spirit.md
  annotations:
    C932CE69-8584-4555-834C-797DF84E6825: 2021-11-02-181250-the-art-spirit.md
  directory: Robert Henri - The Art Spirit

See Context Reference - Annotation for more information.

A Note On Output Structure

When selecting a context mode it's important to understand how the output files will look. The following is a quick visualization.

When the context mode is set to book, a single file for each book is rendered out from the template. Using the following configuration keys:

context: book
structure: nested

The output structure is as follows:

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit
     └── Robert Henri - The Art Spirit.md`

And when the context mode is set to annotation, multiple files are rendered out, one for each annotation within a book. Using the following configuration keys:

context: annotation
structure: nested

The output structure is as follows:

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   └── 2021-11-02-182805-think-on-these-things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   └── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 └── Robert Henri - The Art Spirit
     ├── 2021-11-02-180445-the-art-spirit.md
     ├── 2021-11-02-181250-the-art-spirit.md
     ├── 2021-11-02-181325-the-art-spirit.md
     └── 2021-11-02-181510-the-art-spirit.md

When both templates are rendered to the same directory, we get a complete output with a book and each of its annotations all rendered out to their own file.

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 └── Robert Henri - The Art Spirit
     ├── 2021-11-02-180445-the-art-spirit.md
     ├── 2021-11-02-181250-the-art-spirit.md
     ├── 2021-11-02-181325-the-art-spirit.md
     ├── 2021-11-02-181510-the-art-spirit.md
     └── Robert Henri - The Art Spirit.md

Structure Modes

The structure mode determines how the output directories and files are structured. ReadStor provides four structure modes: flat, flat-grouped, nested and nested-grouped.

Flat Mode

structure: flat

When selected, the template is rendered to the output directory without any structure. All the files are placed directly within the output directory. No additional directories are created.

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit.md
 └── ...

Flat & Grouped Mode

group: my-vault
structure: flat-grouped

When selected, the template is rendered to the output directory and placed inside a directory named after its group. This is useful if multiple template groups are being rendered to the same directory.

[output-directory]
 └── my-vault
     ├── Krishnamurti - Think on These Things.md
     ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
     ├── Robert Henri - The Art Spirit.md
     └── ...

Nested Mode

structure: nested

When selected, the template is rendered to the output directory and placed inside a directory named after the names.directory key. This is useful if a template group contains multiple templates.

[output-directory]
 ├── Krishnamurti - Think on These Things
 │   ├── 2021-11-02-182319-think-on-these-things.md
 │   ├── 2021-11-02-182426-think-on-these-things.md
 │   ├── 2021-11-02-182543-think-on-these-things.md
 │   ├── 2021-11-02-182648-think-on-these-things.md
 │   ├── 2021-11-02-182805-think-on-these-things.md
 │   └── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
 │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit
 │   ├── 2021-11-02-180445-the-art-spirit.md
 │   ├── 2021-11-02-181250-the-art-spirit.md
 │   ├── 2021-11-02-181325-the-art-spirit.md
 │   ├── 2021-11-02-181510-the-art-spirit.md
 │   └── Robert Henri - The Art Spirit.md`
 └── ...

Nested & Grouped Mode

group: my-vault
structure: nested-grouped

When selected, the template is rendered to the output directory and placed inside a directory named after its group and another named after the names.directory key. This is useful if multiple template groups are being rendered to the same directory and if a template group contains multiple templates.

[output-directory]
 └── my-vault
     ├── Krishnamurti - Think on These Things
     │   ├── 2021-11-02-182319-think-on-these-things.md
     │   ├── 2021-11-02-182426-think-on-these-things.md
     │   ├── 2021-11-02-182543-think-on-these-things.md
     │   ├── 2021-11-02-182648-think-on-these-things.md
     │   ├── 2021-11-02-182805-think-on-these-things.md
     │   └── Krishnamurti - Think on These Things.md
     ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!"
     │   ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
     │   └── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
     └── Robert Henri - The Art Spirit
         ├── 2021-11-02-180445-the-art-spirit.md
         ├── 2021-11-02-181250-the-art-spirit.md
         ├── 2021-11-02-181325-the-art-spirit.md
         ├── 2021-11-02-181510-the-art-spirit.md
         └── Robert Henri - The Art Spirit.md

Extension

Defines the template's output file extension. Note that this will be appended to all the output filenames with the format [filename].[extension].

Names

Output files and directory names can be customized using the same Tera syntax. ReadStor will inject a different context into each names during render time and set the template's output file/directory name to the resulting string.

The rendered book, annotation and directory names are sanitized to make sure they interact well with the file system. See String Sanitization for more information.

Additionally, all the rendered values from these keys are available within the template's context under the names variable regardless of the current context mode. See Context Reference - Names for more information.

Note that the template's context matters when setting names.

For example, if the template's context is set to book:

group: my-vault
structure: flat
context: book # <- Here!
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"

ReadStor will generate a single file for each book and name them using the rendered result of names.book. The resulting output would be:

[output-directory]
 ├── Krishnamurti - Think on These Things.md
 ├── Richard P. Feynman - "Surely You're Joking, Mr. Feynman!".md
 ├── Robert Henri - The Art Spirit.md
 └── ...

However, if the context is changed to annotation:

group: my-vault
structure: flat
context: annotation # <- Here!
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"

ReadStor will now generate a single file for each annotation in each book and name them using the rendered result of names.annotation. The resulting output would be:

[output-directory]
 ├── 2021-11-02-180445-the-art-spirit.md
 ├── 2021-11-02-181250-the-art-spirit.md
 ├── 2021-11-02-181325-the-art-spirit.md
 ├── 2021-11-02-181510-the-art-spirit.md
 ├── 2021-11-02-182059-surely-youre-joking-mr-feynman.md
 ├── 2021-11-02-182319-think-on-these-things.md
 ├── 2021-11-02-182426-think-on-these-things.md
 ├── 2021-11-02-182543-think-on-these-things.md
 ├── 2021-11-02-182648-think-on-these-things.md
 ├── 2021-11-02-182805-think-on-these-things.md
 └── ...

Book Names

Defines the filename template to use when the parent template's context mode is set to book. This template only has access to the book context when it's rendered.

Annotation Names

Defines the filename template to use when the parent template's context mode is set to annotation. This template has access to the book and annotation context when its rendered.

Directory Names

Defines the directory name template to use when the parent template's structure mode is set to nested or the nested-grouped. This template only has access to the book context when its rendered.

Limitations

Why does a single template have both a names.book and names.annotation key?

This is mainly the result of a current limitation with ReadStor. Templates that are part of the same group have no relation internally. ReadStor just renders each template it finds. If multiple templates share the same value for group then they end up in the same directory when the structure mode is set to grouped or nested-grouped.

As a result of this limitation, if a template requires awareness of its group's sibling names i.e. the value defined in the names field, they must be defined in each template. This results in some duplication across templates. This should hopefully be resolved in future versions of ReadStor.

For more information on how to generate and use names see Context Reference - Names and Backlinks.

Partial Templates

When working with larger templates it can be helpful to separate them into smaller building blocks. One way to do this is to create partial templates and include them in other templates.

Partial templates differ in two important ways from their normal template counterparts.

1. They must begin with an underscore _ as this is currently the only indicator of whether a template is partial or not.

2. They require no configuration block as they inherit their configuration from the template that includes them.

See the using-partials templates for a fully working example.

Including a Partial Template

Partial templates are included using the include tag:

{% include "_my-partial-template.md" %}

Where "_my-partial-template.md" is a path relative to the templates directory. Therefore, this would be pointing to a template at the root of the templates directory. If we wanted to organize our templates into different directories we would have to add the directory names for any directories between the including template and the templates directory.

For example, with the following structure:

[templates-directory]
 └── my-vault-templates
     ├── _annotation.md
     ├── _book.md
     └── template.md

We would use the following include tags:

{% include "my-vault-templates/_book.jinja2" %}
{% include "my-vault-templates/_annotation.jinja2" %}

See the documentation for Tera's include tag for more information on its features and limitations.

Backlinks

Generating backlinks for a Zettelkasten-like note-taking experience is relatively easy with ReadStor. It requires two separate templates: One for rendering the book and one for rendering each of its annotations.

See the using-backlinks templates for a fully working example.

Template Configuration

First, let's define the configurations for our two templates. They should be almost identical to one another.

• group is set to the same value across the two templates. This makes sure that if we select a grouped Structure Mode, the templates will be rendered to the same directory.

• context is set to book for the book template and annotation for the annotation template.

• structure can be set to any of the four modes, however, nested-grouped feels the most appropriate as it would place each book into its own directory and place them all under a directory named after the value for the group key. See Structure Modes for more information.

• extension is set to md as the template will be outputting Markdown.

• names can be set to anything as long as the values are identical between the two templates. It might seem odd to see the values for names duplicated across the two templates. Shouldn't the book template define names.book and the annotation template define names.annotation? Ideally, yes. This need for duplication is the result of a current limitation of ReadStor, therefore they must be identical, so the backlinks are correctly generated.

Book Template Configuration

group: my-vault
context: book # <- The only difference!
structure: nested-grouped
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

Annotation Template Configuration

group: my-vault
context: annotation # <- The only difference!
structure: nested-grouped
extension: md
names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

Template Body

With our configuration all set up, we can now use the names object, which contains all the rendered Names, to link between our rendered output files. See Context Reference - Names for more information.

Book Template Body

The names.annotations object is a list of dictionaries, where each dictionary refers to a rendered annotation file and contains its filename along with metadata about its respective annotation. This allows us to link back to the annotation and sort the links based of off different criteria. In the example below, Tera's sort filter is used with the attribute argument and the location attribute.

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations | sort(attribute="location")-%}
![[{{ name.filename }}]]
{% endfor %}

Alternatively we can use the names.directory variable to access the rendered name of the parent directory. This value is only available if the Structure Mode is set to nested or nested-grouped.

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations -%}
![[{{ name.directory }}/{{ name.filename }}]]
{% endfor %}

Annotation Template Body

Finally, using the names.book variable we're able to link back to the source book.

# [[{{ names.book }}]]

{{ annotation.body }}

{% if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif -%}

Output Structure

[output-directory]
 ├── my-vault
 │   └── Robert Henri - The Art Spirit
 │       ├── 2021-11-02-180445-the-art-spirit.md
 │       ├── 2021-11-02-181250-the-art-spirit.md
 │       ├── 2021-11-02-181325-the-art-spirit.md
 │       ├── 2021-11-02-181510-the-art-spirit.md
 │       └── Robert Henri - The Art Spirit.md
 │
 ├── [group]
 │    └── ...
 └── ...

Output Rendered Files

Robert Henri - The Art Spirit.md

# Robert Henri - The Art Spirit

![[2021-11-02-180445-the-art-spirit.md]]
![[2021-11-02-181250-the-art-spirit.md]]
![[2021-11-02-181325-the-art-spirit.md]]
![[2021-11-02-181510-the-art-spirit.md]]

2021-11-02-180445-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

Of course it is not easy to go one’s road. Because of our education we
continually get off our track, but the fight is a good one and there is joy in
it if there is any success at all. After all, the goal is not making art. It is
living a life. Those who live their lives will leave the stuff that is really
art. Art is a result. It is the trace of those who have led their lives. It is
interesting to us because we read the struggle and the degree of success the
man made in his struggle to live. The great question is: “What is worth while?”
The majority of people have failed to ask themselves seriously enough, and have
failed to try seriously enough to answer this question.

2021-11-02-181250-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

We are not here to do what has already been done.

2021-11-02-181325-the-art-spirit.md

# [[Robert Henri - The Art Spirit.md]]

The object of painting a picture is not to make a picture—however unreasonable
this may sound. The picture, if a picture results, is a by-product and may be
useful, valuable, interesting as a sign of what has past. The object, which is
back of every true work of art, is the attainment of a state of being, a state
of high functioning, a more than ordinary moment of existence. In such moments
activity is inevitable, and whether this activity is with brush, pen, chisel,
or tongue, its result is but a by-product of the state, a trace, the footprint
of the state.

tags: #artist #being

2021-11-02-181510-the-art-spirit.m

# [[Robert Henri - The Art Spirit.md]]

Do not let the fact that things are not made for you, that conditions are not
as they should be, stop you. Go on anyway. Everything depends on those who go
on anyway.

tags: #inspiration

String Sanitization

To avoid any unexpected behavior, certain user-provided strings are sanitized before they are used for naming files or directories. Characters are either removed or replaced with an underscore _.

This following values are sanitized:

• Template Group set in the group config key.
• The rendered values from Names set in the names config key.

Context Reference

Every template is injected with a "context" i.e. the data currently available to rendering. ReadStor injects three different objects into every template context: book, annotation (or annotations depending on the Context Mode) and names.

Book

A single book object is injected into all template contexts regardless of the template's Context Mode.

Template Fields - Book

Example Data - Book

{
  "title": "The Art Spirit",
  "author": "Robert Henri",
  "tags": ["#artist", "#being", "#inspiration"],
  "metadata": {
    "id": "1969AF0ECA8AE4965029A34316813924",
    "last_opened": "2021-11-02T18:27:04.781938076Z"
  },
  "slugs": {
    "title": "the-art-spirit",
    "author": "robert-henri"
  }
}

Example Template - Book

---
title: {{ book.title }}
author: {{ book.author }}
tags: {{ book.tags | join(sep=" ") }}
id: {{ book.metadata.id }}
last-opened: {{ book.metadata.last_opened | date(format="%Y-%m-%d-%H:%M") }}
---

Here Tera's date filter is used to format a datetime object into a human-readable date.

Annotation

The number of annotation objects injected into a template depends on the template's Context Mode. When the Context Mode is set to annotation, a single annotation object is injected into the template's context. When the Context Mode is set to book, multiple annotation objects, under the name annotations, are injected into the template's context.

Template Fields - Annotation

Example Data - Annotation

{
  "body": "Of course it is not easy to go one’s road...",
  "style": "blue",
  "notes": "",
  "tags": [],
  "metadata": {
    "id": "9D1B71B1-895C-446F-A03F-50C01146F532",
    "book_id": "1969AF0ECA8AE4965029A34316813924",
    "created": "2021-11-02T18:04:45.184863090Z",
    "modified": "2021-11-02T18:12:30.355533123Z",
    "location": "6.26.4.2.446.2.1:0",
    "epubcfi": "epubcfi(/6/26[Part09_Split4]!/4/2/446/2/1,:0,:679)",
    "slugs": {
      "created": "2021-11-02-180445",
      "modified": "2021-11-02-180445"
    }
  }
}

Example Template - Annotation

{% for annotation in annotations -%}

---

{{ annotation.body }}

{%- if annotation.notes %}notes: {{ annotation.notes }}{% endif -%}
{%- if annotation.tags %}tags: {{ annotation.tags | join(sep=" ") }}{% endif %}

{% endfor %}

Here Tera's join filter is used to join an array of items into a space-separated string.

Names

A single names object is injected into all template contexts regardless of the template's Context Mode. These contain all the file and directory names rendered from the names key in the template's config. See Names for more information.

Template Fields - Names

The names.annotations object is a list of dictionaries, where each dictionary refers to a rendered annotation file and contains its filename along with metadata about its respective annotation. Each dictionary consists of the following attributes:

These attributes allow the sorting of the names.annotations list using Tera's sort filter. See Backlinks for example usage.

Example Data - Names

With the following names configuration:

names:
  book: "{{ book.author }} - {{ book.title }}"
  annotation: "{{ annotation.slugs.metadata.created }}-{{ book.slugs.title }}"
  directory: "{{ book.author }} - {{ book.title }}"

{
  "book": "Robert Henri - The Art Spirit.md",
  "annotations": [
    {
      "filename": "2021-11-02-181510-the-art-spirit.md",
      "created": "2021-11-02T18:15:10.700510978Z",
      "modified": "2021-11-02T18:15:20.879488945Z",
      "location": "6.26.4.2.636.2.1:0"
    },
    {
      "filename": "2021-11-02-180445-the-art-spirit.md",
      "created": "2021-11-02T18:04:45.184863090Z",
      "modified": "2021-11-02T18:12:30.355533123Z",
      "location": "6.26.4.2.446.2.1:0"
    },
    {
      "filename": "2021-11-02-181325-the-art-spirit.md",
      "created": "2021-11-02T18:13:25.905355930Z",
      "modified": "2021-11-02T18:14:12.444134950Z",
      "location": "6.24.4.2.296.2.1:0"
    },
    {
      "filename": "2021-11-02-181250-the-art-spirit.md",
      "created": "2021-11-02T18:12:50.826642036Z",
      "modified": "2021-11-02T18:12:51.831905841Z",
      "location": "6.18.4.2.20.2.1:0"
    }
  ],
  "directory": "Robert Henri - The Art Spirit"
}

Example Template - Names

# {{ book.author }} - {{ book.title }}

{% for name in names.annotations | sort(attribute="location") -%}
![[{{ name.filename }}]]
{% endfor %}

Apple Books

The following information assumes that Apple Books syncing with iCloud Drive is disabled!

Complete archive and restore scripts are available in scripts/apple-books.

What To Archive/Restore?

Apple Books stores the bulk of its data in two locations:

1. The location of EPUBs, PDFs, Audiobooks, etc.:

   ~/Library/Containers/com.apple.BKAgentService
   

2. The location of the library's databases:

   ~/Library/Containers/com.apple.iBooksX
   

However, only archiving and restoring these directories might miss some metadata. Searching ~/Library/Containers for anything that contains Books yields some other directories:

$ find ~/Library/Containers -type d -name "*Books*"

./com.apple.BKAgentService/Data/Documents/iBooks
./com.apple.BKAgentService/Data/Documents/iBooks/Books
./com.apple.iBooksX
./com.apple.iBooksX/Data/Documents/BCCloudKit-iBooks
./com.apple.iBooksX/Data/Documents/BCCloudAsset-iBooks
./com.apple.iBooksX/Data/Documents/BKBookstore
./com.apple.iBooksX/Data/Library/Caches/com.apple.iBooksX
./com.apple.iBooksX.CacheDelete
./com.apple.iBooksX.DiskSpaceEfficiency
./com.apple.iBooksX.SharingExtension
./com.apple.iBooksX-SecureUserDefaults

Using BK as a proxy for Books yields no additional directores:

$ find ~/Library/Containers -type d -name "*BK*"

./com.apple.BKAgentService
./com.apple.iBooksX/Data/Documents/BKSeriesDatabase
./com.apple.iBooksX/Data/Documents/BKLibraryDataSourceDevelopment
./com.apple.iBooksX/Data/Documents/BKLibrary
./com.apple.iBooksX/Data/Documents/BKBookstore

So it's safe to assume any directory starting with com.apple.Books or com.apple.BK is important. Therefore, these two globs along with Apple Books' Group Container should be used for archiving and restoring:

~/Library/Containers/com.apple.BK*
~/Library/Containers/com.apple.iBooks*
~/Library/Group\ Containers/group.com.apple.iBooks

Important Note

Archiving/restoring will work only if the path to the username has not changed since the library was archived. Doing a few searches shows that the username has been hard-coded into some files. This is most evident in the BKLibrary-*.sqlite file which contains absolute paths to library files.

For example:

$ find ~/Library/Containers/com.apple.BKAgentService \
    -type f -a -exec grep -l --exclude=\*.{htm,html,xhtml} $USER {} +

.../.com.apple.containermanagerd.metadata.plist
.../Container.plist
.../Data/Documents/iBooks/Books/Books.plist

$ find ~/Library/Containers/com.apple.iBooksX* \
    -type f -a -exec grep -l --exclude=\*.{htm,html,xhtml} $USER {} +

.../com.apple.iBooksX/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX/Container.plist
.../com.apple.iBooksX/Data/Documents/BKLibrary/BKLibrary-1-091020131601.sqlite-wal
.../com.apple.iBooksX/Data/Documents/BKLibrary/BKLibrary-1-091020131601.sqlite
.../com.apple.iBooksX-SecureUserDefaults/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX-SecureUserDefaults/Container.plist
.../com.apple.iBooksX.CacheDelete/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.CacheDelete/Container.plist
.../com.apple.iBooksX.DiskSpaceEfficiency/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.DiskSpaceEfficiency/Container.plist
.../com.apple.iBooksX.SharingExtension/.com.apple.containermanagerd.metadata.plist
.../com.apple.iBooksX.SharingExtension/Container.plist

Archive Library

The following snippet is taken from scripts/apple-books/archive.sh.

Archiving the library is as simple as running two rsync commands. This should save all the relevant Apple Books data and metadata to a single directory. Make sure to replace [PATH-TO-ARCHIVE] with a valid path to said directory.

rsync \
    --archive \
    --extended-attributes \
    $HOME/Library/Containers/com.apple.BK* \
    $HOME/Library/Containers/com.apple.iBooks* \
    [PATH-TO-ARCHIVE]/Containers

rsync \
    --archive \
    --extended-attributes \
    $HOME/Library/Group\ Containers/group.com.apple.iBooks \
    [PATH-TO-ARCHIVE]/Group\ Containers

For example, if [PATH-TO-ARCHIVE] is:

~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6

Our rsync commands would be:

rsync \
    --archive \
    --extended-attributes \
    $HOME/Library/Containers/com.apple.BK* \
    $HOME/Library/Containers/com.apple.iBooks* \
    ~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Containers

rsync \
    --archive \
    --extended-attributes \
    $HOME/Library/Group\ Containers/group.com.apple.iBooks \
    ~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Group\ Containers

And the resulting archive would resemble:

~/archives
  └── 2022-10-08--apple-books-v4.4-5177--macos-v12.6
      ├── Containers
      │   ├── com.apple.BKAgentService
      │   ├── com.apple.iBooks.BooksNotificationContentExtension
      │   ├── com.apple.iBooks.engagementExtension
      │   ├── com.apple.iBooks.iBooksSpotlightExtension
      │   ├── com.apple.iBooksX
      │   ├── com.apple.iBooksX-SecureUserDefaults
      │   ├── com.apple.iBooksX.BooksThumbnail
      │   ├── com.apple.iBooksX.CacheDelete
      │   ├── com.apple.iBooksX.DiskSpaceEfficiency
      │   └── com.apple.iBooksX.SharingExtension
      └── Group Containers
          └── group.com.apple.iBooks

Restore Library

The following snippets are taken from scripts/apple-books/restore.sh.

Restoring the library takes an extra step. First we need to clear out the current Apple Books library. We can delete all the library files and directories by using the paths we determined from What To Archive/Restore?.

rm -rf $HOME/Library/Containers/com.apple.BK*
rm -rf $HOME/Library/Containers/com.apple.iBooks*
rm -rf $HOME/Library/Group\ Containers/group.com.apple.iBooks

Finally, we can run the reverse rsync commands and restore the archive we previously made. Make sure to replace [PATH-TO-ARCHIVE] with a valid path.

rsync \
    --archive \
    --extended-attributes \
    [PATH-TO-ARCHIVE]/Containers/ \
    $HOME/Library/Containers/

rsync \
    --archive \
    --extended-attributes \
    [PATH-TO-ARCHIVE]/Group\ Containers/ \
    $HOME/Library/Group\ Containers/

The trailing forward-slash after Containers and Group\ Containers here is important. It tells rsync to move the archive directory's contents into the target. Otherwise, it would move the archive directory into the target.

For example, if [PATH-TO-ARCHIVE] is:

~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6

Our rsync command would be:

rsync \
    --archive \
    --extended-attributes \
    ~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Containers/ \
    $HOME/Library/Containers/  # Note the forward-slash! ---------------^

rsync \
    --archive \
    --extended-attributes \
    ~/archives/2022-10-08--apple-books-v4.4-5177--macos-v12.6/Group\ Containers/ \
    $HOME/Library/Group\ Containers/  # Note the forward-slash! ---------------^