Automating novel and anthology publishing

I’ve recently created — and then extensively enhanced — a wrapper and configuration for pandoc, to allow me to publish my fiction written in Markdown as both ebooks and paperback novels.

I previously used Ulysses for all of this, but lately I’ve been drawn towards platform-portability, and there were a number of limitations in what I could conveniently do while all my writing was siloed within a proprietary database.

Publishing my novels is relatively straightforward, but things are more complicated for my collections of ultra-short stories. I write in various genres, and creating anthologies always involved a lot of manual busywork, for a few reasons.

First of all, I write the stories primarily to send out via my email newsletter each week. Often, I include a short preamble with some context, personal goings-on, or other remarks, and each story is numbered both in its filename and also in the Markdown content itself. Here’s an abbreviated example from last Christmas:

# 208 — Peace and Friendship

:genre: sci-fi dystopian

The big week has arrived, and my son is very excited for Santa’s visit. I hope  
you’ll have a relaxing break, and indeed that the designated gift-givers are  
good to you, too. Thanks for reading these stories, as always, and best wishes  
to you and your loved ones.

---- 

Carter stood beside the young recruit, and pointed at the machine lying on the  
floor in the centre of the room. Its limbs had been severed, all six of them,  
and its array of sensory devices watched the two men silently.

(Read the rest of the story here, if you like.)

As is probably self-evident, each story’s Markdown file has a level-1 heading with the story number and its title. Then, a line with the story’s genres using a simple syntax (nothing to do with Markdown; made up by me, and based loosely on org mode in emacs). Then there’s the (optional, not always present) preamble, followed by a Markdown horizontal-rule, then the story itself.

To create an anthology of those stories in Ulysses, I had to copy all the relevant stories into a new group (folder), trim out the story-numbers and preambles, make suitable front- and back-matter by duplicating that from another project and then modifying it appropriately, and export the resulting books. Certainly doable, but very manual and inconvenient. One of the aims of my pandoc wrapper was to make publishing much easier for me, and in particular to help me readily customise the output on a per-book basis.

I implemented several features to help with that. First, keeping files in the filesystem is an implicit power-user feature, because simple file-naming allows for stable custom ordering of files, and symlinking trivially allows for the use of the same front- and back-matter templates in arbitrary numbers of different books, no matter where the primary content of those books resides.

So, I have a single master Front Matter directory of suitable Markdown files, named such as to be in the correct order, and the same for Back Matter. For each distinct book, I then just have a directory which contains a symlink to the Front Matter folder, a symlink to the Back Matter folder, and an actual folder with the book’s contents — all three entries named so they’re ordered appropriately. If I add a new book, once the content is written, setup involves copying two symlinks and naming a single folder.

To customise the front- and back-matter to suit the book being built, my wrapper implements a placeholders system, which can use a full templating engine if desired. Metadata JSON files provide the data, which can then be inserted into the content at build time.

Here’s the metadata for my Once Upon A Time anthology:

{
  "lang": "en",
  "title": "Once Upon A Time",
  "subtitle": "The Stories So Far",
  "author": "Matt Gemmell",
  "cover-image": "cover-image.jpg",
  "css": "pdf-ouat.css"
}

As shown above, metadata can provide CSS overrides too, if something needs to be styled in a bespoke way for just that book. And since the wrapper supports specifying metadata files as arguments, and even providing arguments via args files, multiple configurations are available for each book by just passing the filename of the relevant set of parameters.

The ultimate goal for my stories collection is to be able to build genre-specific anthologies with as little effort as possible. So far, we have an easy way to assemble all the content, and to tailor the output in terms of inserting metadata into placeholders, but two obstacles remain: filtering which files are included, and pre-processing the Markdown files to strip out unwanted stuff. My wrapper implements both of those.

The Markdown files are filtered by genre using the exclusions feature, as follows:

Skip any ancillary or in-progress files:
exclude	filename	*							\bTITLE\b										Filename contains 'TITLE'
exclude	filename	*							^_													Filename begins with underscore
exclude	filename	*							(?i)NOTES\.[^.]+$						Basename ends with 'notes'
exclude	contents	*							(?i)\b(TK)+\b								File contains TKs

Ensure only relevant stories are included:
include	contents	(?i)stories$	(?im)^:genre:.*?sci-fi.*?		Genre filter

First, it excludes any in-progress or notes files, and then it excludes any stories whose genre-line doesn’t include the desired genre. I have one of these exclusions files for each of the genres which has enough stories to build an anthology from (currently supernatural and sci-fi, and a couple of other shorter collections).

Then, I pre-process the stories using the transformations feature, via a transformations file like this:

Strip story numbers and preambles	(?m)^(#\s+\d+\s+\S+\s+)(.+)$(\n^(?!#\s[0-9]).*$)+?\n^-{3,}\s*$	# \2\n

This is due to Ulysses' export:
Remove footnote references				(?m)^\[\d+\]:\s+.+$\n

This gets rid of any stray footnote-style link references at the bottom of each file (Ulysses puts them there when exporting from its library, and I only ever have links in the preambles, which are being removed anyway), and of course strips out the story-numbers and preambles themselves. This transformations file is used for all anthologies, regardless of genre.

By creating an args file for each genre (which also overrides the subtitle metadata field), the invocation required to create, for example, a sci-fi anthology becomes just this:

python build-book.py @args-scifi.txt

And of course there are opportunities to further parameterise and simplify.

The overall anthology of all stories runs to more than 1,200 pages in 5x8 paperback format, so the args file for it builds only an epub file; printing a paperback of that length isn’t cost-effective, or indeed even possible for most print-on-demand services. The genre-specific anthologies, though, have a more reasonable page count, and so their respective args files also enable PDF output. As a further convenience, each format’s output book files have a corresponding filename, such as once-upon-a-time-scifi.epub and so on.

I can trigger all of this at any time, and it uses my existing corpus of Markdown files, entirely non-destructively, to create not just a full anthology but also a set of genre-specific ones too, in suitable formats, without intervention.

It certainly took some setup and programming, but it gives me full ownership of my data, no duplication or busywork, and it’ll run on pretty much any platform I care to use, now or in the future. The print layout in particular is also much more book-like than I was able to achieve before.

I find myself thinking of new features or tweaks every day or two, and the wrapper has developed into something quite powerful. Have a look at the documentation if you’re interested in trying it out.