This is the story of how my latest novel tried to implode under the weight of its own subplots, and Emacs’s org-mode rescued it.

If you’re an author who loves geeking out on relatively technical writing tools, or an engineer with a fondness for textual formats / an interest in prose composition, you might find something fun or interesting in this blog post.

If you’ve heard of this whole “org-mode” thing and always wanted to learn more about it—or if you’re already a devotee of the glorious Textual Swiss Army Knife that is org-mode—you’ll almost certainly find something fun or interesting in this blog post.

But if, by any chance, you’re planning to write a comedic sci-fi novel set on a near-light-speed luxury cruise spaceliner and involving at least a baker’s dozen complex, interwined subplots—then this post just might save your life (or at least your book).

So, Who am I?

For the last 20 years, I’ve put bread on the table working as a programmer (I’m currently at Wayfair, leading a team that’s building out the systems to support our custom logistics network). In the evenings I enjoy writing fiction (two novels and a book of short stories to my name so far) and applying the tools of my day job to the challenges of the independent author (e.g. git for version control, the odd Awk script to format a manuscript differently for paper and electronic editions, etc.).

I am also, in author terms, what is known as a “pantser.” Don’t worry, this entire post is totally safe for work.

Pantsers and Plotters

In order to understand how my novel found itself in dire straits, it’s useful to understand the commonly accepted division of authors into “plotters” (those who carefully work out the outline of a novel before they start writing the prose) and “pantsers” (those who jump straight into the prose with only a vague sense of where things are headed, and figure out the structure of the story by actually writing it—and, more importantly, rewriting it).

I’ve tried to write a novel from an outline a few times, but inevitably I’ve given up and gone back to pantsing. And although I always felt vaguely guilty about my apparent inability to create an outline and work from it¹, I had never experienced that inability as a handicap per se.

Writing the first draft of my latest novel, called World Enough (And Time), did stress my pantser approach a bit. The book is a sci-fi comedy of manners, and the first draft had five or six interlocking subplots—misunderstandings in subplot A driving subplot B, objects lost in C popping up in A, and so on. Keeping all those plots straight was a decent lift—a heavier one than my previous books, which tended to have two or three relatively independent subplots—but I pantsed my way over the finish line feeling pretty good.

Reading over the first draft, I saw a couple places where existing subplots could be modified or combined. I also had ideas for a bunch of new subplots to mix in, which promised to make the story even twistier and more eventful and interconnected.

I was excited at this prospect.

I started to revise.

Revising a Novel with No Outline is O(N!)

In my professional life as a software engineer I have refactored codebases that would have made spaghetti blush. I have rearchitected systems piece by piece over months with no downtime², rebuilding, as they say, the engine while the plane is still in flight. I have pulled apart data pipelines coursing with the information-theoretic equivalent of raw sewage and put them back together again.

None of those were as hard as revising this damned novel.

Every change, however small, risked introducing vast, cascading continuity errors. The novel was supposed to feature relativistic time dilation, but instead—unintentionally—the story became a yarn-ball of impossible timelines and violated causality. Nonsensical mentions were made of events belonging to abandoned subplots. Items emerged inexplicably from a character’s pocket, where they had never been, only to be placed there chapters later. Threats were carried out, retracted, then made in the first place.

The only way to ensure that a change had not introduced continuity errors was, in effect, to read through the novel in its entirety—and if I did find an error, to start reading from the beginning once again to ensure that my correction had not created new, unintended consequences. It was chaos. It was madness. It was C-x M-c M-butterfly.

A small dose of this insanity was sufficient to convince me to try the formerly unthinkable: put down the manuscript, burn my official pantser membership card, and create an outline³.

Revising a Novel Against an External Outline is O(N2)

For my first attempt, I created an outline in Google sheets—with rows as chapters and columns as subplots—and at first blush, matters seemed markedly improved. Working directly on the outline, I could edit and move events around much more quickly. Maybe, I thought, the plotters had been right all along.

But then, inevitably, the outline and the prose would drift from each other. In a revision session, I would try to remember what changes I had made in the outline, so that I could mirror them in the manuscript. But along the way, inspiration would inevitably strike (bless my little pantser heart), I would make a few tweaks in the manuscript—and all of a sudden I now had to update the outline in return to reflect the changes I had just made in the prose. If I missed keeping the manuscript and outline in sync with each other—even once—then the only way to correct the problem was to read through the entire novel again and true it up against the Google sheets outline.

Things were better, but not better enough. I was still revising like molasses, and it was going to take me years to finish this book.

At this point I settled for a couple weeks into what could charitably be described as a deep think—or more accurately as a raging, extended funk.

Enter Org-Mode

Org-mode is a bit hard to explain. Or, perhaps more accurately, it’s very easy to explain (“It’s an Emacs mode for note-taking, outlining and todo management mode, which has somehow grown into an excellent text composition suite, a website publishing system, a literate programming tool, and much more”), but it’s very hard to convey why that apparently random grab-bag of functionality is anything other than a terrible idea, or how powerful org-mode feels once you’re able to use it well.

Those who learn it tend to find it growing into more and more areas of their life. The org-mode slogan is, in fact, “your life in plain text,” and for me at least that’s pretty accurate. I keep my projects and todos in org-mode. I take notes in it during meetings. I brainstorm in it, write emails in it, create slides in it, keep recipes in it, and much more. In fact, about the only text I hadn’t been in the habit of composing in org-mode was fiction (because markdown integrated more deeply with pandoc, which is the workhorse of my e-book and paper publishing workflows).

But then, one evening, as I was brainstorming solutions to my subplot revision conundrum in an org-mode document, I realized that the answer was literally staring me right in the face.

Headlines, Text, and Tags (Oh My)

Org-mode has more features than I could possibly hope to convey in this post, but the two structural concepts that are key to understanding how it saved my novel are headlines and tags.

Headlines are hierarchical section names—roughly equivalent to header tags, if you’re familiar with HTML. But org-mode headlines are much more strongly and structurally associated with the blocks of text that follow them than HTML headers are—in that sense, they’re more like HTML divs with built-in titles. That is, if you parsed an HTML document with some headers and subsequent paragraphs, you’d get something like this:

header1: Some section
paragraph: Paragraph under section
header2: Some sub-section
paragraph: Paragraph under sub-section

In org-mode, you’d get something more tree-like:

headline1: Some section
   text: Paragraph under section
   headline2: Some sub-section
      text: Paragraph under sub-section

Org-mode tags are free-text labels that can be applied to any headline, and which can then be used to search, sort, or create views over subsets of tagged headlines.

Let’s put this all together and show a simple org-mode file with a couple tagged headlines and some associated text.

* This is a headline  :tag1:tag2:

Here's some awesome text associated with the top headline.

And some more.

** This is a sub-headline  :tag1:tag3:

And it has some text of its own, how lovely.

* Another top-level headline  :tag3:

And so on.

Getting the Novel into Org-Mode

My basic idea was this: if I made chapters into top-level headlines, scenes within those chapters into second-level headlines, and subplots into tags on scenes, I could take advantage of org-mode’s strong structural tools and rich tag queries to drastically reduce the amount of effort it required to maintain causality in my novel. Don’t worry, we’re going to walk through in detail exactly how this worked.

The first step to using org-mode was actually porting the markdown draft of the novel to an org-mode document structured as above (chapters as top-level headlines, scenes as second-level headlines, plots as tags). This took a fair amount of work, mostly because I had to break up the scenes in each chapter and tag each scene with every plot that it affected or depended on.

When I was done, the manuscript looked something like this.

* Chapter 1: A State of Imminent Default

** Grubel tells Jeremiah about his uncle / work  :grubel:reviews:

"Ferrets?" said Jeremiah Brown.  "I don't remember Uncle Leo particularly /liking/ ferrets."

"People change in 20 years, Mr. Brown," said Mr. Grubel.

"Uncle Leo hardly changed his socks.  Besides, can you even maltreat a ferret?"

Mr. Grubel pushed his lensless silver glasses further up the bridge of his nose.  His hypermodern silver suit shimmered as he squared his shoulders.

"You can maltreat anything," he said.

...

To make sure it’s hyper clear, in the above:

• “Chapter 1: A State of Imminent Default” is a top-level headline, corresponding to a chapter in the book. This title will appear in the finished text.
• “Grubel tells Jeremiah about his uncle / work” is a second-level headline, corresponding to a scene within the chapter. The name of the scene is intended only for me, the author, and will not appear in the finished text (we’ll accomplish that through some light pandoc magic).
• The text starting with “Ferrets?” is the body of the second-level headline, and the actual prose of the scene.
• The tags associated with that second-level headline—“grubel” and “reviews”—indicate the subplots that this scene impacts.

Revising a Novel Against a Tagged, Internal Outline is O(N)

So why was this org-mode structure any better than the approaches I had tried before?

There were three key features to this approach that worked together to save me from tracking down continuity errors until the heat death of the universe.

Outlines are Inline

First, the outline was actually inline with the text of the novel. Unlike shuffling chapters and plots around in my Google sheet and then having to make the same moves separately in the prose, I could move the headlines and blocks of text at the same time—they couldn’t drift from each other.

This process was made even more convenient with org-mode’s visibility cycling, which allows you to fold up the tree structure of the document to any arbitrary headline level.

For example, I could use org-mode’s visibility cycling to fold up to second-level headlines, corresponding to scenes, like so:

* Chapter 1: A State of Imminent Default
** Grubel tells Jeremiah about his uncle / work  :grubel:reviews:
** Reynolds comes to pick up Jeremiah  :reynolds:
* Chapter 2: Now Serving Number
** Reynolds explains job to Jeremiah  :reynolds:
** Porter asks for help with PED  :porter:backgammon:
...

In this view, I can quickly read the scenes with the text folded up beneath them, and—if I decide to move some scenes with cut and paste—the chunks of prose, folded below, will move right along with their headlines.

Tag Queries and Sparse Trees FTW

Second, using org-mode tags to designate subplots let me manage subplots with org-mode’s powerful tag tools. Org-mode allows you to query against the tags in a file with a relatively complex set of operators, so it was trivial to e.g. find all the scenes that impacted the “wendstrom” subplot, or to find all the scenes that impacted either “reynolds” or “katherine”—or even all the scenes that impacted “grubel” and “reviews” but not “inheritance”.

Querying for tags is powerful, but especially useful for solving my continuity problem was how org-mode displayed the result of a query: as a so-called sparse tree.

I find sparse trees are easier to understand through an example rather than a description. Let’s say we’ve got the following org-mode file, set up in our scheme with chapters, scenes, and subplots.

* This is chapter 1

** This is scene 1.1  :plot1:plot2:

This is the text of scene 1.1.

** This is scene 1.2  :plot1:plot3:

This is the text of scene 1.2.

* This is chapter 2

** This is scene 2.1  :plot3:plot4:

This is the text of scene 1.1.

** This is scene 2.2  :plot2:plot3:

This is the text of scene 1.2.

If we run org-sparse-tree and match on a query of plot2, we get a view over the file where only headings tagged with plot2 (and their parents) are visible, and the scene text has been folded up. You’ll note that scenes 1.2 and 2.1, which aren’t tagged with plot2, have been made invisible to reduce visual clutter (making the tree “sparse”).

* This is chapter 1
** This is scene 1.1  :plot1:plot2:
* This is chapter 2
** This is scene 2.2  :plot2:plot3:

Let’s imagine I had been moving around scenes involved in two subplots—“reynolds” and “necklace”. After I’ve hacked and moved things around for a while, I can now run org-sparse-tree and query reynolds|necklace. The resulting view would show me the entire novel folded up to second-level headlines, with only those scenes involving the “reynolds” or “necklace” subplots visible, and all other clutter removed. In all of 60 seconds I could read down the headlines of the scenes and make sure that continuity of the outline for those subplots hadn’t been violated by any recent changes.

Locality Makes Everything Easier

The third key property of org-mode that made revising so much easier was that, as we’ve noted, headlines stayed right above the prose of the scenes they described when moved around. This property drastically reduced the effort required to make sure that the headlines (which I depended on being correct descriptions of the scenes) didn’t drift from the prose. After a marathon session of moving around and editing multiple headlines, or tweaking a bunch of prose, I only had to read through each scene once, making sure that the prose was still in sync with the headline description right above it (instead of, for example, in an external spreadsheet, and perhaps now in the wrong row of that spreadsheet if I had moved the scene to a different chapter). As an extra benefit, the fact that my outline and the prose were both textual meant that I could lean on git to see what headlines and prose I had changed, and focus my efforts on just those areas.

The sensation of relief I experienced in my first few editing sessions after making the switch to org-mode and this headline / tagging strategy reminded me of the feeling of choosing the right data structure in programming—how suddenly, with the move from a linked list to a hashmap, or an array to a set, half of the problem you’ve been hammering on just seems to melt away.

I stuck with the org-mode approach through the next year, revising and re-revising, adding and removing subplots, collapsing and shuffling them, and it served me extremely well, right through the final draft of World Enough (And Time).

The Downsides of Using org-mode for Fiction

The org-mode experience for composing and revising fiction in Emacs was amazing, and one that I plan to repeat with my next book. Once it came time to render drafts for external readers, however, a few annoyances did pop up.

First of all, I had to keep the second-level headlines—the ones with the scene descriptions and tags—out of the manuscript that others would read, and org-mode doesn’t make it easy to suppress a headline while still passing its body through when exporting to a different format⁴.

There were a couple solutions to this, including just adding a grep -v '^\*\*' to the Makefile I use to build e-books and pdfs. I ended up using a pandoc filter, as that approach still worked even afer the manuscript had been converted back to markdown.

That’s right, I said “converted back to markdown.” Once I had finished incorporating feedback from successive rounds of readers, and started to focus on producing the e-book and print edition, I did convert the manuscript back to markdown, which still had substantial advantages over org-mode when it came to production and formatting.

For example, trying to italicize just part of a word in org-mode is a trail of tears, involving endless fiddling with the largely undocumented and widely feared org-emphasis-regexp-components.

Nitty styling of an e-book’s HTML is also considerably easier in pandoc’s native markdown dialect than org-mode (in part because pandoc’s org parser doesn’t know about org-emphasis-regexp-components and the other org-mode nonsense you need to bend org-mode’s complex formatting to your will)⁵.

In Conclusion

Org-mode’s ability to keep the headlines of an outline embedded in your prose, and its rich tagging system, make it a powerful tool for editing and revising fiction with multiple intertwined subplots. Composing and revising in org-mode, then doing a one-time port to markdown once you’re ready to produce the book, lets you take advantage of the strengths of each format with the relatively low effort of a single conversion. If you find yourself writing a densely interconnected story and drowning in continuity errors, it’s worth giving this approach a try.

(And if you’re a fan of Douglas Adams and Jasper Fforde, or have always longed for a Wodehousian story set on a spaceship toodling along at an appreciable fraction of the speed of light—or if you’re just curious to see how a novel with a final count of 19 subplots wrangled in org-mode turned out—give World Enough (And Time) a shot.)

Back to posts