You know what makes me sad? The use of Markdown as a lingua franca for interacting with AI tools, that’s what.

I understand that choice, because Markdown is, in general, a Good Thing. Or at the very least it is a vast improvement, for a vast number of use cases, over what were perhaps the most common formats prior to Gruber an’ all’s contributions: the nut-cracking sledgehammer known as Microsoft Word, and Some Random Plaintext Format I just Made Up. The fact that Markdown is so widely used shows just how welcome it is.

But the fact that we seem to have settled on Markdown in AI configuration means we may be missing an opportunity to deploy some far more suitable approaches. I was reminded of this during a recent discussion in Verilab about table formatting, which prompted one of the team (h/t, MH) to comment, “The limitations of Markdown always make me shed a tear for more powerful formats such as reST.”

I completely agree, although in my case the format whose neglect I mourn is the one used by Emacs Org mode. Org provides much the same kind of presentational power as Markdown, but with a richer semantic vocabulary (added, I should add, with very little additional cognitive load for the user).

The distinction that bothers me is not really one of formatting power. It is one of semantics. Markdown is very good at expressing the relatively limited semantics of ordinary documents: this is a heading; this is a list; this is code. But the things we increasingly want to package for AI systems have richer meanings: this is an instruction; this is metadata; this is evidence; this is a dependency; this has this provenance; this applies in this scope. We can, and routinely do, invent Markdown conventions for all of these. But at some point one has to wonder whether repeatedly building little semantic languages on top of Markdown is evidence that we chose the wrong abstraction in the first place.

And so the original comment on reST versus Markdown triggered thoughts of Orgdown, which in turn got me thinking of a fourth possible format, Wikitext. And then shoving its way into the mix for consideration came Google’s Open Knowledge Format. Admittedly it is a broader concept than the others, being more of a knowledge-packaging specification—a text format++, if you like—than merely a text format, but the fact that the chosen text format is a flavour of Markdown makes it fair game for comparison purposes. And what does one do these days when it becomes clear that a simple choice is much more nuanced than at first sight? One asks an AI, of course.

So Claude and I got busy, starting with my prompt of:

Compare and contrast¹ the following five formats: Markdown (in particular CommonMark), reST, Orgdown, the format used by Wikipedia, and Google’s Open Knowledge Format. Give a general comparison, but then focus it in on the specific use case of a standard for use in AI; for example for packaging and sharing curated context, metadata, and knowledge for AI agents, recording of skills and memories, and so on.

I won’t bore you by reproducing Claude’s response and our subsequent back-and-forth, part of which was based on noticing that the aforementioned blog post announcing OKF had actually described it as a formalization of Andrej Karpathy‘s LLM Wiki:

That’s why today, we’re introducing the Open Knowledge Format (OKF), an open specification that formalizes the LLM-wiki pattern into a portable, interoperable format.

That—Karpathy’s LLM Wiki—constituted a sixth tempting option in what was clearly a rapidly deepening and, more to the point, branching rabbit hole, and so I knew I had better get out quick, before it was Watership Down a-go-go. So, by way of escape, I decided to scribble this note.

My Claude discussion had two fairly simple conclusions:

1. Like my Verilab team-mate, I, too, shed a tear, although perhaps not quite for the same reason. Mine is for the fact that Markdown seems to have become the lingua franca of the ever-growing ecosystem surrounding LLMs, despite its native semantics being primarily those of ordinary human-readable documents.
2. But it could have been worse: we could have instead decided on LaTeX!

But even at that superficial level, perhaps the most interesting result of Claude’s five-way comparison was not that one format obviously won, but that they occupy rather different points in the space between human readability, syntactic simplicity, semantic expressiveness, extensibility, and machine tractability.

However, lurking beneath that surface is, I feel, something more intriguing. It is the possibility that we may be allowing historical accidents in human document formatting to determine the conventions by which we represent and exchange structured information with AI systems. In other words, AI may be creating exactly the conditions under which lightweight markup ought to evolve beyond Markdown, while simultaneously entrenching Markdown more deeply than ever, and all because Markdown happened to be everywhere when LLMs arrived!

1. That old favourite of examiners, which does seem to get good results when researching topics with an LLM. ↩︎