blogs/docs/PARSER.md

# Semantic Markdown Parser

This document describes the reusable parser contract for the blog-family
Markdown pipeline. It is intentionally project-independent: the same ideas
should be portable to future site families such as `neeldhara.courses` and
`books.neeldhara.com`.

The design goal is a single Markdown source that renders predictably to:

- Astro HTML, with React interactives where needed.
- PDF, through Pandoc and LuaLaTeX.
- Obsidian preview, with syntax that remains legible even when Obsidian does
  not understand a custom semantic feature.

## Principles

The source format is mostly ordinary Markdown. Custom syntax is introduced only
when the output needs semantic information that plain Markdown cannot carry.

Use these rules when adding new syntax:

- Prefer fenced attributes for semantic blocks. Pandoc understands them, and
  they are easy to parse in Astro.
- Keep labels explicit. A label such as `thm:hall` should map to a LaTeX
  `\label{thm:hall}` and an HTML `id="thm:hall"`.
- Use normal Markdown links for references. `[Hall's theorem](#thm:hall)`
  becomes an HTML anchor in Astro and `\cref{thm:hall}` in PDF.
- Make HTML and PDF fallbacks explicit for interactive material.
- Keep Obsidian-readable text in the source. A reader opening the vault should
  understand the note even without the final renderers.

## Current Implementation

The parser is split across an HTML path and a PDF path.

HTML path:

```text
Astro Markdown/MDX
  -> remark-gfm
  -> remarkInlineFootnotes
  -> remarkCodeFenceLanguages
  -> remarkSemanticBlocks
  -> remark-math
  -> remarkTypography
  -> remarkCallouts
  -> remarkSymbols
  -> rehype-katex
  -> Astro pages
```

PDF path:

```text
Markdown
  -> Pandoc markdown reader
  -> scripts/pandoc-filters/semantic-blocks.lua
  -> LuaLaTeX
  -> scripts/pandoc-templates/semantic-preamble.tex
```

The HTML parser lives per site at:

```text
sites/<site>/src/remark/
```

The current files are duplicated across the seven blog sites. When reusing this
system in another project, prefer extracting these files into a small shared
package or copying them together as a unit.

The PDF parser lives at:

```text
scripts/pandoc-filters/semantic-blocks.lua
scripts/pandoc-templates/semantic-preamble.tex
scripts/export-pdf.mjs
```

## Dependencies

Astro-side dependencies:

```text
@astrojs/mdx
@astrojs/react
rehype-katex
remark-gfm
remark-math
mdast-util-from-markdown
```

PDF-side dependencies:

```text
pandoc
lualatex
amsmath
amssymb
amsthm
cleveref
tcolorbox
emoji
chessfss
xcolor
```

For TeX Live installations, make sure the `emoji`, `chessfss`, `tcolorbox`,
and `cleveref` packages are installed.

## Base Markdown

The expected source is normal Markdown with YAML frontmatter:

```markdown
---
title: "Example"
description: "A short summary."
pubDate: "2026-01-31"
---

Body text.
```

Supported common Markdown features:

- Headings.
- Lists.
- Tables via `remark-gfm` in HTML and Pandoc's Markdown reader in PDF.
- Footnotes.
- Inline math with `$...$`.
- Display math with `$$...$$`.
- Code fences.
- Raw HTML for web-only fragments.

Use colocated relative assets:

```markdown
![A small diagram](diagram.png)
```

Avoid raw LaTeX in source unless the post is intended only for PDF.

## Typography

Write three hyphens for an em dash:

```markdown
This is one thought --- and this is another.
```

HTML converts text-node `---` into `—` through `remarkTypography`.

PDF leaves `---` in the generated LaTeX, where LuaLaTeX typesets it as an em
dash. This means the source stays ASCII-friendly while both outputs display the
proper dash.

## Callouts

Callouts are portable blockquotes. The callout type controls styling but does
not create a visible title by itself.

Supported types:

```text
note
tip
warning
caution
aside
```

Canonical titleless form:

```markdown
> **Note**
>
> This is a note without a visible heading.
```

Obsidian-style titleless form:

```markdown
> [!tip]
> This is a titleless tip.
```

Explicit title form:

```markdown
> [!warning: Check this assumption]
>
> This warning has a visible title.
```

The pipe spelling is equivalent:

```markdown
> [!warning|Check this assumption]
>
> This warning has a visible title.
```

Do not use this form for a title:

```markdown
> [!tip] Useful observation
```

That is intentionally interpreted as a titleless callout whose body begins with
"Useful observation". This rule keeps parsing unambiguous across Obsidian,
Astro, and Pandoc.

Quick marker forms are supported:

```markdown
> 📝 This becomes a note.
> 💡 This becomes a tip.
> ⚡ This becomes a tip.
> ⚠️ This becomes a warning.
```

HTML output:

```html
<aside class="callout callout-warning">
  <p class="callout-title">Check this assumption</p>
  <p>...</p>
</aside>
```

If no explicit title is present, no `.callout-title` paragraph is emitted.

PDF output:

```tex
\begin{blogwarningbox}{Check this assumption}
...
\end{blogwarningbox}
```

If no explicit title is present, the second environment argument is empty and
the `tcolorbox` title is omitted.

## Semantic Environments

Use semantic environments for theorem-like content, definitions, examples, and
proofs.

Canonical form:

````markdown
```{.env .theorem #thm:hall title="Hall's theorem" html="callout"}
Every bipartite graph satisfying Hall's condition has a matching that covers
the left side.
```
````

Supported environment types:

```text
theorem
lemma
proposition
corollary
definition
example
remark
proof
```

Attributes:

| Attribute        | Required    | Meaning                                      |
| ---------------- | ----------- | -------------------------------------------- |
| `.env`           | Recommended | Marks this as a semantic environment.        |
| `.<type>`        | Yes         | One of the supported environment types.      |
| `#label`         | Recommended | Cross-reference label and HTML id.           |
| `title="..."`    | Optional    | Environment title.                           |
| `html="callout"` | Optional    | Render as a styled block in HTML.            |
| `html="plain"`   | Optional    | Render as a quieter left-rule block in HTML. |

Default HTML mode:

- Theorem-like blocks default to `html="callout"`.
- `proof` defaults to `html="plain"`.

HTML output:

```html
<section
  id="thm:hall"
  class="semantic-env semantic-env-theorem semantic-env-callout callout"
  data-env-type="theorem"
  data-env-title="Hall's theorem"
  data-env-render="callout"
>
  <p class="semantic-env-heading"><strong>Theorem (Hall's theorem)</strong></p>
  ...
</section>
```

PDF output:

```tex
\begin{theorem}[Hall's theorem]
\label{thm:hall}
...
\end{theorem}
```

Short HTML-only form:

````markdown
```theorem id="thm:hall" title="Hall's theorem"
Every bipartite graph satisfying Hall's condition has a matching...
```
````

Use the canonical Pandoc-style form for any material that should export to PDF.

## Interactive Blocks

Interactive blocks reserve a place for a React component in HTML and provide a
text fallback for PDF and non-JavaScript output.

Canonical form:

````markdown
```{.interactive #int:tape-layout caption="Tape layout explorer"}
Drag files along a tape and compare the access cost of each arrangement.
```
````

Attributes:

| Attribute         | Required | Meaning                                              |
| ----------------- | -------- | ---------------------------------------------------- |
| `.interactive`    | Yes      | Marks this as an interactive block.                  |
| `#int:<id>`       | Yes      | Semantic label. The `int:` prefix is for references. |
| `caption="..."`   | Optional | Figure caption in HTML and PDF.                      |
| `title="..."`     | Optional | Fallback caption when `caption` is absent.           |
| `component="..."` | Optional | Override the component id.                           |
| `pdf="..."`       | Optional | PDF-specific replacement text.                       |
| `fallback="..."`  | Optional | General fallback text.                               |

Component resolution:

```text
#int:tape-layout
  -> component id: tape-layout
  -> component path: sites/<site>/src/interactives/tape-layout/index.tsx
```

HTML output:

```html
<figure class="semantic-interactive" id="int:tape-layout">
  <div
    class="interactive-mount"
    data-interactive-id="tape-layout"
    data-interactive-props="..."
  >
    <p class="interactive-fallback">...</p>
  </div>
  <figcaption>Tape layout explorer</figcaption>
</figure>
```

The runtime loaded by `InteractiveRuntime.astro` imports:

```text
sites/<site>/src/interactives/runtime.tsx
```

The runtime uses `import.meta.glob("./*/index.tsx")`, so each interactive
component should live in its own directory with an `index.tsx` entrypoint.

Component props:

```ts
type InteractiveProps = {
  id: string;
  label: string;
  caption: string;
  description: string;
  fallback: string;
  attrs: Record<string, string>;
};
```

PDF output:

```tex
\begin{figure}[htbp]
\centering
\fbox{\begin{minipage}{0.86\linewidth}
Fallback text...
\end{minipage}}
\caption{Tape layout explorer}
\label{int:tape-layout}
\end{figure}
```

Short HTML-only form:

````markdown
```interactive id="tape-layout" caption="Tape layout explorer"
Drag files along a tape and compare the access cost.
```
````

Use the canonical Pandoc-style form for any material that should export to PDF.

## Cross-References

Use ordinary Markdown links to ids:

```markdown
The proof of [Hall's theorem](#thm:hall) is constructive.
See the [tape explorer](#int:tape-layout) for the interactive version.
```

HTML keeps the link:

```html
<a href="#thm:hall">Hall's theorem</a>
```

PDF converts the link to:

```tex
\cref{thm:hall}
```

Recommended label prefixes:

```text
thm: theorem
lem: lemma
prop: proposition
cor: corollary
def: definition
ex: example
rem: remark
prf: proof
int: interactive
fig: figure
tbl: table
eq: equation
```

Do not reuse labels within one document.

## Inline Footnotes

The HTML parser supports inline footnotes:

```markdown
This is a sentence.^[This is the footnote.]
```

`remarkInlineFootnotes` rewrites these into normal Markdown footnote
definitions before rendering.

The parser also supports multi-node inline footnotes when formatting spans the
inside of the footnote.

For maximum PDF portability, ordinary footnotes are safest:

```markdown
This is a sentence.[^note]

[^note]: This is the footnote.
```

## Math

Use standard dollar math:

```markdown
Inline math: $a^2 + b^2 = c^2$.

$$
\sum_{i=1}^n i = \frac{n(n+1)}{2}
$$
```

HTML uses `remark-math` plus `rehype-katex`.

PDF passes math through Pandoc to LuaLaTeX.

Avoid math syntax that only one renderer understands.

## Symbols, Emoji, Chess, And Cards

Write Unicode directly in Markdown:

```markdown
The position after ♘f3 is pleasant.
Hearts and diamonds are red: ♥ ♦.
Spades and clubs are black: ♠ ♣.
This note is important 🎯.
```

HTML wraps these glyphs in semantic spans:

```html
<span
  class="symbol symbol-chess"
  data-symbol="white-knight"
  aria-label="white knight"
  >♘</span
>
<span class="symbol symbol-card" data-symbol="heart" aria-label="heart"
  >♥</span
>
<span class="symbol symbol-emoji">🎯</span>
```

PDF maps them to LaTeX macros:

```text
emoji and emoji sequences -> \BlogEmojiText{...}
♔ ♕ ♖ ♗ ♘ ♙       -> chessfss white-piece macros
♚ ♛ ♜ ♝ ♞ ♟       -> chessfss black-piece macros
♥ ♡ ♦ ♢           -> red amssymb suit macros
♠ ♤ ♣ ♧           -> black amssymb suit macros
playing-card emoji -> \BlogEmojiText{...}
```

Do not write LaTeX-only suit or chess macros in source unless the document is
PDF-only.

## Code Fences

The code-fence language parser normalizes a few imported Quarto forms:

````markdown
```{ojs}
viewof x = Inputs.range([0, 10])
```
````

becomes a JavaScript-highlighted fence:

````markdown
```js
viewof x = Inputs.range([0, 10])
```
````

This pass is intentionally conservative. Add aliases only when imported source
material needs them.

## Parser Ordering

Parser order matters.

Use this order for Astro:

```js
const remarkPlugins = [
  remarkGfm,
  remarkInlineFootnotes,
  remarkCodeFenceLanguages,
  remarkSemanticBlocks,
  remarkMath,
  remarkTypography,
  remarkCallouts,
  remarkSymbols,
];
const rehypePlugins = [rehypeKatex];
```

Reasons:

- Inline footnotes should be normalized before other structure changes.
- Code-fence languages should be normalized before semantic fences are read.
- Semantic blocks should run before code fences become highlighted code.
- Math should be parsed before typography and symbol wrapping can interfere.
- Typography should run before symbol wrapping.
- Callouts should run before symbol wrapping so marker detection sees plain
  text.
- Symbols should run late because it emits raw HTML spans.

## Astro Integration Checklist

Copy these files into the target Astro project:

```text
src/remark/callouts.mjs
src/remark/code-fence-languages.mjs
src/remark/inline-footnotes.mjs
src/remark/semantic-blocks.mjs
src/remark/symbols.mjs
src/remark/typography.mjs
src/interactives/runtime.tsx
src/components/InteractiveRuntime.astro
src/styles/article.css
```

Wire the plugins in `astro.config.mjs`:

```js
import mdx from "@astrojs/mdx";
import rehypeKatex from "rehype-katex";
import remarkGfm from "remark-gfm";
import remarkMath from "remark-math";
import remarkCallouts from "./src/remark/callouts.mjs";
import remarkCodeFenceLanguages from "./src/remark/code-fence-languages.mjs";
import remarkInlineFootnotes from "./src/remark/inline-footnotes.mjs";
import remarkSemanticBlocks from "./src/remark/semantic-blocks.mjs";
import remarkSymbols from "./src/remark/symbols.mjs";
import remarkTypography from "./src/remark/typography.mjs";

const remarkPlugins = [
  remarkGfm,
  remarkInlineFootnotes,
  remarkCodeFenceLanguages,
  remarkSemanticBlocks,
  remarkMath,
  remarkTypography,
  remarkCallouts,
  remarkSymbols,
];
const rehypePlugins = [rehypeKatex];

export default defineConfig({
  integrations: [mdx({ remarkPlugins, rehypePlugins })],
  markdown: { remarkPlugins, rehypePlugins },
});
```

Add the runtime to post pages:

```astro
---
import InteractiveRuntime from "@/components/InteractiveRuntime.astro";
---

<article class="article-content">
  <Content />
</article>
<InteractiveRuntime />
```

Make sure the article body uses the CSS class that `article.css` targets.

## PDF Integration Checklist

Copy these files:

```text
scripts/export-pdf.mjs
scripts/pandoc-filters/semantic-blocks.lua
scripts/pandoc-templates/semantic-preamble.tex
```

Add a package script:

```json
{
  "scripts": {
    "pdf:post": "node scripts/export-pdf.mjs"
  }
}
```

Export a post:

```bash
npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf
```

The export command currently uses:

```text
markdown+fenced_code_attributes+tex_math_dollars+tex_math_single_backslash+footnotes+raw_html
```

and includes the shared preamble with:

```text
--include-in-header scripts/pandoc-templates/semantic-preamble.tex
```

If LuaLaTeX cannot write its font cache in a sandboxed environment, set a
writable TeX cache:

```bash
TEXMFVAR=/tmp/texmf-var TEXMFCONFIG=/tmp/texmf-config \
  npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf
```

## Obsidian Behavior

The syntax is chosen to remain readable in Obsidian:

- Plain Markdown stays plain Markdown.
- `> [!tip]` and `> [!tip: Title]` render as Obsidian callouts.
- Fenced semantic blocks remain visible as code blocks when Obsidian does not
  know how to render them.
- Interactive blocks show their fallback description in the source.

Obsidian will not execute Astro interactives or render LaTeX environments as
final theorem boxes. That is acceptable: Obsidian is the editing surface, not
the final renderer.

## Adding A New Semantic Feature

When adding a feature, update all three layers:

1. Source syntax in this document.
2. Astro parser and CSS.
3. Pandoc Lua filter and LaTeX preamble.

Use this checklist:

- Define the Markdown syntax and at least one canonical example.
- Decide whether it needs a label and cross-reference behavior.
- Decide the HTML output shape and CSS classes.
- Decide the PDF output shape and required LaTeX packages.
- Add an Obsidian-readable fallback.
- Add a smoke-test Markdown file and render both HTML and PDF.
- Document migration rules for old content, if any.

## Known Tradeoffs

The current system is robust, but it has deliberate limits:

- It is a convention layer over Markdown, not a full new markup language.
- HTML and PDF are separate renderers. The contract is shared, but exact visual
  parity is not the goal.
- Interactives are HTML-first. PDF receives a figure-like fallback.
- Raw HTML is allowed for web-only material but should not be used for semantic
  features that need PDF output.
- The blog repo currently duplicates parser files across seven sites. A future
  shared parser package would reduce drift.

## Minimal Smoke Test

Use this sample when porting the parser to another project:

````markdown
---
title: "Parser Smoke Test"
description: "Checks callouts, environments, links, symbols, math, and interactives."
pubDate: "2026-01-31"
---

This sentence has an em dash --- and inline math $a^2+b^2=c^2$.

> [!tip: Explicit title]
>
> This callout has a title, a chess piece ♘, a card suit ♥, and an emoji 🎯.

```{.env .definition #def:test title="Test object" html="callout"}
A test object is something we can refer to later.
```

See [the definition](#def:test).

```{.interactive #int:test-widget caption="Test widget" pdf="The HTML version contains a small interactive test widget."}
This fallback appears when the interactive component is unavailable.
```
````

Expected HTML:

- One em dash.
- KaTeX-rendered math.
- A titled tip callout.
- A semantic definition block with `id="def:test"`.
- A normal anchor link to `#def:test`.
- A framed interactive fallback if no component exists.
- Wrapped symbol spans for `♘`, `♥`, and `🎯`.

Expected PDF:

- A LuaLaTeX em dash.
- Typeset math.
- A titled `tcolorbox` tip.
- A real `definition` environment with `\label{def:test}`.
- A `\cref{def:test}` reference.
- A figure fallback for `int:test-widget`.