Document semantic markdown parser

2026-06-26 04:17:18 +05:30 · 2026-06-26 04:17:18 +05:30 · 2849c65a3d
commit 2849c65a3d
parent b03d27cc3a
3 changed files with 793 additions and 0 deletions
--- a/docs/PARSER.md
+++ b/docs/PARSER.md
@ -0,0 +1,786 @@
+# Semantic Markdown Parser
+
+This document describes the reusable parser contract for the blog-family
+Markdown pipeline. It is intentionally project-independent: the same ideas
+should be portable to future site families such as `neeldhara.courses` and
+`books.neeldhara.com`.
+
+The design goal is a single Markdown source that renders predictably to:
+
+- Astro HTML, with React interactives where needed.
+- PDF, through Pandoc and LuaLaTeX.
+- Obsidian preview, with syntax that remains legible even when Obsidian does
+  not understand a custom semantic feature.
+
+## Principles
+
+The source format is mostly ordinary Markdown. Custom syntax is introduced only
+when the output needs semantic information that plain Markdown cannot carry.
+
+Use these rules when adding new syntax:
+
+- Prefer fenced attributes for semantic blocks. Pandoc understands them, and
+  they are easy to parse in Astro.
+- Keep labels explicit. A label such as `thm:hall` should map to a LaTeX
+  `\label{thm:hall}` and an HTML `id="thm:hall"`.
+- Use normal Markdown links for references. `[Hall's theorem](#thm:hall)`
+  becomes an HTML anchor in Astro and `\cref{thm:hall}` in PDF.
+- Make HTML and PDF fallbacks explicit for interactive material.
+- Keep Obsidian-readable text in the source. A reader opening the vault should
+  understand the note even without the final renderers.
+
+## Current Implementation
+
+The parser is split across an HTML path and a PDF path.
+
+HTML path:
+
+```text
+Astro Markdown/MDX
+  -> remark-gfm
+  -> remarkInlineFootnotes
+  -> remarkCodeFenceLanguages
+  -> remarkSemanticBlocks
+  -> remark-math
+  -> remarkTypography
+  -> remarkCallouts
+  -> remarkSymbols
+  -> rehype-katex
+  -> Astro pages
+```
+
+PDF path:
+
+```text
+Markdown
+  -> Pandoc markdown reader
+  -> scripts/pandoc-filters/semantic-blocks.lua
+  -> LuaLaTeX
+  -> scripts/pandoc-templates/semantic-preamble.tex
+```
+
+The HTML parser lives per site at:
+
+```text
+sites/<site>/src/remark/
+```
+
+The current files are duplicated across the seven blog sites. When reusing this
+system in another project, prefer extracting these files into a small shared
+package or copying them together as a unit.
+
+The PDF parser lives at:
+
+```text
+scripts/pandoc-filters/semantic-blocks.lua
+scripts/pandoc-templates/semantic-preamble.tex
+scripts/export-pdf.mjs
+```
+
+## Dependencies
+
+Astro-side dependencies:
+
+```text
+@astrojs/mdx
+@astrojs/react
+rehype-katex
+remark-gfm
+remark-math
+mdast-util-from-markdown
+```
+
+PDF-side dependencies:
+
+```text
+pandoc
+lualatex
+amsmath
+amssymb
+amsthm
+cleveref
+tcolorbox
+emoji
+chessfss
+xcolor
+```
+
+For TeX Live installations, make sure the `emoji`, `chessfss`, `tcolorbox`,
+and `cleveref` packages are installed.
+
+## Base Markdown
+
+The expected source is normal Markdown with YAML frontmatter:
+
+```markdown
+---
+title: "Example"
+description: "A short summary."
+pubDate: "2026-01-31"
+---
+
+Body text.
+```
+
+Supported common Markdown features:
+
+- Headings.
+- Lists.
+- Tables via `remark-gfm` in HTML and Pandoc's Markdown reader in PDF.
+- Footnotes.
+- Inline math with `$...$`.
+- Display math with `$$...$$`.
+- Code fences.
+- Raw HTML for web-only fragments.
+
+Use colocated relative assets:
+
+```markdown
+![A small diagram](diagram.png)
+```
+
+Avoid raw LaTeX in source unless the post is intended only for PDF.
+
+## Typography
+
+Write three hyphens for an em dash:
+
+```markdown
+This is one thought --- and this is another.
+```
+
+HTML converts text-node `---` into `—` through `remarkTypography`.
+
+PDF leaves `---` in the generated LaTeX, where LuaLaTeX typesets it as an em
+dash. This means the source stays ASCII-friendly while both outputs display the
+proper dash.
+
+## Callouts
+
+Callouts are portable blockquotes. The callout type controls styling but does
+not create a visible title by itself.
+
+Supported types:
+
+```text
+note
+tip
+warning
+caution
+aside
+```
+
+Canonical titleless form:
+
+```markdown
+> **Note**
+>
+> This is a note without a visible heading.
+```
+
+Obsidian-style titleless form:
+
+```markdown
+> [!tip]
+> This is a titleless tip.
+```
+
+Explicit title form:
+
+```markdown
+> [!warning: Check this assumption]
+>
+> This warning has a visible title.
+```
+
+The pipe spelling is equivalent:
+
+```markdown
+> [!warning|Check this assumption]
+>
+> This warning has a visible title.
+```
+
+Do not use this form for a title:
+
+```markdown
+> [!tip] Useful observation
+```
+
+That is intentionally interpreted as a titleless callout whose body begins with
+"Useful observation". This rule keeps parsing unambiguous across Obsidian,
+Astro, and Pandoc.
+
+Quick marker forms are supported:
+
+```markdown
+> 📝 This becomes a note.
+> 💡 This becomes a tip.
+> ⚡ This becomes a tip.
+> ⚠️ This becomes a warning.
+```
+
+HTML output:
+
+```html
+<aside class="callout callout-warning">
+  <p class="callout-title">Check this assumption</p>
+  <p>...</p>
+</aside>
+```
+
+If no explicit title is present, no `.callout-title` paragraph is emitted.
+
+PDF output:
+
+```tex
+\begin{blogwarningbox}{Check this assumption}
+...
+\end{blogwarningbox}
+```
+
+If no explicit title is present, the second environment argument is empty and
+the `tcolorbox` title is omitted.
+
+## Semantic Environments
+
+Use semantic environments for theorem-like content, definitions, examples, and
+proofs.
+
+Canonical form:
+
+````markdown
+```{.env .theorem #thm:hall title="Hall's theorem" html="callout"}
+Every bipartite graph satisfying Hall's condition has a matching that covers
+the left side.
+```
+````
+
+Supported environment types:
+
+```text
+theorem
+lemma
+proposition
+corollary
+definition
+example
+remark
+proof
+```
+
+Attributes:
+
+| Attribute        | Required    | Meaning                                      |
+| ---------------- | ----------- | -------------------------------------------- |
+| `.env`           | Recommended | Marks this as a semantic environment.        |
+| `.<type>`        | Yes         | One of the supported environment types.      |
+| `#label`         | Recommended | Cross-reference label and HTML id.           |
+| `title="..."`    | Optional    | Environment title.                           |
+| `html="callout"` | Optional    | Render as a styled block in HTML.            |
+| `html="plain"`   | Optional    | Render as a quieter left-rule block in HTML. |
+
+Default HTML mode:
+
+- Theorem-like blocks default to `html="callout"`.
+- `proof` defaults to `html="plain"`.
+
+HTML output:
+
+```html
+<section
+  id="thm:hall"
+  class="semantic-env semantic-env-theorem semantic-env-callout callout"
+  data-env-type="theorem"
+  data-env-title="Hall's theorem"
+  data-env-render="callout"
+>
+  <p class="semantic-env-heading"><strong>Theorem (Hall's theorem)</strong></p>
+  ...
+</section>
+```
+
+PDF output:
+
+```tex
+\begin{theorem}[Hall's theorem]
+\label{thm:hall}
+...
+\end{theorem}
+```
+
+Short HTML-only form:
+
+````markdown
+```theorem id="thm:hall" title="Hall's theorem"
+Every bipartite graph satisfying Hall's condition has a matching...
+```
+````
+
+Use the canonical Pandoc-style form for any material that should export to PDF.
+
+## Interactive Blocks
+
+Interactive blocks reserve a place for a React component in HTML and provide a
+text fallback for PDF and non-JavaScript output.
+
+Canonical form:
+
+````markdown
+```{.interactive #int:tape-layout caption="Tape layout explorer"}
+Drag files along a tape and compare the access cost of each arrangement.
+```
+````
+
+Attributes:
+
+| Attribute         | Required | Meaning                                              |
+| ----------------- | -------- | ---------------------------------------------------- |
+| `.interactive`    | Yes      | Marks this as an interactive block.                  |
+| `#int:<id>`       | Yes      | Semantic label. The `int:` prefix is for references. |
+| `caption="..."`   | Optional | Figure caption in HTML and PDF.                      |
+| `title="..."`     | Optional | Fallback caption when `caption` is absent.           |
+| `component="..."` | Optional | Override the component id.                           |
+| `pdf="..."`       | Optional | PDF-specific replacement text.                       |
+| `fallback="..."`  | Optional | General fallback text.                               |
+
+Component resolution:
+
+```text
+#int:tape-layout
+  -> component id: tape-layout
+  -> component path: sites/<site>/src/interactives/tape-layout/index.tsx
+```
+
+HTML output:
+
+```html
+<figure class="semantic-interactive" id="int:tape-layout">
+  <div
+    class="interactive-mount"
+    data-interactive-id="tape-layout"
+    data-interactive-props="..."
+  >
+    <p class="interactive-fallback">...</p>
+  </div>
+  <figcaption>Tape layout explorer</figcaption>
+</figure>
+```
+
+The runtime loaded by `InteractiveRuntime.astro` imports:
+
+```text
+sites/<site>/src/interactives/runtime.tsx
+```
+
+The runtime uses `import.meta.glob("./*/index.tsx")`, so each interactive
+component should live in its own directory with an `index.tsx` entrypoint.
+
+Component props:
+
+```ts
+type InteractiveProps = {
+  id: string;
+  label: string;
+  caption: string;
+  description: string;
+  fallback: string;
+  attrs: Record<string, string>;
+};
+```
+
+PDF output:
+
+```tex
+\begin{figure}[htbp]
+\centering
+\fbox{\begin{minipage}{0.86\linewidth}
+Fallback text...
+\end{minipage}}
+\caption{Tape layout explorer}
+\label{int:tape-layout}
+\end{figure}
+```
+
+Short HTML-only form:
+
+````markdown
+```interactive id="tape-layout" caption="Tape layout explorer"
+Drag files along a tape and compare the access cost.
+```
+````
+
+Use the canonical Pandoc-style form for any material that should export to PDF.
+
+## Cross-References
+
+Use ordinary Markdown links to ids:
+
+```markdown
+The proof of [Hall's theorem](#thm:hall) is constructive.
+See the [tape explorer](#int:tape-layout) for the interactive version.
+```
+
+HTML keeps the link:
+
+```html
+<a href="#thm:hall">Hall's theorem</a>
+```
+
+PDF converts the link to:
+
+```tex
+\cref{thm:hall}
+```
+
+Recommended label prefixes:
+
+```text
+thm: theorem
+lem: lemma
+prop: proposition
+cor: corollary
+def: definition
+ex: example
+rem: remark
+prf: proof
+int: interactive
+fig: figure
+tbl: table
+eq: equation
+```
+
+Do not reuse labels within one document.
+
+## Inline Footnotes
+
+The HTML parser supports inline footnotes:
+
+```markdown
+This is a sentence.^[This is the footnote.]
+```
+
+`remarkInlineFootnotes` rewrites these into normal Markdown footnote
+definitions before rendering.
+
+The parser also supports multi-node inline footnotes when formatting spans the
+inside of the footnote.
+
+For maximum PDF portability, ordinary footnotes are safest:
+
+```markdown
+This is a sentence.[^note]
+
+[^note]: This is the footnote.
+```
+
+## Math
+
+Use standard dollar math:
+
+```markdown
+Inline math: $a^2 + b^2 = c^2$.
+
+$$
+\sum_{i=1}^n i = \frac{n(n+1)}{2}
+$$
+```
+
+HTML uses `remark-math` plus `rehype-katex`.
+
+PDF passes math through Pandoc to LuaLaTeX.
+
+Avoid math syntax that only one renderer understands.
+
+## Symbols, Emoji, Chess, And Cards
+
+Write Unicode directly in Markdown:
+
+```markdown
+The position after ♘f3 is pleasant.
+Hearts and diamonds are red: ♥ ♦.
+Spades and clubs are black: ♠ ♣.
+This note is important 🎯.
+```
+
+HTML wraps these glyphs in semantic spans:
+
+```html
+<span
+  class="symbol symbol-chess"
+  data-symbol="white-knight"
+  aria-label="white knight"
+  >♘</span
+>
+<span class="symbol symbol-card" data-symbol="heart" aria-label="heart"
+  >♥</span
+>
+<span class="symbol symbol-emoji">🎯</span>
+```
+
+PDF maps them to LaTeX macros:
+
+```text
+emoji and emoji sequences -> \BlogEmojiText{...}
+♔ ♕ ♖ ♗ ♘ ♙       -> chessfss white-piece macros
+♚ ♛ ♜ ♝ ♞ ♟       -> chessfss black-piece macros
+♥ ♡ ♦ ♢           -> red amssymb suit macros
+♠ ♤ ♣ ♧           -> black amssymb suit macros
+playing-card emoji -> \BlogEmojiText{...}
+```
+
+Do not write LaTeX-only suit or chess macros in source unless the document is
+PDF-only.
+
+## Code Fences
+
+The code-fence language parser normalizes a few imported Quarto forms:
+
+````markdown
+```{ojs}
+viewof x = Inputs.range([0, 10])
+```
+````
+
+becomes a JavaScript-highlighted fence:
+
+````markdown
+```js
+viewof x = Inputs.range([0, 10])
+```
+````
+
+This pass is intentionally conservative. Add aliases only when imported source
+material needs them.
+
+## Parser Ordering
+
+Parser order matters.
+
+Use this order for Astro:
+
+```js
+const remarkPlugins = [
+  remarkGfm,
+  remarkInlineFootnotes,
+  remarkCodeFenceLanguages,
+  remarkSemanticBlocks,
+  remarkMath,
+  remarkTypography,
+  remarkCallouts,
+  remarkSymbols,
+];
+const rehypePlugins = [rehypeKatex];
+```
+
+Reasons:
+
+- Inline footnotes should be normalized before other structure changes.
+- Code-fence languages should be normalized before semantic fences are read.
+- Semantic blocks should run before code fences become highlighted code.
+- Math should be parsed before typography and symbol wrapping can interfere.
+- Typography should run before symbol wrapping.
+- Callouts should run before symbol wrapping so marker detection sees plain
+  text.
+- Symbols should run late because it emits raw HTML spans.
+
+## Astro Integration Checklist
+
+Copy these files into the target Astro project:
+
+```text
+src/remark/callouts.mjs
+src/remark/code-fence-languages.mjs
+src/remark/inline-footnotes.mjs
+src/remark/semantic-blocks.mjs
+src/remark/symbols.mjs
+src/remark/typography.mjs
+src/interactives/runtime.tsx
+src/components/InteractiveRuntime.astro
+src/styles/article.css
+```
+
+Wire the plugins in `astro.config.mjs`:
+
+```js
+import mdx from "@astrojs/mdx";
+import rehypeKatex from "rehype-katex";
+import remarkGfm from "remark-gfm";
+import remarkMath from "remark-math";
+import remarkCallouts from "./src/remark/callouts.mjs";
+import remarkCodeFenceLanguages from "./src/remark/code-fence-languages.mjs";
+import remarkInlineFootnotes from "./src/remark/inline-footnotes.mjs";
+import remarkSemanticBlocks from "./src/remark/semantic-blocks.mjs";
+import remarkSymbols from "./src/remark/symbols.mjs";
+import remarkTypography from "./src/remark/typography.mjs";
+
+const remarkPlugins = [
+  remarkGfm,
+  remarkInlineFootnotes,
+  remarkCodeFenceLanguages,
+  remarkSemanticBlocks,
+  remarkMath,
+  remarkTypography,
+  remarkCallouts,
+  remarkSymbols,
+];
+const rehypePlugins = [rehypeKatex];
+
+export default defineConfig({
+  integrations: [mdx({ remarkPlugins, rehypePlugins })],
+  markdown: { remarkPlugins, rehypePlugins },
+});
+```
+
+Add the runtime to post pages:
+
+```astro
+---
+import InteractiveRuntime from "@/components/InteractiveRuntime.astro";
+---
+
+<article class="article-content">
+  <Content />
+</article>
+<InteractiveRuntime />
+```
+
+Make sure the article body uses the CSS class that `article.css` targets.
+
+## PDF Integration Checklist
+
+Copy these files:
+
+```text
+scripts/export-pdf.mjs
+scripts/pandoc-filters/semantic-blocks.lua
+scripts/pandoc-templates/semantic-preamble.tex
+```
+
+Add a package script:
+
+```json
+{
+  "scripts": {
+    "pdf:post": "node scripts/export-pdf.mjs"
+  }
+}
+```
+
+Export a post:
+
+```bash
+npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf
+```
+
+The export command currently uses:
+
+```text
+markdown+fenced_code_attributes+tex_math_dollars+tex_math_single_backslash+footnotes+raw_html
+```
+
+and includes the shared preamble with:
+
+```text
+--include-in-header scripts/pandoc-templates/semantic-preamble.tex
+```
+
+If LuaLaTeX cannot write its font cache in a sandboxed environment, set a
+writable TeX cache:
+
+```bash
+TEXMFVAR=/tmp/texmf-var TEXMFCONFIG=/tmp/texmf-config \
+  npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf
+```
+
+## Obsidian Behavior
+
+The syntax is chosen to remain readable in Obsidian:
+
+- Plain Markdown stays plain Markdown.
+- `> [!tip]` and `> [!tip: Title]` render as Obsidian callouts.
+- Fenced semantic blocks remain visible as code blocks when Obsidian does not
+  know how to render them.
+- Interactive blocks show their fallback description in the source.
+
+Obsidian will not execute Astro interactives or render LaTeX environments as
+final theorem boxes. That is acceptable: Obsidian is the editing surface, not
+the final renderer.
+
+## Adding A New Semantic Feature
+
+When adding a feature, update all three layers:
+
+1. Source syntax in this document.
+2. Astro parser and CSS.
+3. Pandoc Lua filter and LaTeX preamble.
+
+Use this checklist:
+
+- Define the Markdown syntax and at least one canonical example.
+- Decide whether it needs a label and cross-reference behavior.
+- Decide the HTML output shape and CSS classes.
+- Decide the PDF output shape and required LaTeX packages.
+- Add an Obsidian-readable fallback.
+- Add a smoke-test Markdown file and render both HTML and PDF.
+- Document migration rules for old content, if any.
+
+## Known Tradeoffs
+
+The current system is robust, but it has deliberate limits:
+
+- It is a convention layer over Markdown, not a full new markup language.
+- HTML and PDF are separate renderers. The contract is shared, but exact visual
+  parity is not the goal.
+- Interactives are HTML-first. PDF receives a figure-like fallback.
+- Raw HTML is allowed for web-only material but should not be used for semantic
+  features that need PDF output.
+- The blog repo currently duplicates parser files across seven sites. A future
+  shared parser package would reduce drift.
+
+## Minimal Smoke Test
+
+Use this sample when porting the parser to another project:
+
+````markdown
+---
+title: "Parser Smoke Test"
+description: "Checks callouts, environments, links, symbols, math, and interactives."
+pubDate: "2026-01-31"
+---
+
+This sentence has an em dash --- and inline math $a^2+b^2=c^2$.
+
+> [!tip: Explicit title]
+>
+> This callout has a title, a chess piece ♘, a card suit ♥, and an emoji 🎯.
+
+```{.env .definition #def:test title="Test object" html="callout"}
+A test object is something we can refer to later.
+```
+
+See [the definition](#def:test).
+
+```{.interactive #int:test-widget caption="Test widget" pdf="The HTML version contains a small interactive test widget."}
+This fallback appears when the interactive component is unavailable.
+```
+````
+
+Expected HTML:
+
+- One em dash.
+- KaTeX-rendered math.
+- A titled tip callout.
+- A semantic definition block with `id="def:test"`.
+- A normal anchor link to `#def:test`.
+- A framed interactive fallback if no component exists.
+- Wrapped symbol spans for `♘`, `♥`, and `🎯`.
+
+Expected PDF:
+
+- A LuaLaTeX em dash.
+- Typeset math.
+- A titled `tcolorbox` tip.
+- A real `definition` environment with `\label{def:test}`.
+- A `\cref{def:test}` reference.
+- A figure fallback for `int:test-widget`.