diff --git a/CONTENT_BRIDGE.md b/CONTENT_BRIDGE.md index 9e3013d..48a11f0 100644 --- a/CONTENT_BRIDGE.md +++ b/CONTENT_BRIDGE.md @@ -83,6 +83,10 @@ strips unknown schema fields, but they are useful in Obsidian for provenance. Standard Markdown headings, lists, links, tables, code fences, footnotes, and raw HTML are preserved. +This section is the short writer-facing version. The reusable implementation +contract for future site families is documented in +[`docs/PARSER.md`](./docs/PARSER.md). + Use colocated relative assets: ```markdown diff --git a/README.md b/README.md index f581e73..5d5a86c 100644 --- a/README.md +++ b/README.md @@ -67,3 +67,6 @@ The seven blog content folders are synced from the public Obsidian vault. See [CONTENT_BRIDGE.md](./CONTENT_BRIDGE.md) for conventions, preview/publish commands, and the Quarto import workflow. See [IMPORT_REPORT.md](./IMPORT_REPORT.md) for the current imported post mapping. + +For the reusable Markdown parser contract shared by Astro HTML and Pandoc/PDF, +see [docs/PARSER.md](./docs/PARSER.md). diff --git a/docs/PARSER.md b/docs/PARSER.md new file mode 100644 index 0000000..d16cfbf --- /dev/null +++ b/docs/PARSER.md @@ -0,0 +1,786 @@ +# Semantic Markdown Parser + +This document describes the reusable parser contract for the blog-family +Markdown pipeline. It is intentionally project-independent: the same ideas +should be portable to future site families such as `neeldhara.courses` and +`books.neeldhara.com`. + +The design goal is a single Markdown source that renders predictably to: + +- Astro HTML, with React interactives where needed. +- PDF, through Pandoc and LuaLaTeX. +- Obsidian preview, with syntax that remains legible even when Obsidian does + not understand a custom semantic feature. + +## Principles + +The source format is mostly ordinary Markdown. Custom syntax is introduced only +when the output needs semantic information that plain Markdown cannot carry. + +Use these rules when adding new syntax: + +- Prefer fenced attributes for semantic blocks. Pandoc understands them, and + they are easy to parse in Astro. +- Keep labels explicit. A label such as `thm:hall` should map to a LaTeX + `\label{thm:hall}` and an HTML `id="thm:hall"`. +- Use normal Markdown links for references. `[Hall's theorem](#thm:hall)` + becomes an HTML anchor in Astro and `\cref{thm:hall}` in PDF. +- Make HTML and PDF fallbacks explicit for interactive material. +- Keep Obsidian-readable text in the source. A reader opening the vault should + understand the note even without the final renderers. + +## Current Implementation + +The parser is split across an HTML path and a PDF path. + +HTML path: + +```text +Astro Markdown/MDX + -> remark-gfm + -> remarkInlineFootnotes + -> remarkCodeFenceLanguages + -> remarkSemanticBlocks + -> remark-math + -> remarkTypography + -> remarkCallouts + -> remarkSymbols + -> rehype-katex + -> Astro pages +``` + +PDF path: + +```text +Markdown + -> Pandoc markdown reader + -> scripts/pandoc-filters/semantic-blocks.lua + -> LuaLaTeX + -> scripts/pandoc-templates/semantic-preamble.tex +``` + +The HTML parser lives per site at: + +```text +sites//src/remark/ +``` + +The current files are duplicated across the seven blog sites. When reusing this +system in another project, prefer extracting these files into a small shared +package or copying them together as a unit. + +The PDF parser lives at: + +```text +scripts/pandoc-filters/semantic-blocks.lua +scripts/pandoc-templates/semantic-preamble.tex +scripts/export-pdf.mjs +``` + +## Dependencies + +Astro-side dependencies: + +```text +@astrojs/mdx +@astrojs/react +rehype-katex +remark-gfm +remark-math +mdast-util-from-markdown +``` + +PDF-side dependencies: + +```text +pandoc +lualatex +amsmath +amssymb +amsthm +cleveref +tcolorbox +emoji +chessfss +xcolor +``` + +For TeX Live installations, make sure the `emoji`, `chessfss`, `tcolorbox`, +and `cleveref` packages are installed. + +## Base Markdown + +The expected source is normal Markdown with YAML frontmatter: + +```markdown +--- +title: "Example" +description: "A short summary." +pubDate: "2026-01-31" +--- + +Body text. +``` + +Supported common Markdown features: + +- Headings. +- Lists. +- Tables via `remark-gfm` in HTML and Pandoc's Markdown reader in PDF. +- Footnotes. +- Inline math with `$...$`. +- Display math with `$$...$$`. +- Code fences. +- Raw HTML for web-only fragments. + +Use colocated relative assets: + +```markdown +![A small diagram](diagram.png) +``` + +Avoid raw LaTeX in source unless the post is intended only for PDF. + +## Typography + +Write three hyphens for an em dash: + +```markdown +This is one thought --- and this is another. +``` + +HTML converts text-node `---` into `—` through `remarkTypography`. + +PDF leaves `---` in the generated LaTeX, where LuaLaTeX typesets it as an em +dash. This means the source stays ASCII-friendly while both outputs display the +proper dash. + +## Callouts + +Callouts are portable blockquotes. The callout type controls styling but does +not create a visible title by itself. + +Supported types: + +```text +note +tip +warning +caution +aside +``` + +Canonical titleless form: + +```markdown +> **Note** +> +> This is a note without a visible heading. +``` + +Obsidian-style titleless form: + +```markdown +> [!tip] +> This is a titleless tip. +``` + +Explicit title form: + +```markdown +> [!warning: Check this assumption] +> +> This warning has a visible title. +``` + +The pipe spelling is equivalent: + +```markdown +> [!warning|Check this assumption] +> +> This warning has a visible title. +``` + +Do not use this form for a title: + +```markdown +> [!tip] Useful observation +``` + +That is intentionally interpreted as a titleless callout whose body begins with +"Useful observation". This rule keeps parsing unambiguous across Obsidian, +Astro, and Pandoc. + +Quick marker forms are supported: + +```markdown +> 📝 This becomes a note. +> 💡 This becomes a tip. +> ⚡ This becomes a tip. +> ⚠️ This becomes a warning. +``` + +HTML output: + +```html + +``` + +If no explicit title is present, no `.callout-title` paragraph is emitted. + +PDF output: + +```tex +\begin{blogwarningbox}{Check this assumption} +... +\end{blogwarningbox} +``` + +If no explicit title is present, the second environment argument is empty and +the `tcolorbox` title is omitted. + +## Semantic Environments + +Use semantic environments for theorem-like content, definitions, examples, and +proofs. + +Canonical form: + +````markdown +```{.env .theorem #thm:hall title="Hall's theorem" html="callout"} +Every bipartite graph satisfying Hall's condition has a matching that covers +the left side. +``` +```` + +Supported environment types: + +```text +theorem +lemma +proposition +corollary +definition +example +remark +proof +``` + +Attributes: + +| Attribute | Required | Meaning | +| ---------------- | ----------- | -------------------------------------------- | +| `.env` | Recommended | Marks this as a semantic environment. | +| `.` | Yes | One of the supported environment types. | +| `#label` | Recommended | Cross-reference label and HTML id. | +| `title="..."` | Optional | Environment title. | +| `html="callout"` | Optional | Render as a styled block in HTML. | +| `html="plain"` | Optional | Render as a quieter left-rule block in HTML. | + +Default HTML mode: + +- Theorem-like blocks default to `html="callout"`. +- `proof` defaults to `html="plain"`. + +HTML output: + +```html +
+

Theorem (Hall's theorem)

+ ... +
+``` + +PDF output: + +```tex +\begin{theorem}[Hall's theorem] +\label{thm:hall} +... +\end{theorem} +``` + +Short HTML-only form: + +````markdown +```theorem id="thm:hall" title="Hall's theorem" +Every bipartite graph satisfying Hall's condition has a matching... +``` +```` + +Use the canonical Pandoc-style form for any material that should export to PDF. + +## Interactive Blocks + +Interactive blocks reserve a place for a React component in HTML and provide a +text fallback for PDF and non-JavaScript output. + +Canonical form: + +````markdown +```{.interactive #int:tape-layout caption="Tape layout explorer"} +Drag files along a tape and compare the access cost of each arrangement. +``` +```` + +Attributes: + +| Attribute | Required | Meaning | +| ----------------- | -------- | ---------------------------------------------------- | +| `.interactive` | Yes | Marks this as an interactive block. | +| `#int:` | Yes | Semantic label. The `int:` prefix is for references. | +| `caption="..."` | Optional | Figure caption in HTML and PDF. | +| `title="..."` | Optional | Fallback caption when `caption` is absent. | +| `component="..."` | Optional | Override the component id. | +| `pdf="..."` | Optional | PDF-specific replacement text. | +| `fallback="..."` | Optional | General fallback text. | + +Component resolution: + +```text +#int:tape-layout + -> component id: tape-layout + -> component path: sites//src/interactives/tape-layout/index.tsx +``` + +HTML output: + +```html +
+
+

...

+
+
Tape layout explorer
+
+``` + +The runtime loaded by `InteractiveRuntime.astro` imports: + +```text +sites//src/interactives/runtime.tsx +``` + +The runtime uses `import.meta.glob("./*/index.tsx")`, so each interactive +component should live in its own directory with an `index.tsx` entrypoint. + +Component props: + +```ts +type InteractiveProps = { + id: string; + label: string; + caption: string; + description: string; + fallback: string; + attrs: Record; +}; +``` + +PDF output: + +```tex +\begin{figure}[htbp] +\centering +\fbox{\begin{minipage}{0.86\linewidth} +Fallback text... +\end{minipage}} +\caption{Tape layout explorer} +\label{int:tape-layout} +\end{figure} +``` + +Short HTML-only form: + +````markdown +```interactive id="tape-layout" caption="Tape layout explorer" +Drag files along a tape and compare the access cost. +``` +```` + +Use the canonical Pandoc-style form for any material that should export to PDF. + +## Cross-References + +Use ordinary Markdown links to ids: + +```markdown +The proof of [Hall's theorem](#thm:hall) is constructive. +See the [tape explorer](#int:tape-layout) for the interactive version. +``` + +HTML keeps the link: + +```html +Hall's theorem +``` + +PDF converts the link to: + +```tex +\cref{thm:hall} +``` + +Recommended label prefixes: + +```text +thm: theorem +lem: lemma +prop: proposition +cor: corollary +def: definition +ex: example +rem: remark +prf: proof +int: interactive +fig: figure +tbl: table +eq: equation +``` + +Do not reuse labels within one document. + +## Inline Footnotes + +The HTML parser supports inline footnotes: + +```markdown +This is a sentence.^[This is the footnote.] +``` + +`remarkInlineFootnotes` rewrites these into normal Markdown footnote +definitions before rendering. + +The parser also supports multi-node inline footnotes when formatting spans the +inside of the footnote. + +For maximum PDF portability, ordinary footnotes are safest: + +```markdown +This is a sentence.[^note] + +[^note]: This is the footnote. +``` + +## Math + +Use standard dollar math: + +```markdown +Inline math: $a^2 + b^2 = c^2$. + +$$ +\sum_{i=1}^n i = \frac{n(n+1)}{2} +$$ +``` + +HTML uses `remark-math` plus `rehype-katex`. + +PDF passes math through Pandoc to LuaLaTeX. + +Avoid math syntax that only one renderer understands. + +## Symbols, Emoji, Chess, And Cards + +Write Unicode directly in Markdown: + +```markdown +The position after ♘f3 is pleasant. +Hearts and diamonds are red: ♥ ♦. +Spades and clubs are black: ♠ ♣. +This note is important 🎯. +``` + +HTML wraps these glyphs in semantic spans: + +```html +♘ +♥ +🎯 +``` + +PDF maps them to LaTeX macros: + +```text +emoji and emoji sequences -> \BlogEmojiText{...} +♔ ♕ ♖ ♗ ♘ ♙ -> chessfss white-piece macros +♚ ♛ ♜ ♝ ♞ ♟ -> chessfss black-piece macros +♥ ♡ ♦ ♢ -> red amssymb suit macros +♠ ♤ ♣ ♧ -> black amssymb suit macros +playing-card emoji -> \BlogEmojiText{...} +``` + +Do not write LaTeX-only suit or chess macros in source unless the document is +PDF-only. + +## Code Fences + +The code-fence language parser normalizes a few imported Quarto forms: + +````markdown +```{ojs} +viewof x = Inputs.range([0, 10]) +``` +```` + +becomes a JavaScript-highlighted fence: + +````markdown +```js +viewof x = Inputs.range([0, 10]) +``` +```` + +This pass is intentionally conservative. Add aliases only when imported source +material needs them. + +## Parser Ordering + +Parser order matters. + +Use this order for Astro: + +```js +const remarkPlugins = [ + remarkGfm, + remarkInlineFootnotes, + remarkCodeFenceLanguages, + remarkSemanticBlocks, + remarkMath, + remarkTypography, + remarkCallouts, + remarkSymbols, +]; +const rehypePlugins = [rehypeKatex]; +``` + +Reasons: + +- Inline footnotes should be normalized before other structure changes. +- Code-fence languages should be normalized before semantic fences are read. +- Semantic blocks should run before code fences become highlighted code. +- Math should be parsed before typography and symbol wrapping can interfere. +- Typography should run before symbol wrapping. +- Callouts should run before symbol wrapping so marker detection sees plain + text. +- Symbols should run late because it emits raw HTML spans. + +## Astro Integration Checklist + +Copy these files into the target Astro project: + +```text +src/remark/callouts.mjs +src/remark/code-fence-languages.mjs +src/remark/inline-footnotes.mjs +src/remark/semantic-blocks.mjs +src/remark/symbols.mjs +src/remark/typography.mjs +src/interactives/runtime.tsx +src/components/InteractiveRuntime.astro +src/styles/article.css +``` + +Wire the plugins in `astro.config.mjs`: + +```js +import mdx from "@astrojs/mdx"; +import rehypeKatex from "rehype-katex"; +import remarkGfm from "remark-gfm"; +import remarkMath from "remark-math"; +import remarkCallouts from "./src/remark/callouts.mjs"; +import remarkCodeFenceLanguages from "./src/remark/code-fence-languages.mjs"; +import remarkInlineFootnotes from "./src/remark/inline-footnotes.mjs"; +import remarkSemanticBlocks from "./src/remark/semantic-blocks.mjs"; +import remarkSymbols from "./src/remark/symbols.mjs"; +import remarkTypography from "./src/remark/typography.mjs"; + +const remarkPlugins = [ + remarkGfm, + remarkInlineFootnotes, + remarkCodeFenceLanguages, + remarkSemanticBlocks, + remarkMath, + remarkTypography, + remarkCallouts, + remarkSymbols, +]; +const rehypePlugins = [rehypeKatex]; + +export default defineConfig({ + integrations: [mdx({ remarkPlugins, rehypePlugins })], + markdown: { remarkPlugins, rehypePlugins }, +}); +``` + +Add the runtime to post pages: + +```astro +--- +import InteractiveRuntime from "@/components/InteractiveRuntime.astro"; +--- + +
+ +
+ +``` + +Make sure the article body uses the CSS class that `article.css` targets. + +## PDF Integration Checklist + +Copy these files: + +```text +scripts/export-pdf.mjs +scripts/pandoc-filters/semantic-blocks.lua +scripts/pandoc-templates/semantic-preamble.tex +``` + +Add a package script: + +```json +{ + "scripts": { + "pdf:post": "node scripts/export-pdf.mjs" + } +} +``` + +Export a post: + +```bash +npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf +``` + +The export command currently uses: + +```text +markdown+fenced_code_attributes+tex_math_dollars+tex_math_single_backslash+footnotes+raw_html +``` + +and includes the shared preamble with: + +```text +--include-in-header scripts/pandoc-templates/semantic-preamble.tex +``` + +If LuaLaTeX cannot write its font cache in a sandboxed environment, set a +writable TeX cache: + +```bash +TEXMFVAR=/tmp/texmf-var TEXMFCONFIG=/tmp/texmf-config \ + npm run pdf:post -- path/to/post/index.md --out /tmp/post.pdf +``` + +## Obsidian Behavior + +The syntax is chosen to remain readable in Obsidian: + +- Plain Markdown stays plain Markdown. +- `> [!tip]` and `> [!tip: Title]` render as Obsidian callouts. +- Fenced semantic blocks remain visible as code blocks when Obsidian does not + know how to render them. +- Interactive blocks show their fallback description in the source. + +Obsidian will not execute Astro interactives or render LaTeX environments as +final theorem boxes. That is acceptable: Obsidian is the editing surface, not +the final renderer. + +## Adding A New Semantic Feature + +When adding a feature, update all three layers: + +1. Source syntax in this document. +2. Astro parser and CSS. +3. Pandoc Lua filter and LaTeX preamble. + +Use this checklist: + +- Define the Markdown syntax and at least one canonical example. +- Decide whether it needs a label and cross-reference behavior. +- Decide the HTML output shape and CSS classes. +- Decide the PDF output shape and required LaTeX packages. +- Add an Obsidian-readable fallback. +- Add a smoke-test Markdown file and render both HTML and PDF. +- Document migration rules for old content, if any. + +## Known Tradeoffs + +The current system is robust, but it has deliberate limits: + +- It is a convention layer over Markdown, not a full new markup language. +- HTML and PDF are separate renderers. The contract is shared, but exact visual + parity is not the goal. +- Interactives are HTML-first. PDF receives a figure-like fallback. +- Raw HTML is allowed for web-only material but should not be used for semantic + features that need PDF output. +- The blog repo currently duplicates parser files across seven sites. A future + shared parser package would reduce drift. + +## Minimal Smoke Test + +Use this sample when porting the parser to another project: + +````markdown +--- +title: "Parser Smoke Test" +description: "Checks callouts, environments, links, symbols, math, and interactives." +pubDate: "2026-01-31" +--- + +This sentence has an em dash --- and inline math $a^2+b^2=c^2$. + +> [!tip: Explicit title] +> +> This callout has a title, a chess piece ♘, a card suit ♥, and an emoji 🎯. + +```{.env .definition #def:test title="Test object" html="callout"} +A test object is something we can refer to later. +``` + +See [the definition](#def:test). + +```{.interactive #int:test-widget caption="Test widget" pdf="The HTML version contains a small interactive test widget."} +This fallback appears when the interactive component is unavailable. +``` +```` + +Expected HTML: + +- One em dash. +- KaTeX-rendered math. +- A titled tip callout. +- A semantic definition block with `id="def:test"`. +- A normal anchor link to `#def:test`. +- A framed interactive fallback if no component exists. +- Wrapped symbol spans for `♘`, `♥`, and `🎯`. + +Expected PDF: + +- A LuaLaTeX em dash. +- Typeset math. +- A titled `tcolorbox` tip. +- A real `definition` environment with `\label{def:test}`. +- A `\cref{def:test}` reference. +- A figure fallback for `int:test-widget`.