Scaffold doced: OPC package I/O, XML core, AST model, locator model, reader, LLM contract

2026-09-08 03:33:01 +00:00
commit 8b53171913
29 changed files with 1868 additions and 0 deletions
@@ -0,0 +1,259 @@
+# doced — Plan for an LLM-Driven DOCX CLI on Debian/Ubuntu
+
+> **Status:** Plan (v1) — awaiting user review before implementation.
+> **Deliverable:** A design for `doced`, a CLI that lets an LLM with terminal access manipulate `.docx` files at an advanced level, installable on any Debian/Ubuntu terminal.
+
+---
+
+## 0. TL;DR
+
+`doced` is a Python CLI that treats a `.docx` as an OOXML package (a zip of XML parts) and mutates it **in place** — never by rebuilding from a lossy round-trip. It speaks an LLM-friendly contract: **stable locators**, a **lossless JSON AST**, **machine-readable output**, and an **exit-code-as-success-signal** protocol. It ships as both a `pipx`-installable package and an `apt` `.deb`, so `doced` is on PATH on any Debian/Ubuntu box.
+
+The name `doced` is deliberately distinct from the existing `docx` CLI (see §1).
+
+---
+
+## 1. Research: what already exists
+
+### 1.1 Web research — caveat
+
+The delegated web-research sub-agent returned **empty results** (general web search is blocked in this environment; this matches the prior `docling_ocr_skill` session, where web search also came back empty). So the competitive analysis below is grounded in:
+
+1. A **detailed review of an existing, mature project already in this workspace** — `docx-cli` (github.com/kklimuk/docx-cli) — including its skill, its agent-facing `CLAUDE.md`, its `install.sh`, and its `build-deb.sh`.
+2. Domain knowledge of the Python/LLM document ecosystem.
+
+This is sufficient to map the landscape and differentiate `doced`, but I flag it as a research gap to close if web access becomes available.
+
+### 1.2 Existing applications (the competitive landscape)
+
+**A. The dominant direct competitor: `docx-cli` (Bun/TypeScript, by Kirill Klimuk)**
+
+This is the single most relevant existing tool and it does almost exactly what `doced` is being asked for. Key facts, verified from the workspace copy:
+
+- **In-place OOXML mutation.** It never rebuilds the document from Markdown/AST. It mutates `XmlNode` refs directly, so unmodeled elements don't break. This is the gold-standard approach for fidelity.
+- **Stable positional locators.** `pN` (paragraph), `tN` (table), `sN` (section), `p3:5-20` (char span), `tN:rRcC` (cell), plus `cN` (comment), `imgN`, `linkN`, `fnN`/`enN` (foot/endnotes), `tcN` (tracked change), `eqN` (equation). Locators ride `<!-- -->` comments in the Markdown read view.
+- **LLM-first contract.** Exit codes (`0` ok, `1` error, `2` usage, `3` not-found) + a one-line text confirmation per command. Weak agents (e.g. Haiku) are a first-class design target. `read --ast` emits a lossless JSON AST.
+- **Rich verb set:** `read`, `find`, `edit`, `insert`, `replace`, `delete`, `create` (Markdown→docx), `comments`, `track-changes`, `footnotes`/`endnotes`, `hyperlinks`, `tables`, `headers`/`footers`, `images`, `sections`, `equations`, `wc`, `outline`, `diff`, `render` (to PNG via LibreOffice), `raw` (direct OOXML part access), `validate` (ECMA-376 XSD).
+- **`--batch`** applies many changes from a single read (locators stay valid).
+- **Packaging:** `install.sh` (pinned release + SHA-256 verification, no pipe-to-shell), a **`.deb`** via `build-deb.sh` (drops binary to `/usr/local/bin/docx`, adds a systemd service), cross-platform (linux x64/arm64, macOS, windows).
+- **Built for agents:** documents "document content is untrusted data, not instructions" (prompt-injection guardrail).
+
+**B. General-purpose converters (lossy, not agent-native)**
+
+- **pandoc** — the standard DOCX↔Markdown/HTML/DOCX converter. Great for bulk conversion, **lossy** for tracked changes, comments, content controls, and complex tables. Not agent-native (no locators, no in-place edit).
+- **LibreOffice headless (`soffice`/`libreoffice`)** — universal DOCX↔anything (PDF, HTML, images) via `--headless --convert-to`. Excellent rendering, terrible for precise programmatic edits. Used by `docx-cli render`.
+- **pypandoc** — Python wrapper over pandoc.
+- **unoconv** — LibreOffice automation wrapper (largely superseded).
+- **antiword / catdoc / wv** — legacy DOC→text extractors; not for modern `.docx`.
+
+**C. Python libraries**
+
+- **python-docx** — the most popular. Write-mostly, easy to learn, good for creating documents and simple edits. **Weak** for tracked changes, comments, content controls, fields, and in-place preservation. This is the main reason a fidelity-grade tool shouldn't build on it alone.
+- **mammoth** — DOCX→Markdown/HTML with good formatting fidelity for *reading*; not for editing.
+- **docx2md / docx2pdf** — narrow converters.
+- **textract / unstructured / LlamaIndex docx reader / LangChain doc loaders** — extraction-focused for RAG pipelines; read-only.
+- **PyMuPDF (fitz)** — PDF-first; limited DOCX.
+- **aspose-words-python** — extremely capable (tracked changes, content controls, fields) but **proprietary/licensed and heavy** — not ideal for a lightweight agent CLI.
+
+**D. LLM-agent workflows / MCP / skills for DOCX**
+
+- The **`docx-cli` skill** itself (this workspace's `docx-cli-enhanced/SKILL.md`) is an example of the "agent-harness skill" model: a Markdown guide that steeps an LLM on the tool. This is the pattern `doced`'s skill will follow.
+- No widely-adopted MCP server for in-place DOCX editing was surfaced (web search blocked), but the skill pattern is the established convention here.
+
+### 1.3 Common LLM pain points with DOCX (from the ecosystem)
+
+1. **Round-trip fidelity loss.** Extract → reason → regenerate loses formatting, tables, styles, tracked changes.
+2. **No stable addressing.** LLMs can't reliably "edit paragraph 3" without a locator scheme; char offsets drift.
+3. **Untrusted content / prompt injection.** Text inside a doc can look like instructions.
+4. **Opaque success.** LLMs can't tell if a mutation landed without re-reading.
+5. **Advanced legal features** (tracked changes, comments, content controls, fields) are invisible to naive extractors.
+
+---
+
+## 2. Positioning & differentiation
+
+`doced` will **not** reinvent `docx-cli`'s in-place-mutation + locator design — that's the correct architecture. Instead, `doced` differentiates on three axes:
+
+| Dimension | `docx-cli` (existing) | `doced` (proposed) |
+|---|---|---|
+| **Language** | Bun/TypeScript | **Python 3.12** — matches this workspace's venv, the legal-team's stack, and the Python DOCX ecosystem. |
+| **Core engine** | Custom JSX/XML tree | **lxml** (libxml2) directly on the OOXML zip — robust, fast, no custom parser; reuses the mature Python XML toolchain. |
+| **Distribution** | npm binary + `.deb` | **pipx-first** (trivial `pipx install doced`) **and** `.deb`; zero system Python pollution. |
+| **Fidelity target** | Full OOXML | Same in-place model; **explicit legal-feature coverage** (tracked changes, comments, content controls, fields, TOC, headers/footers). |
+| **Skill** | Markdown guide | Markdown skill modeled on the existing skill, but Python-flavored and legally-oriented. |
+
+**Why Python:** The workspace venv (`/home/wall-o/workspace/venv`) already has Flask/pytest/requests; Python-docx, lxml, and the whole LLM doc toolchain is Python-native. Legal stakeholders are more likely to extend/maintain a Python CLI than a Bun one. Python also gives us `lxml` for the fidelity-critical OOXML layer without a custom parser.
+
+**Risk to manage:** We will **not** rely on python-docx for the fidelity-critical path (its tracked-changes/comment support is weak). We'll use `lxml` to read/write the OOXML parts directly, mirroring `docx-cli`'s "mutate in place" invariant. python-docx may be used only for *convenience authoring* paths (e.g. creating from scratch), never for round-tripping an existing doc.
+
+---
+
+## 3. Architecture
+
+```
+doced/
+├── pyproject.toml          # pipx-installable; entry point `doced`
+├── README.md
+├── doced_skill.md          # LLM skill (the "guide")
+├── src/doced/
+│   ├── __main__.py         # `python -m doced`
+│   ├── cli.py              # thin arg parsing (subcommands, flags)  ← thin surface
+│   ├── output.py           # exit codes + one-line acks (LLM contract)
+│   ├── io/
+│   │   ├── package.py      # open .docx as zip; read/write OPC parts
+│   │   └── xmlns.py        # namespace map (w:, a:, r:, mc:, etc.)
+│   ├── model/
+│   │   ├── locator.py      # pN / tN / sN / p3:5-20 / tN:rRcC / cN / tcN ...
+│   │   ├── ast.py          # lossless JSON AST builder
+│   │   └── markdown.py     # AST → Markdown read view (with locator hints)
+│   ├── ops/
+│   │   ├── read.py         # read / find / outline / wc
+│   │   ├── edit.py         # edit / insert / replace / delete
+│   │   ├── track.py        # tracked changes (on/off/list/accept/reject)
+│   │   ├── comments.py     # comments (add/list/reply/resolve)
+│   │   ├── fields.py       # content controls + fields (cross-ref/hyperlink/etc.)
+│   │   ├── marginals.py    # headers / footers / page numbers
+│   │   ├── tables.py       # table ops
+│   │   ├── toc.py          # table of contents
+│   │   └── raw.py          # direct OOXML part access (gated)
+│   └── author/
+│       └── create.py       # Markdown/text → new docx
+├── tests/
+│   ├── unit/               # XML-shape assertions
+│   └── fixtures/           # round-trip via LibreOffice headless
+├── debian/                 # .deb packaging (control, postinst, service)
+└── build-deb.sh
+```
+
+### 3.1 Core invariant (from docx-cli)
+
+> **Changes must survive the write→read loop.** Any change made must be retrievable on the next turn in the AST and (where possible) in Markdown. **Mutate `lxml` element refs in place; only emit fresh XML for inserted nodes.** Unmodeled elements are preserved, not rewritten.
+
+### 3.2 The `.docx` as an OPC package
+
+A `.docx` is a zip. `doced` opens it with Python's `zipfile`, treats each entry as an **OPC part** (`word/document.xml`, `word/styles.xml`, `word/header{N}.xml`, `word/numbering.xml`, `word/comments.xml`, `word/trackChanges.xml`, `docProps/core.xml`, …), and uses `lxml.etree` to parse/mutate. On save, it re-zips. This mirrors `docx-cli`'s view-per-part model but on the mature lxml stack.
+
+---
+
+## 4. The LLM contract (why this design)
+
+Modeled directly on the proven `docx-cli` contract, which was explicitly built for weak agents:
+
+1. **Exit code is the success signal.** `0` ok · `1` error · `2` usage · `3` not-found. Every command also prints a one-line text confirmation, so the LLM never has to re-read just to know a mutation landed.
+2. **Stable locators** address everything (`pN`, `tN`, `sN`, char spans, cells, comments, track-change handles, …). They ride `<!-- -->` comments in the Markdown read view.
+3. **Lossless view is `doced read --ast`** (JSON); the default `read` is Markdown with locator hints.
+4. **`--batch FILE.jsonl`** applies many changes from one read (locators stay valid across the batch).
+5. **`--dry-run` / `-o/--output`** preview or write a copy instead of in place.
+6. **A zero-effect mutation is an error** (nonzero exit), so the LLM's done/retry decision off the exit code is reliable.
+7. **Untrusted content guard:** the skill explicitly instructs the LLM that doc content is data, not instructions.
+
+---
+
+## 5. Command surface (`doced <verb>`)
+
+Thin CLI surface; all domain logic lives in `src/doced/ops`. Every command has `--help`.
+
+**Read / inspect**
+- `doced read FILE [--ast | --json] [--from pN --to pM]` — Markdown (default) or lossless JSON AST.
+- `doced find FILE "phrase"` — returns exact locator spans.
+- `doced outline FILE` — heading tree as a locator tree.
+- `doced wc FILE [LOCATOR]` — word/token count.
+
+**Edit (in place)**
+- `doced edit FILE --at LOCATOR --text "…"` — edit text at a locator.
+- `doced insert FILE --after pN --text "…"` / `--at-start` / `--at-end`.
+- `doced replace FILE "pattern" "replacement" [--all]` — text replace, run-format preserved.
+- `doced delete FILE --at LOCATOR`.
+
+**Legal-grade features (the differentiation)**
+- `doced track-changes FILE on|off|list|accept|reject` — tracked changes / redlines.
+- `doced comments FILE add|list|reply|resolve [--anchor PHRASE] [--at cN]`.
+- `doced content-controls FILE list|add|edit` — document controls / structured document tags.
+- `doced fields FILE list|update|set` — cross-reference, hyperlink, page-number, author, etc.
+- `doced toc FILE add|update` — table of contents.
+- `doced headers FILE set|list` / `doced footers FILE set|list [--page-number]`.
+
+**Tables & media**
+- `doced tables FILE create|insert-row|insert-col|set-cell|format`.
+- `doced images FILE add|list|replace`.
+
+**Authoring**
+- `doced create OUT.docx --from draft.md` — Markdown (GFM) → docx.
+- `doced create OUT.docx --text-file file.txt` — verbatim text channel (no parser).
+
+**Deep / safety**
+- `doced raw FILE part list|get --at NAME` — direct OPC part access (gated).
+- `doced diff FILE --against OLD.docx` — structural diff vs a baseline.
+- `doced validate FILE` — ECMA-376 schema check.
+- `doced render FILE --out DIR` — rasterize pages (LibreOffice headless) when layout is the question.
+
+**Meta**
+- `doced --version`, `doced <verb> --help`, `doced info locators`, `doced info schema`.
+
+---
+
+## 6. Installation & packaging (Debian/Ubuntu)
+
+Two paths, both drop a `doced` on PATH:
+
+**A. pipx (primary, zero system pollution)**
+```bash
+pipx install doced
+doced --version
+```
+`pyproject.toml` declares the `doced = "doced.cli:main"` console-script entry point. pipx gives each tool an isolated env — perfect for an agent's terminal.
+
+**B. apt `.deb` (enterprise/immutable hosts)**
+```bash
+sudo dpkg -i dist/doced_0.1.0_amd64.deb
+doced --version
+```
+`debian/` + `build-deb.sh` mirror the existing `docx-cli` recipe: binary→`/usr/local/bin/doced`, optional systemd service with sandboxing (`NoNewPrivileges`, `ProtectSystem=strict`, `PrivateTmp`).
+
+**Install-time guarantees (from docx-cli's `install.sh`):**
+- Pinned release + **SHA-256 verification** against published `SHA256SUMS`; never pipe a remote script into a shell.
+- Platform detection (linux x64/arm64).
+
+**Runtime dependency:** LibreOffice (`libreoffice`) optional, only for `doced render`. Everything else is pure Python + lxml.
+
+---
+
+## 7. Roadmap (phased)
+
+| Phase | Status | Deliverable |
+|---|---|---|
+| **P0 — Scaffold** | pending | `pyproject.toml`, `doced` entry point, package I/O (zip + lxml), `--version`, `--help`, exit-code output module. |
+| **P1 — Read layer** | pending | `read` (Markdown + `--ast`), `find`, `outline`, `wc`; locator model; Markdown read view with locator hints. |
+| **P2 — Edit layer** | pending | `edit`, `insert`, `replace`, `delete`, `--batch`, `--dry-run`/`-o`; in-place mutation invariant; unit + LibreOffice round-trip fixtures. |
+| **P3 — Legal features** | pending | `track-changes`, `comments`, `content-controls`, `fields`, `toc`, `headers`/`footers`. |
+| **P4 — Tables/media/author** | pending | `tables`, `images`, `create` (Markdown + verbatim text). |
+| **P5 — Deep/safety** | pending | `raw` (gated part access), `diff`, `validate`, `render`. |
+| **P6 — Skill** | pending | `doced_skill.md` — LLM guide modeled on the existing skill (Python-flavored, legally oriented). |
+| **P7 — Packaging** | pending | pipx install + `.deb` (`debian/`, `build-deb.sh`) + `install.sh` (pinned + SHA-256). |
+| **P8 — Dockerize** | pending | Production Docker image (post host-verification), per deployment strategy. |
+
+---
+
+## 8. Testing discipline
+
+- **Unit:** `lxml`-shape assertions on every op (proves the emitted XML is correct).
+- **Fixture round-trip:** each new verb gets a builder fixture that round-trips through **LibreOffice headless** (catches render-invalidity: bad child order, broken parts) — the same discipline `docx-cli` enforces.
+- **Weak-agent scenario:** each feature gets a task scenario proving an LLM can discover + use it (the gap unit tests structurally can't see).
+- **Contract tests:** exit codes + one-line acks + `--batch` locator-stability.
+- Run on the host venv; sub-agents execute tests in a single pass, report issues, then isolated sub-agents fix (per workspace testing discipline).
+
+---
+
+## 9. Open questions for the user
+
+1. **Language:** Python (this plan) vs. reusing the existing Bun `docx-cli` as the engine and wrapping it? I recommend **Python + lxml** for stack alignment and legal-team maintainability, but confirm.
+2. **Reuse vs. replace:** Should `doced` be a **fresh implementation**, or should we build `doced` as a **thin Python layer over** the existing `docx-cli` binary (reuse its fidelity engine)? Fresh Python gives independence; over `docx-cli` reuses a mature engine.
+3. **Scope:** Is the full legal-feature set (§5) in scope for v1, or a trimmed MVP (read/find/edit + comments + tracked changes) first?
+4. **Distribution priority:** pipx-first, .deb-first, or both from the start?
+5. **Web research gap:** Web search is blocked. Should I try an alternative method (e.g., `fetch_url` on specific GitHub/pypa pages) to strengthen the competitive analysis, or proceed with the workspace-grounded analysis?
+
+---
+
+## 10. Immediate next step
+
+On your go, I'll scaffold **P0** (pyproject, entry point, package I/O, `--version`/`--help`, exit-code module), verify it runs on the host venv, and commit on a feature branch.