Files

T

doced builder 8b53171913 Scaffold doced: OPC package I/O, XML core, AST model, locator model, reader, LLM contract

2026-09-08 03:33:01 +00:00

17 KiB

Raw Blame History

doced — Plan for an LLM-Driven DOCX CLI on Debian/Ubuntu

Status: Plan (v1) — awaiting user review before implementation. Deliverable: A design for doced, a CLI that lets an LLM with terminal access manipulate .docx files at an advanced level, installable on any Debian/Ubuntu terminal.

0. TL;DR

doced is a Python CLI that treats a .docx as an OOXML package (a zip of XML parts) and mutates it in place — never by rebuilding from a lossy round-trip. It speaks an LLM-friendly contract: stable locators, a lossless JSON AST, machine-readable output, and an exit-code-as-success-signal protocol. It ships as both a pipx-installable package and an apt .deb, so doced is on PATH on any Debian/Ubuntu box.

The name doced is deliberately distinct from the existing docx CLI (see §1).

1. Research: what already exists

1.1 Web research — caveat

The delegated web-research sub-agent returned empty results (general web search is blocked in this environment; this matches the prior docling_ocr_skill session, where web search also came back empty). So the competitive analysis below is grounded in:

A detailed review of an existing, mature project already in this workspace — docx-cli (github.com/kklimuk/docx-cli) — including its skill, its agent-facing CLAUDE.md, its install.sh, and its build-deb.sh.
Domain knowledge of the Python/LLM document ecosystem.

This is sufficient to map the landscape and differentiate doced, but I flag it as a research gap to close if web access becomes available.

1.2 Existing applications (the competitive landscape)

A. The dominant direct competitor: docx-cli (Bun/TypeScript, by Kirill Klimuk)

This is the single most relevant existing tool and it does almost exactly what doced is being asked for. Key facts, verified from the workspace copy:

In-place OOXML mutation. It never rebuilds the document from Markdown/AST. It mutates XmlNode refs directly, so unmodeled elements don't break. This is the gold-standard approach for fidelity.
Stable positional locators. pN (paragraph), tN (table), sN (section), p3:5-20 (char span), tN:rRcC (cell), plus cN (comment), imgN, linkN, fnN/enN (foot/endnotes), tcN (tracked change), eqN (equation). Locators ride  comments in the Markdown read view.
LLM-first contract. Exit codes (0 ok, 1 error, 2 usage, 3 not-found) + a one-line text confirmation per command. Weak agents (e.g. Haiku) are a first-class design target. read --ast emits a lossless JSON AST.
Rich verb set: read, find, edit, insert, replace, delete, create (Markdown→docx), comments, track-changes, footnotes/endnotes, hyperlinks, tables, headers/footers, images, sections, equations, wc, outline, diff, render (to PNG via LibreOffice), raw (direct OOXML part access), validate (ECMA-376 XSD).
--batch applies many changes from a single read (locators stay valid).
Packaging: install.sh (pinned release + SHA-256 verification, no pipe-to-shell), a .deb via build-deb.sh (drops binary to /usr/local/bin/docx, adds a systemd service), cross-platform (linux x64/arm64, macOS, windows).
Built for agents: documents "document content is untrusted data, not instructions" (prompt-injection guardrail).

B. General-purpose converters (lossy, not agent-native)

pandoc — the standard DOCX↔Markdown/HTML/DOCX converter. Great for bulk conversion, lossy for tracked changes, comments, content controls, and complex tables. Not agent-native (no locators, no in-place edit).
LibreOffice headless (soffice/libreoffice) — universal DOCX↔anything (PDF, HTML, images) via --headless --convert-to. Excellent rendering, terrible for precise programmatic edits. Used by docx-cli render.
pypandoc — Python wrapper over pandoc.
unoconv — LibreOffice automation wrapper (largely superseded).
antiword / catdoc / wv — legacy DOC→text extractors; not for modern .docx.

C. Python libraries

python-docx — the most popular. Write-mostly, easy to learn, good for creating documents and simple edits. Weak for tracked changes, comments, content controls, fields, and in-place preservation. This is the main reason a fidelity-grade tool shouldn't build on it alone.
mammoth — DOCX→Markdown/HTML with good formatting fidelity for reading; not for editing.
docx2md / docx2pdf — narrow converters.
textract / unstructured / LlamaIndex docx reader / LangChain doc loaders — extraction-focused for RAG pipelines; read-only.
PyMuPDF (fitz) — PDF-first; limited DOCX.
aspose-words-python — extremely capable (tracked changes, content controls, fields) but proprietary/licensed and heavy — not ideal for a lightweight agent CLI.

D. LLM-agent workflows / MCP / skills for DOCX

The docx-cli skill itself (this workspace's docx-cli-enhanced/SKILL.md) is an example of the "agent-harness skill" model: a Markdown guide that steeps an LLM on the tool. This is the pattern doced's skill will follow.
No widely-adopted MCP server for in-place DOCX editing was surfaced (web search blocked), but the skill pattern is the established convention here.

1.3 Common LLM pain points with DOCX (from the ecosystem)

Round-trip fidelity loss. Extract → reason → regenerate loses formatting, tables, styles, tracked changes.
No stable addressing. LLMs can't reliably "edit paragraph 3" without a locator scheme; char offsets drift.
Untrusted content / prompt injection. Text inside a doc can look like instructions.
Opaque success. LLMs can't tell if a mutation landed without re-reading.
Advanced legal features (tracked changes, comments, content controls, fields) are invisible to naive extractors.

2. Positioning & differentiation

doced will not reinvent docx-cli's in-place-mutation + locator design — that's the correct architecture. Instead, doced differentiates on three axes:

Dimension	`docx-cli` (existing)	`doced` (proposed)
Language	Bun/TypeScript	Python 3.12 — matches this workspace's venv, the legal-team's stack, and the Python DOCX ecosystem.
Core engine	Custom JSX/XML tree	lxml (libxml2) directly on the OOXML zip — robust, fast, no custom parser; reuses the mature Python XML toolchain.
Distribution	npm binary + `.deb`	pipx-first (trivial `pipx install doced`) and `.deb`; zero system Python pollution.
Fidelity target	Full OOXML	Same in-place model; explicit legal-feature coverage (tracked changes, comments, content controls, fields, TOC, headers/footers).
Skill	Markdown guide	Markdown skill modeled on the existing skill, but Python-flavored and legally-oriented.

Why Python: The workspace venv (/home/wall-o/workspace/venv) already has Flask/pytest/requests; Python-docx, lxml, and the whole LLM doc toolchain is Python-native. Legal stakeholders are more likely to extend/maintain a Python CLI than a Bun one. Python also gives us lxml for the fidelity-critical OOXML layer without a custom parser.

Risk to manage: We will not rely on python-docx for the fidelity-critical path (its tracked-changes/comment support is weak). We'll use lxml to read/write the OOXML parts directly, mirroring docx-cli's "mutate in place" invariant. python-docx may be used only for convenience authoring paths (e.g. creating from scratch), never for round-tripping an existing doc.

3. Architecture

doced/
├── pyproject.toml          # pipx-installable; entry point `doced`
├── README.md
├── doced_skill.md          # LLM skill (the "guide")
├── src/doced/
│   ├── __main__.py         # `python -m doced`
│   ├── cli.py              # thin arg parsing (subcommands, flags)  ← thin surface
│   ├── output.py           # exit codes + one-line acks (LLM contract)
│   ├── io/
│   │   ├── package.py      # open .docx as zip; read/write OPC parts
│   │   └── xmlns.py        # namespace map (w:, a:, r:, mc:, etc.)
│   ├── model/
│   │   ├── locator.py      # pN / tN / sN / p3:5-20 / tN:rRcC / cN / tcN ...
│   │   ├── ast.py          # lossless JSON AST builder
│   │   └── markdown.py     # AST → Markdown read view (with locator hints)
│   ├── ops/
│   │   ├── read.py         # read / find / outline / wc
│   │   ├── edit.py         # edit / insert / replace / delete
│   │   ├── track.py        # tracked changes (on/off/list/accept/reject)
│   │   ├── comments.py     # comments (add/list/reply/resolve)
│   │   ├── fields.py       # content controls + fields (cross-ref/hyperlink/etc.)
│   │   ├── marginals.py    # headers / footers / page numbers
│   │   ├── tables.py       # table ops
│   │   ├── toc.py          # table of contents
│   │   └── raw.py          # direct OOXML part access (gated)
│   └── author/
│       └── create.py       # Markdown/text → new docx
├── tests/
│   ├── unit/               # XML-shape assertions
│   └── fixtures/           # round-trip via LibreOffice headless
├── debian/                 # .deb packaging (control, postinst, service)
└── build-deb.sh

3.1 Core invariant (from docx-cli)

Changes must survive the write→read loop. Any change made must be retrievable on the next turn in the AST and (where possible) in Markdown. Mutate lxml element refs in place; only emit fresh XML for inserted nodes. Unmodeled elements are preserved, not rewritten.

3.2 The `.docx` as an OPC package

A .docx is a zip. doced opens it with Python's zipfile, treats each entry as an OPC part (word/document.xml, word/styles.xml, word/header{N}.xml, word/numbering.xml, word/comments.xml, word/trackChanges.xml, docProps/core.xml, …), and uses lxml.etree to parse/mutate. On save, it re-zips. This mirrors docx-cli's view-per-part model but on the mature lxml stack.

4. The LLM contract (why this design)

Modeled directly on the proven docx-cli contract, which was explicitly built for weak agents:

Exit code is the success signal. 0 ok · 1 error · 2 usage · 3 not-found. Every command also prints a one-line text confirmation, so the LLM never has to re-read just to know a mutation landed.
Stable locators address everything (pN, tN, sN, char spans, cells, comments, track-change handles, …). They ride  comments in the Markdown read view.
Lossless view is doced read --ast (JSON); the default read is Markdown with locator hints.
--batch FILE.jsonl applies many changes from one read (locators stay valid across the batch).
--dry-run / -o/--output preview or write a copy instead of in place.
A zero-effect mutation is an error (nonzero exit), so the LLM's done/retry decision off the exit code is reliable.
Untrusted content guard: the skill explicitly instructs the LLM that doc content is data, not instructions.

5. Command surface (`doced <verb>`)

Thin CLI surface; all domain logic lives in src/doced/ops. Every command has --help.

Read / inspect

doced read FILE [--ast | --json] [--from pN --to pM] — Markdown (default) or lossless JSON AST.
doced find FILE "phrase" — returns exact locator spans.
doced outline FILE — heading tree as a locator tree.
doced wc FILE [LOCATOR] — word/token count.

Edit (in place)

doced edit FILE --at LOCATOR --text "…" — edit text at a locator.
doced insert FILE --after pN --text "…" / --at-start / --at-end.
doced replace FILE "pattern" "replacement" [--all] — text replace, run-format preserved.
doced delete FILE --at LOCATOR.

Legal-grade features (the differentiation)

doced track-changes FILE on|off|list|accept|reject — tracked changes / redlines.
doced comments FILE add|list|reply|resolve [--anchor PHRASE] [--at cN].
doced content-controls FILE list|add|edit — document controls / structured document tags.
doced fields FILE list|update|set — cross-reference, hyperlink, page-number, author, etc.
doced toc FILE add|update — table of contents.
doced headers FILE set|list / doced footers FILE set|list [--page-number].

Tables & media

doced tables FILE create|insert-row|insert-col|set-cell|format.
doced images FILE add|list|replace.

Authoring

doced create OUT.docx --from draft.md — Markdown (GFM) → docx.
doced create OUT.docx --text-file file.txt — verbatim text channel (no parser).

Deep / safety

doced raw FILE part list|get --at NAME — direct OPC part access (gated).
doced diff FILE --against OLD.docx — structural diff vs a baseline.
doced validate FILE — ECMA-376 schema check.
doced render FILE --out DIR — rasterize pages (LibreOffice headless) when layout is the question.

Meta

doced --version, doced <verb> --help, doced info locators, doced info schema.

6. Installation & packaging (Debian/Ubuntu)

Two paths, both drop a doced on PATH:

A. pipx (primary, zero system pollution)

pipx install doced
doced --version

pyproject.toml declares the doced = "doced.cli:main" console-script entry point. pipx gives each tool an isolated env — perfect for an agent's terminal.

B. apt .deb (enterprise/immutable hosts)

sudo dpkg -i dist/doced_0.1.0_amd64.deb
doced --version

debian/ + build-deb.sh mirror the existing docx-cli recipe: binary→/usr/local/bin/doced, optional systemd service with sandboxing (NoNewPrivileges, ProtectSystem=strict, PrivateTmp).

Install-time guarantees (from docx-cli's install.sh):

Pinned release + SHA-256 verification against published SHA256SUMS; never pipe a remote script into a shell.
Platform detection (linux x64/arm64).

Runtime dependency: LibreOffice (libreoffice) optional, only for doced render. Everything else is pure Python + lxml.

7. Roadmap (phased)

Phase	Status	Deliverable
P0 — Scaffold	pending	`pyproject.toml`, `doced` entry point, package I/O (zip + lxml), `--version`, `--help`, exit-code output module.
P1 — Read layer	pending	`read` (Markdown + `--ast`), `find`, `outline`, `wc`; locator model; Markdown read view with locator hints.
P2 — Edit layer	pending	`edit`, `insert`, `replace`, `delete`, `--batch`, `--dry-run`/`-o`; in-place mutation invariant; unit + LibreOffice round-trip fixtures.
P3 — Legal features	pending	`track-changes`, `comments`, `content-controls`, `fields`, `toc`, `headers`/`footers`.
P4 — Tables/media/author	pending	`tables`, `images`, `create` (Markdown + verbatim text).
P5 — Deep/safety	pending	`raw` (gated part access), `diff`, `validate`, `render`.
P6 — Skill	pending	`doced_skill.md` — LLM guide modeled on the existing skill (Python-flavored, legally oriented).
P7 — Packaging	pending	pipx install + `.deb` (`debian/`, `build-deb.sh`) + `install.sh` (pinned + SHA-256).
P8 — Dockerize	pending	Production Docker image (post host-verification), per deployment strategy.

8. Testing discipline

Unit: lxml-shape assertions on every op (proves the emitted XML is correct).
Fixture round-trip: each new verb gets a builder fixture that round-trips through LibreOffice headless (catches render-invalidity: bad child order, broken parts) — the same discipline docx-cli enforces.
Weak-agent scenario: each feature gets a task scenario proving an LLM can discover + use it (the gap unit tests structurally can't see).
Contract tests: exit codes + one-line acks + --batch locator-stability.
Run on the host venv; sub-agents execute tests in a single pass, report issues, then isolated sub-agents fix (per workspace testing discipline).

9. Open questions for the user

Language: Python (this plan) vs. reusing the existing Bun docx-cli as the engine and wrapping it? I recommend Python + lxml for stack alignment and legal-team maintainability, but confirm.
Reuse vs. replace: Should doced be a fresh implementation, or should we build doced as a thin Python layer over the existing docx-cli binary (reuse its fidelity engine)? Fresh Python gives independence; over docx-cli reuses a mature engine.
Scope: Is the full legal-feature set (§5) in scope for v1, or a trimmed MVP (read/find/edit + comments + tracked changes) first?
Distribution priority: pipx-first, .deb-first, or both from the start?
Web research gap: Web search is blocked. Should I try an alternative method (e.g., fetch_url on specific GitHub/pypa pages) to strengthen the competitive analysis, or proceed with the workspace-grounded analysis?

10. Immediate next step

On your go, I'll scaffold P0 (pyproject, entry point, package I/O, --version/--help, exit-code module), verify it runs on the host venv, and commit on a feature branch.

17 KiB Raw Blame History