fd1210e557
Complete task002 through task016 across cache correctness, editor lifecycle, C boundaries, build and release governance, dictionary provenance, performance, interactive workflows, tests, documentation, and final cleanup.
1.7 KiB
Measure Before Replacing Text Builders
Context
The tokenizer appended each character to an accumulator string, dense hyphen insertion rebuilt the whole prefix at every break, and the cache used nil for both “absent” and “computed with no breaks.” These are familiar smells, but ordinary paragraph performance did not prove they mattered.
An adversarial 1,000–8,000-character benchmark established the boundary: 8× input grew tokenizer time 10.3× and dense insertion 11.5×. Repeated no-break lookups also called the dictionary computation twice.
Decision
Keep ownership in the existing tokenizer and hyphenator. Accumulate fragments locally, join once at the point each complete value is emitted, and use one private sentinel to distinguish a missing cache entry from a cached nil. No general builder layer is introduced.
Alternatives
- Leaving the loops unchanged was rejected after scaling and absolute time both showed material cost.
- A reusable rope/builder abstraction was rejected because the two loops have different owners and need only local lists.
- Moving either operation to C was rejected because text properties, Unicode segmentation, and dictionary semantics remain Elisp-owned.
Consequences
At 8,000 characters, source-mode tokenizer time fell from 3.133 s to 1.100 s and dense insertion from 0.945 s to 0.013 s. Growth across the range is 6.3× and 7.7× respectively for 8× input. Negative hyphenation results now reuse the cache. Exact output and text-property regressions protect the representation change.
Rollback
Revert the fragment builders together with their benchmark claims. Keep the nil-cache sentinel and all output regressions unless evidence shows the cache invalidation boundary itself is wrong.