fd1210e557
Complete task002 through task016 across cache correctness, editor lifecycle, C boundaries, build and release governance, dictionary provenance, performance, interactive workflows, tests, documentation, and final cleanup.
43 lines
1.7 KiB
Markdown
43 lines
1.7 KiB
Markdown
# Measure Before Replacing Text Builders
|
||
|
||
## Context
|
||
|
||
The tokenizer appended each character to an accumulator string, dense
|
||
hyphen insertion rebuilt the whole prefix at every break, and the cache used
|
||
nil for both “absent” and “computed with no breaks.” These are familiar
|
||
smells, but ordinary paragraph performance did not prove they mattered.
|
||
|
||
An adversarial 1,000–8,000-character benchmark established the boundary:
|
||
8× input grew tokenizer time 10.3× and dense insertion 11.5×. Repeated
|
||
no-break lookups also called the dictionary computation twice.
|
||
|
||
## Decision
|
||
|
||
Keep ownership in the existing tokenizer and hyphenator. Accumulate
|
||
fragments locally, join once at the point each complete value is emitted,
|
||
and use one private sentinel to distinguish a missing cache entry from a
|
||
cached nil. No general builder layer is introduced.
|
||
|
||
## Alternatives
|
||
|
||
- Leaving the loops unchanged was rejected after scaling and absolute time
|
||
both showed material cost.
|
||
- A reusable rope/builder abstraction was rejected because the two loops
|
||
have different owners and need only local lists.
|
||
- Moving either operation to C was rejected because text properties,
|
||
Unicode segmentation, and dictionary semantics remain Elisp-owned.
|
||
|
||
## Consequences
|
||
|
||
At 8,000 characters, source-mode tokenizer time fell from 3.133 s to
|
||
1.100 s and dense insertion from 0.945 s to 0.013 s. Growth across the
|
||
range is 6.3× and 7.7× respectively for 8× input. Negative hyphenation
|
||
results now reuse the cache. Exact output and text-property regressions
|
||
protect the representation change.
|
||
|
||
## Rollback
|
||
|
||
Revert the fragment builders together with their benchmark claims. Keep
|
||
the nil-cache sentinel and all output regressions unless evidence shows the
|
||
cache invalidation boundary itself is wrong.
|