Reuse prepared paragraph data, safe Elisp DP states, common layout lines, and dirty-island source reconstruction for exact append commits. Simplify C int32 validation and add the frozen task030 evaluator, regressions, and decision records.
Replace the physical ekp-region renderer with ekp-buffer text-property projection backed by shared semantic layout plans and stable live-edit transactions.
Optimize the C-backed resize hot path, and add deterministic performance, ERT, GUI, fuzz, release, and documentation coverage for task017 through task032.
BREAKING CHANGE: require ekp-buffer instead of ekp-region and rename module-owned ekp-region-* settings to ekp-buffer-*.
Complete task002 through task016 across cache correctness, editor lifecycle, C boundaries, build and release governance, dictionary provenance, performance, interactive workflows, tests, documentation, and final cleanup.
Two root causes made justified text overrun the window's right edge
in real user sessions (every line ending in the "$" truncation
marker), while emacs -Q looked fine:
1. Windows WITHOUT fringes draw the truncation/continuation
indicator in the text area's LAST COLUMN, so the usable width is
one character less than window-body-width. ekp reserved only a
2px margin: with fringes disabled (a common minimal setup, and
all ttys) every line that hit the target width exactly had its
final glyph displaced by the "$". ekp-region--window-pixel now
reserves one frame-char-width when the window has no right
fringe. (The same lesson ebox-playground encodes in its
viewport-width reserve.)
2. Measurement was blind to the buffer's display context:
string-pixel-width works in a bare hidden buffer, ignoring
face-remapping-alist — which is where text-scale-mode, themes
and per-buffer font tweaks live. Under a remap, the DP laid
lines out with one font's metrics and the display rendered them
with another's: scale +3 made 5 of 7 sample lines overflow a
1330px window by up to 900px (GUI-measured). Measurement now
runs with the destination buffer's face-remapping-alist (the
29/30-compatible equivalent of Emacs 31's string-pixel-width
BUFFER argument), and the width/paragraph caches key on that
context so buffers at different scales never alias. text-scale
changes also trigger a re-flow in ekp-auto-justify-mode.
Ground truth, measured with window-text-pixel-size in GUI Emacs
across 7 display contexts (plain / text-scale ±| face remap /
no-fringes / no-fringes+scale / narrow+scale): the widest justified
line equals the target width exactly in every case, zero lines
overflow. Verbatim code blocks are exempt by design (they never
reflow, like any code line in a narrow window).
New tooling so this never regresses invisibly:
- M-x ekp-diagnose: renders a probe line in YOUR buffer and reports
target vs rendered width — run it in any session where justified
text looks wrong.
- tests/ekp-gui-verify.el: M-x ekp-gui-verify (single check in a
customized session) and ekp-gui-verify-matrix (the 7-case table,
for emacs -Q).
- 2 new batch ERT tests pin the context-keyed caches (94 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three quality gaps closed, one of them unlocking the C engine for
the most common Chinese configuration:
- first-line indent now runs on the 1D DP and the C engine (module
1.5: ekp-c-break-with-arrays grows a 15th arg FIRST-LINE-WIDTH).
A line starts at box 0 exactly when i = 0, so no (position x
line-count) state is needed; only true parshape and looseness
still take the 2D Elisp path. Indented Chinese sample, w=400:
C engine 82 ms (this configuration previously bypassed C entirely
and ran the heaviest Elisp DP). The C-result reconstruction
mirrors the per-line width, and an equivalence test pins
indent == the equivalent parshape across engines.
- LEFTHYPHENMIN / RIGHTHYPHENMIN are parsed from dictionaries and
applied (en_US declares 2/3; the hardcoded 2/2 allowed breaks
like "gen-cy" that the dictionary forbids). Explicit overrides
still win; partial overrides keep the dictionary's other side.
- JIS X 4051 line-start prohibition for small kana, the prolonged
sound mark and iteration marks (new defcustom
ekp-cjk-no-line-start-extra, char-table backed): っょー々 can no
longer start a line. Pure break-permission change; spacing and
the DP are untouched.
Also fixed in passing:
- ekp--split-with-hyphen extracted regexp groups AFTER resolving
the hyphenator; dictionary compilation clobbers the match data
(latent since the resolver was hoisted; exposed by the HYPHENMIN
parser) — groups are captured first now.
- ekp--first-indent-pixel goes through the width cache; it runs
once per rendered line and was re-measuring the reference glyph
every time (C path was 397 ms before, 82 ms after).
4 new ERT tests (92 total); fuzz 300/300 against C 1.5.
C module must be rebuilt: make -C ekp_c clean all
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurement was the largest cost outside the DP and entirely outside
the C module's reach. Boxes repeat across paragraphs constantly
(CJK text re-uses a small alphabet of glyphs), yet each paragraph
measured its own copies: 43% of string-pixel-width calls on the
bundled Chinese sample were cross-paragraph duplicates.
- new session-global box-to-width cache behind all measurement sites
(boxes, styled hyphens, protrusion tails); paragraph-local dedup
stays as the first level. Same machine, byte-compiled, elisp
engine: para-create -26%, justify zh w=400 -41%, justify zh w=200
-8% (cold), no case slower. ekp-clear-caches flushes it (needed
after font/theme changes, as before).
- cache keys (paragraph and width) now strip 'fontified' and other
volatile bookkeeping properties: font-lock flipping 'fontified'
as text scrolled halved the paragraph-cache hit rate in fontified
buffers by aliasing one paragraph into several entries.
- fix: dp-cache results are now keyed by looseness too -- a cached
looseness-0 layout was returned after 'ekp-looseness' changed
(reproduced: 3-line stale result where a fresh run gives 4).
- fix: variable watchers invalidate the same-string fast path when a
style variable changes -- (setq ekp-alignment 'center) kept
returning the paragraph resolved under the previous style for
callers reusing the same string object.
- render path fetches the dp plist once instead of three lookups.
4 new ERT tests (88 total); fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-parshape gives each line its own (INDENT . WIDTH) — TeX \parshape
— with the last entry repeating; ekp-first-line-indent is sugar for
the CJK 2-em paragraph convention (t derives 2 em from the paragraph's
CJK font, or pass pixels). Line-number-dependent widths require the
(position × line-count) DP, so both ride the loose engine and bypass
the C module, exactly like looseness.
Indents render as leading spacers marked ekp-glue, so buffer
roundtrips remain character- and property-exact. Composes with
alignment modes (indent adds to the centering lead) and protrusion.
Tests: 63 ERT green; fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-protrusion enables microtype-style character protrusion at the
right edge: a line ending in fullwidth punctuation (。、」…) hangs
the whitespace half of the glyph past the flush edge — visually
identical to CLREQ line-end punctuation compression — and latin
periods/commas/quotes and soft hyphens protrude by their configured
ratios (ekp-protrusion-ratios).
Mechanics: per-box tail protrusion, folded through trailing space
boxes into a per-gap array; each DP candidate widens its effective
target (lw = width + release), mirrored exactly in the renderer's
per-line effective width and in the C-result reconstruction, so DP,
rendering and both engines stay coherent. Left-edge protrusion is
documented as out of scope (Emacs cannot render before the line
origin). ekp-region reserves the protrusion width in auto layouts
(ekp-region-protrusion-reserve).
Off by default; all-zero arrays make it a no-op on the shared path.
C module 1.4: break-with-arrays 12→14 args (tail-protrudes array,
hyphen-protrude scalar), batch vectors 14 elements.
Tests: 59 ERT green (deterministic hang check, flush-when-off,
protrusion C parity); fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-alignment selects the paragraph alignment. Non-justify modes keep
inter-word glue rigid (arrays and class params zeroed at para build)
while the DP gains a per-line flexibility ekp-ragged-stretch-pixel
(default ≈2 em): badness = 100·(shortfall/R)³, so Knuth-Plass still
globally minimizes raggedness — this is \raggedright with a finite
\rightskip stretch, not first-fit.
Rendering distributes each line's leftover by mode: trailing
(ragged-right), split evenly (center), leading (ragged-left); the
spacers are ekp-glue-marked, so buffer roundtrips stay exact.
C module 1.3: set-penalties gains an extra-stretch scalar (reset when
omitted); max_w widens by it, so the prefix-based flexibility picks it
up automatically. Cache keys include alignment and R.
Tests: 56 ERT green incl. per-mode invariants (fit, rigid interior
glue, flush edges, centered leftover ±1px) and four-mode C parity;
fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- text property ekp-no-break turns any span into a rigid typesetting
atom: no line break inside, no hyphenation, literal spacing (space
runs inside the span become preserved space boxes instead of
stretchable glue) — inline code, product names, numbers with units
- character-driven joins: NBSP, NARROW NBSP, FIGURE SPACE keep their
neighbors together (they are boxes whose adjacent gaps are
unbreakable and glue-free — the character supplies its spacing);
WORD JOINER / ZWNBSP attach invisibly like combining marks
- new commands: ekp-no-break-region / ekp-allow-break-region
- overlong atoms fall back to single emergency lines (atomic-run
handling from the break-permission layer); zero C changes needed
Tests: 52 ERT green incl. atomicity across widths, hyphenation
suppression, NBSP/WJ adjacency sweeps; fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kinsoku moves from merge-based box attachment to a general break-
permission layer, the foundation for no-break spans, NBSP, verbatim
atoms and punctuation protrusion:
- tokenizer: every CJK char (punctuation included) is its own box;
the hold/attach machinery is gone. Fixes two latent bugs: a second
consecutive closer (字。」) could start a line, and an opener held
across spaces reordered box content vs the original string
- ekp--str-type: cjk-punct splits into cjk-open (Ps/Pi) / cjk-close
- ekp-para gains breaks-allowed (bool-vector) + forbidden-positions
(sparse, for C); glue at unbreakable punct gaps is nws, so
punctuation hugs its content and justification stretch no longer
opens gaps at 「x or x。
- halfwidth kinsoku: boxes consisting purely of .,;:!?)]}’”»›… may
not start a line ((&[{‘“«‹ may not end one) — an ASCII comma after
a CJK char no longer dangles at line start (old behavior violated
CLREQ)
- DP (elisp 1D, elisp loose 2D, C): forbidden gaps are skipped as
candidates while the line keeps extending; the emergency fallback
generalizes from single boxes to atomic runs (no permitted break
inside), recording real gap counts so multi-box emergency lines
render correctly
- C module 1.2: ekp-c-break-with-arrays 11→12 args
(forbidden-positions), batch vectors 12 elements, version gate bumped
Hyphenation improvement: 「Hello / Hello」 previously failed the
word regexp as merged boxes and were never hyphenated; as separate
boxes they hyphenate normally.
Tests: 48 ERT green (split tests migrated to the new box contract;
new unit test for break permissions and a rendered-output kinsoku
sweep across widths); 300-case fuzz 0 failures with C 1.2 parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fix: ekp-param-set silently reset after first justify (now persists;
ekp-param-reset added)
- fix: narrow-width CJK returned empty string (data loss); two-pass
emergency-break strategy guarantees output for any input
- fix: K-P penalties never synced to C module; space-box metrics
divergence between C and Elisp engines
- fix: para cache ignored ekp-latin-lang (stale hyphenation after
language switch) and used collision-prone sxhash keys
- fix: fullwidth letters/digits misclassified as CJK punctuation
- fix: combining chars split from their base char in the tokenizer
- fix: punctuation-wrapped words (word!/(word)/word;) never hyphenated
- fix: renderer double-counted stripped space widths; negative glue
clamped; batch/tty font detection no longer crashes
- feat: real looseness support via (position × line-count) DP
- perf: O(1) line metrics and gap counts via prefix arrays (inner loop
previously allocated O(n) subsequences → O(n³) total); box measurement
dedupe; eq fast-path para lookup; prebuilt per-para glue arrays
→ zh justify 7547ms → 96ms (compiled elisp) / 57ms (C);
range-justify 68.5s → 0.48s / 34ms; C module itself 3–19× faster
- test: 36 batch-safe ERT tests + 300-case property fuzz (C/elisp
byte-identical output, zero content loss) replacing ad-hoc suite
- docs: readme/readme_zh/DEVELOPER/DEVELOPER_ZH/ekp_c-README rewritten
to match the implementation; phase handoff in .phrase/phases/
BREAKING: requires Emacs 29.1+; C module must be rebuilt (v1.1, new
arities); ekp-threshold-factor / ekp-flagged-penalty /
ekp-forced-break-penalty removed; Rust module stubs removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
