Three quality gaps closed, one of them unlocking the C engine for
the most common Chinese configuration:
- first-line indent now runs on the 1D DP and the C engine (module
1.5: ekp-c-break-with-arrays grows a 15th arg FIRST-LINE-WIDTH).
A line starts at box 0 exactly when i = 0, so no (position x
line-count) state is needed; only true parshape and looseness
still take the 2D Elisp path. Indented Chinese sample, w=400:
C engine 82 ms (this configuration previously bypassed C entirely
and ran the heaviest Elisp DP). The C-result reconstruction
mirrors the per-line width, and an equivalence test pins
indent == the equivalent parshape across engines.
- LEFTHYPHENMIN / RIGHTHYPHENMIN are parsed from dictionaries and
applied (en_US declares 2/3; the hardcoded 2/2 allowed breaks
like "gen-cy" that the dictionary forbids). Explicit overrides
still win; partial overrides keep the dictionary's other side.
- JIS X 4051 line-start prohibition for small kana, the prolonged
sound mark and iteration marks (new defcustom
ekp-cjk-no-line-start-extra, char-table backed): っょー々 can no
longer start a line. Pure break-permission change; spacing and
the DP are untouched.
Also fixed in passing:
- ekp--split-with-hyphen extracted regexp groups AFTER resolving
the hyphenator; dictionary compilation clobbers the match data
(latent since the resolver was hoisted; exposed by the HYPHENMIN
parser) — groups are captured first now.
- ekp--first-indent-pixel goes through the width cache; it runs
once per rendered line and was re-measuring the reference glyph
every time (C path was 397 ms before, 82 ms after).
4 new ERT tests (92 total); fuzz 300/300 against C 1.5.
C module must be rebuilt: make -C ekp_c clean all
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurement was the largest cost outside the DP and entirely outside
the C module's reach. Boxes repeat across paragraphs constantly
(CJK text re-uses a small alphabet of glyphs), yet each paragraph
measured its own copies: 43% of string-pixel-width calls on the
bundled Chinese sample were cross-paragraph duplicates.
- new session-global box-to-width cache behind all measurement sites
(boxes, styled hyphens, protrusion tails); paragraph-local dedup
stays as the first level. Same machine, byte-compiled, elisp
engine: para-create -26%, justify zh w=400 -41%, justify zh w=200
-8% (cold), no case slower. ekp-clear-caches flushes it (needed
after font/theme changes, as before).
- cache keys (paragraph and width) now strip 'fontified' and other
volatile bookkeeping properties: font-lock flipping 'fontified'
as text scrolled halved the paragraph-cache hit rate in fontified
buffers by aliasing one paragraph into several entries.
- fix: dp-cache results are now keyed by looseness too -- a cached
looseness-0 layout was returned after 'ekp-looseness' changed
(reproduced: 3-line stale result where a fresh run gives 4).
- fix: variable watchers invalidate the same-string fast path when a
style variable changes -- (setq ekp-alignment 'center) kept
returning the paragraph resolved under the previous style for
callers reusing the same string object.
- render path fetches the dp plist once instead of three lookups.
4 new ERT tests (88 total); fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-parshape gives each line its own (INDENT . WIDTH) — TeX \parshape
— with the last entry repeating; ekp-first-line-indent is sugar for
the CJK 2-em paragraph convention (t derives 2 em from the paragraph's
CJK font, or pass pixels). Line-number-dependent widths require the
(position × line-count) DP, so both ride the loose engine and bypass
the C module, exactly like looseness.
Indents render as leading spacers marked ekp-glue, so buffer
roundtrips remain character- and property-exact. Composes with
alignment modes (indent adds to the centering lead) and protrusion.
Tests: 63 ERT green; fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-protrusion enables microtype-style character protrusion at the
right edge: a line ending in fullwidth punctuation (。、」…) hangs
the whitespace half of the glyph past the flush edge — visually
identical to CLREQ line-end punctuation compression — and latin
periods/commas/quotes and soft hyphens protrude by their configured
ratios (ekp-protrusion-ratios).
Mechanics: per-box tail protrusion, folded through trailing space
boxes into a per-gap array; each DP candidate widens its effective
target (lw = width + release), mirrored exactly in the renderer's
per-line effective width and in the C-result reconstruction, so DP,
rendering and both engines stay coherent. Left-edge protrusion is
documented as out of scope (Emacs cannot render before the line
origin). ekp-region reserves the protrusion width in auto layouts
(ekp-region-protrusion-reserve).
Off by default; all-zero arrays make it a no-op on the shared path.
C module 1.4: break-with-arrays 12→14 args (tail-protrudes array,
hyphen-protrude scalar), batch vectors 14 elements.
Tests: 59 ERT green (deterministic hang check, flush-when-off,
protrusion C parity); fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-alignment selects the paragraph alignment. Non-justify modes keep
inter-word glue rigid (arrays and class params zeroed at para build)
while the DP gains a per-line flexibility ekp-ragged-stretch-pixel
(default ≈2 em): badness = 100·(shortfall/R)³, so Knuth-Plass still
globally minimizes raggedness — this is \raggedright with a finite
\rightskip stretch, not first-fit.
Rendering distributes each line's leftover by mode: trailing
(ragged-right), split evenly (center), leading (ragged-left); the
spacers are ekp-glue-marked, so buffer roundtrips stay exact.
C module 1.3: set-penalties gains an extra-stretch scalar (reset when
omitted); max_w widens by it, so the prefix-based flexibility picks it
up automatically. Cache keys include alignment and R.
Tests: 56 ERT green incl. per-mode invariants (fit, rigid interior
glue, flush edges, centered leftover ±1px) and four-mode C parity;
fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- text property ekp-no-break turns any span into a rigid typesetting
atom: no line break inside, no hyphenation, literal spacing (space
runs inside the span become preserved space boxes instead of
stretchable glue) — inline code, product names, numbers with units
- character-driven joins: NBSP, NARROW NBSP, FIGURE SPACE keep their
neighbors together (they are boxes whose adjacent gaps are
unbreakable and glue-free — the character supplies its spacing);
WORD JOINER / ZWNBSP attach invisibly like combining marks
- new commands: ekp-no-break-region / ekp-allow-break-region
- overlong atoms fall back to single emergency lines (atomic-run
handling from the break-permission layer); zero C changes needed
Tests: 52 ERT green incl. atomicity across widths, hyphenation
suppression, NBSP/WJ adjacency sweeps; fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kinsoku moves from merge-based box attachment to a general break-
permission layer, the foundation for no-break spans, NBSP, verbatim
atoms and punctuation protrusion:
- tokenizer: every CJK char (punctuation included) is its own box;
the hold/attach machinery is gone. Fixes two latent bugs: a second
consecutive closer (字。」) could start a line, and an opener held
across spaces reordered box content vs the original string
- ekp--str-type: cjk-punct splits into cjk-open (Ps/Pi) / cjk-close
- ekp-para gains breaks-allowed (bool-vector) + forbidden-positions
(sparse, for C); glue at unbreakable punct gaps is nws, so
punctuation hugs its content and justification stretch no longer
opens gaps at 「x or x。
- halfwidth kinsoku: boxes consisting purely of .,;:!?)]}’”»›… may
not start a line ((&[{‘“«‹ may not end one) — an ASCII comma after
a CJK char no longer dangles at line start (old behavior violated
CLREQ)
- DP (elisp 1D, elisp loose 2D, C): forbidden gaps are skipped as
candidates while the line keeps extending; the emergency fallback
generalizes from single boxes to atomic runs (no permitted break
inside), recording real gap counts so multi-box emergency lines
render correctly
- C module 1.2: ekp-c-break-with-arrays 11→12 args
(forbidden-positions), batch vectors 12 elements, version gate bumped
Hyphenation improvement: 「Hello / Hello」 previously failed the
word regexp as merged boxes and were never hyphenated; as separate
boxes they hyphenate normally.
Tests: 48 ERT green (split tests migrated to the new box contract;
new unit test for break permissions and a rendered-output kinsoku
sweep across widths); 300-case fuzz 0 failures with C 1.2 parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- fix: ekp-param-set silently reset after first justify (now persists;
ekp-param-reset added)
- fix: narrow-width CJK returned empty string (data loss); two-pass
emergency-break strategy guarantees output for any input
- fix: K-P penalties never synced to C module; space-box metrics
divergence between C and Elisp engines
- fix: para cache ignored ekp-latin-lang (stale hyphenation after
language switch) and used collision-prone sxhash keys
- fix: fullwidth letters/digits misclassified as CJK punctuation
- fix: combining chars split from their base char in the tokenizer
- fix: punctuation-wrapped words (word!/(word)/word;) never hyphenated
- fix: renderer double-counted stripped space widths; negative glue
clamped; batch/tty font detection no longer crashes
- feat: real looseness support via (position × line-count) DP
- perf: O(1) line metrics and gap counts via prefix arrays (inner loop
previously allocated O(n) subsequences → O(n³) total); box measurement
dedupe; eq fast-path para lookup; prebuilt per-para glue arrays
→ zh justify 7547ms → 96ms (compiled elisp) / 57ms (C);
range-justify 68.5s → 0.48s / 34ms; C module itself 3–19× faster
- test: 36 batch-safe ERT tests + 300-case property fuzz (C/elisp
byte-identical output, zero content loss) replacing ad-hoc suite
- docs: readme/readme_zh/DEVELOPER/DEVELOPER_ZH/ekp_c-README rewritten
to match the implementation; phase handoff in .phrase/phases/
BREAKING: requires Emacs 29.1+; C module must be rebuilt (v1.1, new
arities); ekp-threshold-factor / ekp-flagged-penalty /
ekp-forced-break-penalty removed; Rust module stubs removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
