ekp/DEVELOPER.md

Developer Documentation for Emacs-KP

This document describes the internal architecture, algorithms and APIs of emacs-kp, as implemented. It is intended for contributors and advanced users.

1. Pipeline Overview

A justification call flows through five stages:

 string
   │
   ▼
 ① Tokenize          ekp-split-to-boxes           (ekp-utils.el)
   │                 Latin words / CJK chars / space runs → boxes,
   │                 kinsoku attachment of CJK punctuation
   ▼
 ② Hyphenate         ekp--split-with-hyphen        (ekp.el + ekp-hyphen.el)
   │                 Latin word boxes → syllable boxes (Liang patterns)
   ▼
 ③ Measure & index   ekp--make-para                (ekp.el)
   │                 pixel widths, glue types, prefix-sum arrays
   │                 → cached `ekp-para` struct
   ▼
 ④ Break (DP)        ekp--dp-run-1d / C module     (ekp.el / ekp_c/)
   │                 Knuth-Plass dynamic program → break positions
   ▼
 ⑤ Render            ekp-line-glues, ekp--pixel-justify
                     distribute glue pixels, strip edge space boxes,
                     attach hyphens → lines joined with "\n"

ekp-pixel-justify splits its input on "\n" and runs each non-blank segment through this pipeline as an independent paragraph (in parallel via the C batch API when available).

2. Data Structures

ekp-para (the paragraph cache entry)

Everything the DP and renderer need, computed once per paragraph:

Field	Contents
string, latin-font, cjk-font	source text and detected fonts
boxes	vector of box strings
boxes-widths	pixel width per box (measured with deduplication)
boxes-types	(START-TYPE . END-TYPE) per box: latin/cjk/cjk-punct/space
glues-types	glue class before each box: lws/mws/cws/nws
hyphen-pixel, hyphen-positions	hyphen width; sorted vector of box indices after which a hyphen may be inserted
ideal/min/max-prefixs	prefix sums of box+glue widths at ideal / max-shrunk / max-stretched (n+1 elements)
glue-ideals/shrinks/stretches	leading-glue values per box index (n elements) — also passed verbatim to C
lws/mws/cws-prefixs	prefix counts of each stretchable glue class → O(1) gap counting per candidate line
lead-spaces	lead-spaces[i] = width of the space-box run starting at box i; index 0 forced to 0 (first-line indentation is kept)
trail-spaces	trail-spaces[k] = width of the space-box run ending at box k−1
glue-params	plist snapshot of the nine spacing values at creation time
dp-cache	hash: line-width → dp-result plist

The paragraph cache (ekp--para-cache) is keyed with equal on a structured key — string content, printed text-property intervals, detected fonts, the hyphenation language (ekp-latin-lang), and either the nine explicit spacing values or the symbol auto. Structured keys make hash collisions harmless (they were possible with the previous sxhash-integer scheme). The cache is flushed when it exceeds ekp-para-cache-limit. A one-entry fast path (ekp--last-para, checked by string eq + language) covers the many same-string lookups inside one justification call.

dp-result

(:rests R :gaps G :breaks B :cost C :line-count N) where breaks are exclusive end indices per line, rests[i] = line-width − line-ideal (the pixels the glue must absorb), gaps[i] = (lws-count mws-count cws-count) for glue distribution (nil for single-box and last lines).

3. Line Metrics

For a candidate line spanning boxes [i, k):

raw       = prefix[k] − prefix[i] − leading-glue(i)
space-w   = min(raw, lead-spaces[i] + trail-spaces[k])
width     = raw − space-w  (+ hyphen-pixel if box k−1 hyphenates)

computed for ideal, min and max in O(1). Space-box runs at the line edges are excluded because the renderer strips them; the DP and the renderer therefore agree exactly, and every justified line renders at precisely the target width (ekp-test-justify-line-width-invariant).

4. The Knuth-Plass DP

ekp--dp-run-1d relaxes positions left to right. For each reachable start i it scans end positions k until the line's minimum width exceeds the target. A break at k is valid when min ≤ target ≤ max, or for the last line when ideal ≤ target.

Demerits (per line, matching ekp_c/ekp_kp.c exactly):

demerits = (line-penalty + badness)²
         + penalty²                       ; hyphen-penalty at hyphen breaks
         + adjacent-fitness-penalty       ; if |fitness − prev-fitness| > 1
         + consecutive-hyphen-penalty × run²
badness  = min(10000, 100·|adjustment/flexibility|³)

Fitness classes (tight/decent/loose/very-loose) follow the TeX ratio thresholds. Special cases: single-box lines use flexibility 1 and fitness decent; the last line pays (line-penalty + short-badness)² where short-badness = last-line-short-penalty × (1 − fill) when the fill ratio is below ekp-last-line-min-ratio.

Deviations from the 1981 paper, by design: penalties are always added as +p² (no negative/flagged penalties), there is no q/looseness in the main pass (see §6), and adjacent-fitness is a flat constant.

Two-pass emergency strategy

Some inputs admit no valid layout: an unbreakable box wider than the line, or a rigid (all-nws) region that cannot stretch to the target. A strict pass runs first; if the paragraph end is unreachable, a second pass additionally allows emergency breaks — single-box lines with demerits (line-penalty + 10000)² + rest², at least as bad as any regular line. This guarantees, by induction over positions, that every input produces output (regression: narrow CJK used to return an empty string), while the common case pays nothing and keeps pure K-P optimality. Both engines implement the identical strategy.

5. Rendering

ekp-line-glues turns each line's rest into per-glue pixel values:

• rest > 0 → stretch, distributed latin → mixed → CJK; CJK gaps absorb any leftover beyond nominal capacity (emergency spreading).
• rest < 0 → shrink, same priority order, never below the per-class shrink limit; glue widths are clamped at ≥ 0.
• Last lines are ragged-right (ideal glues + trailing filler); single-box lines get a trailing filler clamped at ≥ 0.

ekp--pixel-justify then strips leading space boxes (except on the first line — indentation) and trailing space boxes, and appends a hyphen — propertized like the word it breaks — where a line ends at a hyphenation point. Stripped widths are not redistributed: the DP already excluded them (§3).

Glues become (space :width (N)) display properties, so justification is pixel-exact in GUI Emacs and column-exact in batch/tty.

The output is lossless: boxes are located in the source string (ekp--box-offsets), and every synthesized or hidden piece records the original text it stands for —

property	on	value / meaning
ekp-glue	synthesized glue space	original text it replaced
ekp-soft-break	inserted \n	whitespace swallowed at break
ekp-soft-hyphen	inserted hyphen	marker only
ekp-hidden	paragraph-edge text	kept verbatim, display ""

Zero-width glue with a non-empty original renders as the hidden original itself, so no character is ever dropped. ekp-region.el inverts these four structurally (ekp-unjustify-region) — exact even after the justified text was edited — and builds ekp-justify-region / ekp-auto-justify-mode on top.

5.1 Break permissions, alignment, protrusion, shapes

• Break permissions: every CJK char (punctuation included) is its own box; ekp-para-breaks-allowed forbids gaps per kinsoku (full- and halfwidth), ekp-no-break' spans and NBSP-family joiners. Forbidden gaps carry no glue. The DP skips them as candidates while the line keeps extending; the emergency fallback treats a run with no permitted inner break as atomic. C receives the sparse forbidden-positions` vector.
• Alignment (ekp-alignment): non-justify modes zero the glue stretch/shrink arrays and class params; the DP widens max_w by an extra per-line stretch R (ekp-c-set-penalties arg 7), so badness = 100·(shortfall/R)³. The renderer places each line's leftover per mode (trailing / split / leading).
• Protrusion (ekp-protrusion): per-gap tail-protrudes[k] (the last non-space box's allowance, looked through trailing spaces) and a hyphen-protrude scalar widen each candidate's effective target (lw = width + release) in the DP, in `ekp-line-glues', and in the C-result reconstruction — all three must stay in lockstep.
• Per-line widths (ekp-parshape' / ekp-first-line-indent'): resolved by ekp--line-spec' (line-index → INDENT . WIDTH); they require the (position × line-count) DP and bypass C, like looseness. Indents render as leading ekp-glue' spacers.

C module 1.4: ekp-c-break-with-arrays takes 14 args (…, forbidden-positions, tail-protrudes, hyphen-protrude); batch vectors have 14 elements; ekp-c-set-penalties takes 4–7.

Performance after the feature wave (byte-compiled + C, Apple Silicon, batch): justify zh w=200 ≈ 54 ms, range zh ≈ 117 ms — justify at parity with the pre-feature numbers, range ≈ +55 % from the larger box count. Hot-path caches: per-char `ekp--str-type' memo, interned glue strings, per-(paragraph, width) rendered-output cache in the dp-cache (capped at 64 widths). Continuous-reflow reality check (60-paragraph, 26 k-char article, region layer included): ≈ 73 ms per width change cold, less on revisit; incremental single-paragraph re-justify after an edit ≈ 17 ms.

6. Looseness

ekp-looseness ≠ 0 switches to ekp--dp-run-loose, a full (position × line-count) DP that keeps the best path per line count, then picks the final count closest to (optimal + looseness), breaking ties by demerits. This is heavier than the 1D pass and is Elisp-only; ekp--c-available-p returns nil while looseness is active so both engines never disagree.

7. C Module Integration

The C module (ekp_c/, version 1.1) runs only stage ④. Elisp remains the source of truth for all font-dependent data.

• ekp-c-break-with-arrays (11 args): the para's prefix arrays, glue arrays, hyphen data, line width and the two space-run arrays. Returns (breaks . cost).
• ekp-c-break-batch: a vector of 11-element vectors, processed in parallel by a pthread pool — one task per paragraph (that is the correct granularity; the DP itself is sequential by nature).
• ekp-c-set-penalties (4–6 args): called by ekp--c-sync-params before every C entry, so ekp-line-penalty & friends always take effect (regression: they were never synced before).
• ekp-c-module-load refuses modules older than ekp-c-module-required-version and falls back to Elisp, preventing arity mismatches after upgrades.

Any C failure (NULL result) silently falls back to the Elisp engine. The two engines are verified to produce byte-identical output by ekp-test-c-parity-simple / ekp-test-c-parity-files.

ekp-c-break-lines (C-side tokenization via ekp_paragraph.c and ekp_hyphen.c) is an experimental, self-contained path that ekp.el does not use; see ekp_c/README.md.

8. Hyphenation (ekp-hyphen.el)

Liang's pattern algorithm, Pyphen-compatible:

• dictionaries/hyph_*.dic are compiled to a pattern hash on first use and cached per path. Files may be UTF-8 or ISO-8859 (Emacs auto-detects; verified by ekp-test-hyphen-de-iso8859-dict).
• ekp-hyphen-create LANG resolves exact codes, then progressively shorter prefixes ("de_CH" → "de").
• Margins default to 2 characters on each side of a break.

Word boxes are matched against ^[left-punct]* (latin-word) [right-punct]*$ so that punctuation- wrapped words ((word), word!, »word«) still hyphenate; the punctuation stays glued to the first/last syllable box.

9. Testing & Benchmarks

tests/run-tests.sh [emacs]        # 36 ERT tests, batch-safe
emacs -Q --batch -L . --eval '(setq ekp-use-c-module nil)' -l tests/ekp-bench.el
emacs -Q --batch -L . --eval '(progn (require (quote ekp)) (ekp-c-module-load))' \
      -l tests/ekp-bench.el

Key invariants under test: rendered line width == target (pixel-exact justification), no content loss at any width, brute-force cross-checks of the O(1) prefix machinery, Elisp/C parity on the bundled texts, and parameter persistence/sync regressions.

Benchmark results (batch Emacs 30.2, Apple Silicon M-series, tests/text-zh.txt ≈ 3.6 KB Chinese + samples; min of 3 cold-cache runs) — before is the pre-rewrite implementation, interpreted:

Case	Before (Elisp)	After (Elisp, interpreted)	After (Elisp, compiled)	After (C)
justify zh w=200	7547 ms	1780 ms	96 ms	57 ms
justify zh w=400	2928 ms	815 ms	71 ms	57 ms
justify mixed w=300	5540 ms	1275 ms	53 ms	23 ms
range-justify zh 340–380	29696 ms	8937 ms	294 ms	75 ms
range-justify mix 280–320	68534 ms	14552 ms	480 ms	34 ms
DP only, zh w=400	2382 ms	591 ms	15 ms	1.3 ms

("After (C)" columns measured with byte-compiled Elisp around the C calls. For reference, the pre-rewrite C module measured 197 ms / 430 ms / 25 ms on justify-zh-200 / range-zh / DP-only — the rewrite also sped up the C path 3–19× via prebuilt per-para glue arrays, an `eq' fast path in the para cache, and O(1) rest/gap reconstruction.)

The dominant wins: O(1) line metrics via prefix arrays (the old inner loop allocated O(n) subsequences per candidate, O(n³) total), the two-pass emergency strategy (keeps the DP sparse), box-measurement deduplication, and per-para glue arrays reused across C calls.

10. File Map

ekp.el            Core: para struct, caching, DP (1D + looseness),
                  glue distribution, rendering, public API
ekp-utils.el      Tokenizer (boxes, kinsoku), font detection with
                  batch/tty fallbacks, C module loading
ekp-hyphen.el     Liang hyphenation + dictionary registry
ekp_c/            C dynamic module (see ekp_c/README.md)
dictionaries/     Hunspell hyphenation patterns (from Pyphen)
tests/            ekp-tests.el (ERT), ekp-bench.el, ekp-demo.el,
                  sample texts, run-tests.sh
archive/          Historical prototypes; not loaded, kept for reference