Reuse prepared paragraph data, safe Elisp DP states, common layout lines, and dirty-island source reconstruction for exact append commits. Simplify C int32 validation and add the frozen task030 evaluator, regressions, and decision records.
Replace the physical ekp-region renderer with ekp-buffer text-property projection backed by shared semantic layout plans and stable live-edit transactions.
Optimize the C-backed resize hot path, and add deterministic performance, ERT, GUI, fuzz, release, and documentation coverage for task017 through task032.
BREAKING CHANGE: require ekp-buffer instead of ekp-region and rename module-owned ekp-region-* settings to ekp-buffer-*.
Complete task002 through task016 across cache correctness, editor lifecycle, C boundaries, build and release governance, dictionary provenance, performance, interactive workflows, tests, documentation, and final cleanup.
Two root causes made justified text overrun the window's right edge
in real user sessions (every line ending in the "$" truncation
marker), while emacs -Q looked fine:
1. Windows WITHOUT fringes draw the truncation/continuation
indicator in the text area's LAST COLUMN, so the usable width is
one character less than window-body-width. ekp reserved only a
2px margin: with fringes disabled (a common minimal setup, and
all ttys) every line that hit the target width exactly had its
final glyph displaced by the "$". ekp-region--window-pixel now
reserves one frame-char-width when the window has no right
fringe. (The same lesson ebox-playground encodes in its
viewport-width reserve.)
2. Measurement was blind to the buffer's display context:
string-pixel-width works in a bare hidden buffer, ignoring
face-remapping-alist — which is where text-scale-mode, themes
and per-buffer font tweaks live. Under a remap, the DP laid
lines out with one font's metrics and the display rendered them
with another's: scale +3 made 5 of 7 sample lines overflow a
1330px window by up to 900px (GUI-measured). Measurement now
runs with the destination buffer's face-remapping-alist (the
29/30-compatible equivalent of Emacs 31's string-pixel-width
BUFFER argument), and the width/paragraph caches key on that
context so buffers at different scales never alias. text-scale
changes also trigger a re-flow in ekp-auto-justify-mode.
Ground truth, measured with window-text-pixel-size in GUI Emacs
across 7 display contexts (plain / text-scale ±| face remap /
no-fringes / no-fringes+scale / narrow+scale): the widest justified
line equals the target width exactly in every case, zero lines
overflow. Verbatim code blocks are exempt by design (they never
reflow, like any code line in a narrow window).
New tooling so this never regresses invisibly:
- M-x ekp-diagnose: renders a probe line in YOUR buffer and reports
target vs rendered width — run it in any session where justified
text looks wrong.
- tests/ekp-gui-verify.el: M-x ekp-gui-verify (single check in a
customized session) and ekp-gui-verify-matrix (the 7-case table,
for emacs -Q).
- 2 new batch ERT tests pin the context-keyed caches (94 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
showcase: the demo font was applied with face-remap-add-relative, but
string-pixel-width (how ekp measures) ignores buffer face remapping
while honoring an explicit :family — so measurement used the session
default font and rendering used the demo font. On a wide follow-
window that per-glyph discrepancy accumulated past the safety margin
and every justified line overflowed the right edge (invisible at the
narrow 480px default). Pin the font as an appended :family face text
property instead, so measurement and rendering agree; inline atom/
code faces keep precedence.
checkdoc: all four elisp files are now checkdoc-clean (docstrings,
argument mentions, sentence spacing) with no logic change; the three
validation messages read naturally again (dropped the capitalized
"Ekp:" prefix checkdoc's capital-letter rule had forced).
92 ERT green; fuzz 300/300 with C parity; byte-compile warning-free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assessed the C marshal cost (dominates range-justify: ~12 ms/width
warm, mostly env extraction, not DP) and deliberately deferred the
make_user_ptr handle API: it is a breaking 2.0 ABI change introducing
C-side object lifetime, and the interactive paths already avoid the
repeat marshal via the dp-cache. Documented in both DEVELOPER guides
as the clear next step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- readme_zh: installation, editor-integration behavior, JIS kinsoku,
HYPHENMIN, refreshed perf table (parity with English readme)
- DEVELOPER/DEVELOPER_ZH: C module 1.5 (15-arg API, first-line-width,
lazy pool, clean failure fallback), first-line indent on the 1D/C
path, file map includes ekp-region.el and the real tests/ layout
- correct dictionary attribution to LibreOffice (per dictionaries/
update.sh and the bundled README_hyph_*.txt)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The self-contained C path (ekp-c-break-lines, ekp-c-load-hyphenator,
ekp-c-hyphenate, ekp-c-set-spacing, ekp_paragraph.c, ekp_hyphen.c,
~1500 lines) was never used by ekp.el, diverged semantically from the
real pipeline (no kinsoku, no protrusion, no two-pass emergency), and
contained an exploitable heap overflow reachable from Lisp:
ekp_para_create sized its box array as box_count * 2, but a long word
hyphenates into arbitrarily many syllable boxes, overflowing the
calloc'd buffer. Deleting the path deletes the bug class.
Hardening of the live path:
- thread pool: sized from the machine's core count instead of a
hardcoded 8; created lazily on the first multi-paragraph batch
(single-paragraph users never start worker threads); a full queue
now blocks the submitter until a worker makes room — tasks were
silently dropped before, degrading the batch to the Elisp fallback
exactly when parallelism mattered most.
- unified failure gate: a partial allocation used to silently drop
kinsoku, hyphenation or protrusion data and continue with a subtly
different layout; any allocation failure or pending Lisp signal
(non-local exit from a bad element type) now fails the whole call,
and ekp.el falls back to the Elisp engine. The Elisp bridge wraps
both C entry points in condition-case, and a whole-batch nil no
longer crashes the per-paragraph loop.
- integer safety: every extracted pixel value is clamped to int32
instead of silently wrapping.
- EKP_INFINITY (the unreachable-state sentinel) is now a real
infinity: extremely degenerate paragraphs could legitimately
accumulate demerits past the old 1e10 constant, making C consider
reachable states dead and diverge from the Elisp engine.
BREAKING: the four experimental module functions are gone; rebuild
with make -C ekp_c clean all (version gate unchanged at 1.5).
92 ERT green; fuzz 300/300 byte-identical across engines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three quality gaps closed, one of them unlocking the C engine for
the most common Chinese configuration:
- first-line indent now runs on the 1D DP and the C engine (module
1.5: ekp-c-break-with-arrays grows a 15th arg FIRST-LINE-WIDTH).
A line starts at box 0 exactly when i = 0, so no (position x
line-count) state is needed; only true parshape and looseness
still take the 2D Elisp path. Indented Chinese sample, w=400:
C engine 82 ms (this configuration previously bypassed C entirely
and ran the heaviest Elisp DP). The C-result reconstruction
mirrors the per-line width, and an equivalence test pins
indent == the equivalent parshape across engines.
- LEFTHYPHENMIN / RIGHTHYPHENMIN are parsed from dictionaries and
applied (en_US declares 2/3; the hardcoded 2/2 allowed breaks
like "gen-cy" that the dictionary forbids). Explicit overrides
still win; partial overrides keep the dictionary's other side.
- JIS X 4051 line-start prohibition for small kana, the prolonged
sound mark and iteration marks (new defcustom
ekp-cjk-no-line-start-extra, char-table backed): っょー々 can no
longer start a line. Pure break-permission change; spacing and
the DP are untouched.
Also fixed in passing:
- ekp--split-with-hyphen extracted regexp groups AFTER resolving
the hyphenator; dictionary compilation clobbers the match data
(latent since the resolver was hoisted; exposed by the HYPHENMIN
parser) — groups are captured first now.
- ekp--first-indent-pixel goes through the width cache; it runs
once per rendered line and was re-measuring the reference glyph
every time (C path was 397 ms before, 82 ms after).
4 new ERT tests (92 total); fuzz 300/300 against C 1.5.
C module must be rebuilt: make -C ekp_c clean all
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurement was the largest cost outside the DP and entirely outside
the C module's reach. Boxes repeat across paragraphs constantly
(CJK text re-uses a small alphabet of glyphs), yet each paragraph
measured its own copies: 43% of string-pixel-width calls on the
bundled Chinese sample were cross-paragraph duplicates.
- new session-global box-to-width cache behind all measurement sites
(boxes, styled hyphens, protrusion tails); paragraph-local dedup
stays as the first level. Same machine, byte-compiled, elisp
engine: para-create -26%, justify zh w=400 -41%, justify zh w=200
-8% (cold), no case slower. ekp-clear-caches flushes it (needed
after font/theme changes, as before).
- cache keys (paragraph and width) now strip 'fontified' and other
volatile bookkeeping properties: font-lock flipping 'fontified'
as text scrolled halved the paragraph-cache hit rate in fontified
buffers by aliasing one paragraph into several entries.
- fix: dp-cache results are now keyed by looseness too -- a cached
looseness-0 layout was returned after 'ekp-looseness' changed
(reproduced: 3-line stale result where a fresh run gives 4).
- fix: variable watchers invalidate the same-string fast path when a
style variable changes -- (setq ekp-alignment 'center) kept
returning the paragraph resolved under the previous style for
callers reusing the same string object.
- render path fetches the dp plist once instead of three lookups.
4 new ERT tests (88 total); fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.phrase/ (phase handoffs, agent role modules) and AGENTS.md (a
personal cross-project agent protocol referencing .phrase/) are
local working files, not project documentation. They stay on disk,
untracked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- a dirty area larger than the lazy threshold (big paste, revert)
is chunked into the background queue instead of freezing the
command loop with one synchronous re-justification
- queued chunks intersecting the visible span move to the queue
front on scroll (window-scroll-functions): scrolling into an
unprocessed area no longer waits for the whole queue
- background ticks work under a time budget
(ekp-auto-justify-tick-budget, default 5 ms): several small
chunks per tick, guaranteed progress of at least one, peek-then-
pop so an aborted justification never loses a chunk
- ticks also yield while an input-method composition is active
3 new ERT tests (84 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- new commands ekp-justify-buffer / ekp-unjustify-buffer
- ekp-justify-region / ekp-unjustify-region without an active region
now act on the hard paragraph at point (DWIM) instead of erroring
- ekp-auto-justify-mode remaps fill-paragraph to the new
ekp-refill-paragraph — plain M-q treated glue spaces and soft
breaks as content and destroyed the original whitespace
- ekp-org-setup / ekp-markdown-setup wire the existing skip-face
presets in one call; enabling the mode in an Org/Markdown buffer
applies them automatically unless the user configured their own.
The markdown setup also stops font-lock from managing the display
property: refontification was stripping the pixel glue and
wrecking the layout permanently.
- justify runs font-lock-ensure over the region first when
face-based verbatim detection is configured — paragraphs jit-lock
had never fontified (off-screen code blocks) were not skipped
- showcase/demo buffers disable undo recording (width sweeps
generated thousands of useless undo entries)
5 new ERT tests (81 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- isearch searches the logical text: entering isearch temporarily
un-justifies (spans remembered as markers), exiting restores the
layout. Justified CJK lines contain real injected spaces, which
made even same-line phrases unfindable.
- the kill ring receives the logical text via a buffer-local
filter-buffer-substring-function: yanking justified text elsewhere
transports words, not the source window's pixel layout.
- multi-window: the layout width is now the narrowest window showing
the buffer (was: whichever window resized last, leaving the other
window overflowing), and window-configuration-change-hook catches
the buffer becoming displayed, splits, and closed windows.
- paragraphs containing field or read-only text (comint prompts)
are never re-written; interactive justify/unjustify on read-only
buffers signals instead of editing through inhibit-read-only.
- edit re-flows wait for an active input-method composition (quail
preedit) to finish instead of re-writing the text under it.
4 new ERT tests (76 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Justification is a re-layout of the same logical text; it now behaves
that way toward the rest of the editor:
- saving a justified buffer writes the LOGICAL text: before-save-hook
unjustifies (spans remembered as markers), after-save-hook restores
the justified view; the deterministic pair stays off undo history.
Previously soft newlines and break hyphens were silently persisted
to disk and the original whitespace was unrecoverable after reopen.
- renderer marker properties (ekp-glue/soft-break/soft-hyphen/hidden/
justified) are registered in text-property-default-nonsticky: text
typed after a glue no longer inherits the marker and is no longer
deleted as a synthesized space by the next re-flow.
- pure re-layout preserves buffer-modified-p when it was nil: no more
lock files, auto-saves and "buffer modified" prompts from merely
enabling ekp-auto-justify-mode.
- changes applied by undo are not re-dirtied (no timer war against
the user's undo sequence).
- switching major modes tears justified state down cleanly
(change-major-mode-hook), widening first so narrowing never leaves
justified orphans.
5 new ERT tests (72 total).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- add COPYING (GPL-3.0-or-later) and license headers to all elisp
and C sources; real author/maintainer info replaces placeholders
- convert user-facing options to defcustom under new group `ekp'
(spacing internals managed by ekp-param-set stay defvars)
- autoload user commands: ekp-param-reset, ekp-clear-caches,
ekp-c-module-load, ekp-c-module-build
- ekp--load-dicts: a missing dictionaries/ directory now only
disables hyphenation instead of breaking (require 'ekp);
ekp--split-with-hyphen degrades gracefully (and resolves the
hyphenator once per call instead of once per word)
- package summary no longer redundantly says "for Emacs"
- Makefile: drop personal Windows EMACS_ROOT default; add guidance
- ekp_c/README.md: fix stale 1.1 version references (module is 1.4)
- DEVELOPER*.md: remove archive/ from file map (not in the repo)
- remove unreferenced 10 MB demo GIF
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keys are now always visible at the top (help-key-binding key caps),
and the live state readout — width, alignment, protrusion, indent,
parshape, effective engine, last re-flow ms — moves to the mode line,
so both are permanently on screen without fighting for space.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One self-contained demo buffer for every feature, driven by single
keys, with per-reflow milliseconds in the header line:
-/+ width ±20 px (arrows: ±4) d animated width sweep + fps report
a cycle 4 alignments p hanging punctuation
i first-line indent (2 em) s wedge parshape (Elisp 2D path)
c C engine vs pure Elisp w follow window width (auto-mode)
The sample text bundles a verbatim-protected code block, an inline
no-break atom, and NBSP-joined figures; the engine indicator honestly
shows when a feature (indent/parshape) bypasses C. Launch:
emacs -Q -L /path/to/emacs-kp -l tests/ekp-showcase.el -f ekp-showcase
Verified interactively in GUI Emacs (justify 480→360 px, protrusion +
indent on: state and timing update live).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Buffers past ekp-auto-justify-lazy-threshold (20k chars) no longer
re-justify wholesale on a width change: the visible span updates
synchronously and the rest follows in polite background chunks
(run-with-timer ticks that yield to pending input). Measured on a
300-paragraph, 132k-char article: perceived latency 332 ms → 15 ms,
background completion ~0.5 s.
Fixed in the process: adjacent chunk markers shared a boundary, and
the earlier chunk's delete+insert pushed the next chunk's start
marker back over the freshly inserted text — chunks grew linearly
(2.2k, 4.9k, 7.4k… chars) and background work went quadratic (14 s).
Start markers now use insertion-type t. Regression test asserts the
chunked result equals the one-shot result property-for-property.
Also: org/markdown skip-face presets (ekp-region-org-skip-faces /
ekp-region-markdown-skip-faces), MELPA header hygiene (Version + URL
on the main file, stale Package-Requires dropped from ekp-hyphen.el),
readme notes.
67 ERT green; fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Investigation: the apparent 7× bench regression was interpreted-mode
measurement (tests always deleted .elc); byte-compiled + C the real
P1/P2 cost was +70–90%. Worktree A/B across phase commits confirmed
the features themselves added ~+20% interpreted.
Optimizations (all verified by the 66-test suite + 300-case fuzz):
- per-char memo table for ekp--str-type (get-char-code-property was
the para-build hot spot)
- precomputed punctuation char lists (no per-call list rebuild)
- ekp--box-offsets memoized in the paragraph struct
- interned glue strings for the two dominant payload shapes
- rendered-output cache per (paragraph, width) inside dp-cache
(capped at 64 widths per paragraph), so resize sweeps that revisit
a width skip rendering entirely
- unjustify walks hop property boundaries instead of chars
Numbers (byte-compiled + C): justify zh w=200 53.8 ms — at parity
with the pre-feature record (57 ms) despite punctuation boxes and
break permissions; range zh 117 ms (+55%, inherent box-count cost).
Continuous reflow, 60-paragraph 26k-char article through the full
region layer: 187 ms → 73 ms per width change; incremental
single-paragraph re-justify after an edit: 62 ms → 17 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
readme/readme_zh gain Typography and verbatim-protection sections
(alignment, hanging punctuation, parshape/first-line indent,
no-break characters, halfwidth kinsoku) plus the honest limitation
notes (mid-line CLREQ compression and left protrusion are not
renderable in Emacs). DEVELOPER/DEVELOPER_ZH document the break-
permission layer, alignment extra-stretch, protrusion lockstep and
per-line specs (§5.1) and the C 1.4 API surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paragraphs stay verbatim (never justified, byte-identical through
justify/auto-mode/unjustify) when they carry the ekp-verbatim text
property, wear a face from ekp-region-skip-faces (point it at
org-block / markdown-code-face and code blocks are exempt), or match
the buffer-local ekp-region-skip-predicate — three layers of the same
general mechanism, no mode-specific code. Inline code inside prose
is the existing ekp-no-break atom (rigid, unbreakable, unhyphenated).
New commands: ekp-verbatim-region / ekp-clear-verbatim-region.
Core engine untouched: block protection lives entirely in the region
layer, so skipped text never even reaches the tokenizer.
Tests: 66 ERT green (property / face / predicate paths, exact
roundtrips around justified prose).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-parshape gives each line its own (INDENT . WIDTH) — TeX \parshape
— with the last entry repeating; ekp-first-line-indent is sugar for
the CJK 2-em paragraph convention (t derives 2 em from the paragraph's
CJK font, or pass pixels). Line-number-dependent widths require the
(position × line-count) DP, so both ride the loose engine and bypass
the C module, exactly like looseness.
Indents render as leading spacers marked ekp-glue, so buffer
roundtrips remain character- and property-exact. Composes with
alignment modes (indent adds to the centering lead) and protrusion.
Tests: 63 ERT green; fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-protrusion enables microtype-style character protrusion at the
right edge: a line ending in fullwidth punctuation (。、」…) hangs
the whitespace half of the glyph past the flush edge — visually
identical to CLREQ line-end punctuation compression — and latin
periods/commas/quotes and soft hyphens protrude by their configured
ratios (ekp-protrusion-ratios).
Mechanics: per-box tail protrusion, folded through trailing space
boxes into a per-gap array; each DP candidate widens its effective
target (lw = width + release), mirrored exactly in the renderer's
per-line effective width and in the C-result reconstruction, so DP,
rendering and both engines stay coherent. Left-edge protrusion is
documented as out of scope (Emacs cannot render before the line
origin). ekp-region reserves the protrusion width in auto layouts
(ekp-region-protrusion-reserve).
Off by default; all-zero arrays make it a no-op on the shared path.
C module 1.4: break-with-arrays 12→14 args (tail-protrudes array,
hyphen-protrude scalar), batch vectors 14 elements.
Tests: 59 ERT green (deterministic hang check, flush-when-off,
protrusion C parity); fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ekp-alignment selects the paragraph alignment. Non-justify modes keep
inter-word glue rigid (arrays and class params zeroed at para build)
while the DP gains a per-line flexibility ekp-ragged-stretch-pixel
(default ≈2 em): badness = 100·(shortfall/R)³, so Knuth-Plass still
globally minimizes raggedness — this is \raggedright with a finite
\rightskip stretch, not first-fit.
Rendering distributes each line's leftover by mode: trailing
(ragged-right), split evenly (center), leading (ragged-left); the
spacers are ekp-glue-marked, so buffer roundtrips stay exact.
C module 1.3: set-penalties gains an extra-stretch scalar (reset when
omitted); max_w widens by it, so the prefix-based flexibility picks it
up automatically. Cache keys include alignment and R.
Tests: 56 ERT green incl. per-mode invariants (fit, rigid interior
glue, flush edges, centered leftover ±1px) and four-mode C parity;
fuzz 300/300.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- text property ekp-no-break turns any span into a rigid typesetting
atom: no line break inside, no hyphenation, literal spacing (space
runs inside the span become preserved space boxes instead of
stretchable glue) — inline code, product names, numbers with units
- character-driven joins: NBSP, NARROW NBSP, FIGURE SPACE keep their
neighbors together (they are boxes whose adjacent gaps are
unbreakable and glue-free — the character supplies its spacing);
WORD JOINER / ZWNBSP attach invisibly like combining marks
- new commands: ekp-no-break-region / ekp-allow-break-region
- overlong atoms fall back to single emergency lines (atomic-run
handling from the break-permission layer); zero C changes needed
Tests: 52 ERT green incl. atomicity across widths, hyphenation
suppression, NBSP/WJ adjacency sweeps; fuzz 300/300 with C parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kinsoku moves from merge-based box attachment to a general break-
permission layer, the foundation for no-break spans, NBSP, verbatim
atoms and punctuation protrusion:
- tokenizer: every CJK char (punctuation included) is its own box;
the hold/attach machinery is gone. Fixes two latent bugs: a second
consecutive closer (字。」) could start a line, and an opener held
across spaces reordered box content vs the original string
- ekp--str-type: cjk-punct splits into cjk-open (Ps/Pi) / cjk-close
- ekp-para gains breaks-allowed (bool-vector) + forbidden-positions
(sparse, for C); glue at unbreakable punct gaps is nws, so
punctuation hugs its content and justification stretch no longer
opens gaps at 「x or x。
- halfwidth kinsoku: boxes consisting purely of .,;:!?)]}’”»›… may
not start a line ((&[{‘“«‹ may not end one) — an ASCII comma after
a CJK char no longer dangles at line start (old behavior violated
CLREQ)
- DP (elisp 1D, elisp loose 2D, C): forbidden gaps are skipped as
candidates while the line keeps extending; the emergency fallback
generalizes from single boxes to atomic runs (no permitted break
inside), recording real gap counts so multi-box emergency lines
render correctly
- C module 1.2: ekp-c-break-with-arrays 11→12 args
(forbidden-positions), batch vectors 12 elements, version gate bumped
Hyphenation improvement: 「Hello / Hello」 previously failed the
word regexp as merged boxes and were never hyphenated; as separate
boxes they hyphenate normally.
Tests: 48 ERT green (split tests migrated to the new box contract;
new unit test for break permissions and a rendered-output kinsoku
sweep across widths); 300-case fuzz 0 failures with C 1.2 parity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
