# Emacs-KP: Knuth-Plass Line Breaking for Emacs [中文文档](./readme_zh.md) | [Developer Guide](./DEVELOPER.md) | [Repository Audit](./Docs/REPOSITORY_AUDIT_20260728.md) Emacs-kp implements the Knuth-Plass optimal line breaking algorithm with full support for CJK (Chinese, Japanese, Korean) and Latin mixed text typesetting, entirely inside Emacs. ## Features - **Optimal line breaking** — the Knuth-Plass dynamic program finds the globally optimal set of breaks for a paragraph, not greedy first-fit. - **CJK support** — every CJK character is a breakable box; kinsoku rules keep punctuation attached (`，。` never start a line, `「《` never end one); dedicated inter-CJK and CJK↔Latin spacing. - **Hyphenation** — Frank Liang's algorithm (the TeX algorithm) with 49 checksum-pinned Hunspell pattern dictionaries bundled. - **Pixel-accurate justification** — every justified line renders at exactly the requested pixel width, using `display (space :width ...)` properties; works with variable-width fonts. - **Text properties preserved** — faces, colors and other properties survive justification; inserted hyphens inherit the face of the word they break. - **Robust on hard input** — unbreakable overlong tokens (URLs, long words at narrow widths) degrade to emergency breaks instead of losing text; every input produces output. - **Optional C module** — a dynamic module runs the DP in C with a thread pool that processes paragraphs in parallel (see benchmarks). ## Requirements - Emacs **29.1+** (uses `string-pixel-width` and `object-intervals`) - Optional, for the C module: a C11 compiler and pthreads ## Installation Clone the repository and add it to your `load-path` (the `dictionaries/` directory must sit next to the `.el` files): ```elisp (add-to-list 'load-path "/path/to/emacs-kp") (require 'ekp) (require 'ekp-region) ; buffer/region commands ``` Byte-compiling is strongly recommended — the Elisp engine is about 10× faster compiled. ## Quick Start ```elisp (require 'ekp) ;; Justify a paragraph to 600 pixels (insert (ekp-pixel-justify "Your paragraph text here..." 600)) ;; Find the best width in a range; returns (justified-text . width) (ekp-pixel-range-justify "Your text" 400 800) ``` Multiline strings are treated as one paragraph per line; blank lines are preserved. ### C module (recommended for long texts) ```bash cd ekp_c && make PROFILE=portable # default; produces ekp.dylib/.so/.dll ``` ```elisp (ekp-c-module-load) ; prints "ekp-c module loaded (version 1.6, N threads)" (ekp-c-module-build) ; prompts for portable/native/debug/sanitize ``` Once loaded (and since `ekp-use-c-module` defaults to `t`), all justification calls automatically use the C engine. The Elisp and C engines produce **identical output**; Elisp is the always-available path when no module is enabled or C returns no result. An enabled module signal is surfaced as a backend contract failure. If the module on disk is older than the Elisp code expects, loading refuses with a message asking you to rebuild. ## Interactive Use (buffer & region) `ekp-region.el` turns the string API into buffer-level commands: ```elisp (require 'ekp-region) ``` - `M-x ekp-justify-region` — justify the region to the window text width (with a numeric prefix argument, to that many pixels). With no active region, it justifies the paragraph at point. - `M-x ekp-justify-buffer` — justify the whole buffer. - `M-x ekp-unjustify-region` / `ekp-unjustify-buffer` — restore the original text **exactly**, including collapsed whitespace runs. Justification is lossless: every synthesized space, soft line break, and soft hyphen carries the original text it replaced, so restoring is a structural transform that also works after you edited the justified text. - `M-x ekp-auto-justify-mode` — keep the whole buffer justified to the window width. Re-flows (debounced by `ekp-auto-justify-resize-delay`) when the window width changes, and after edits re-justifies only the touched paragraphs (`ekp-auto-justify-edit-delay`), so unchanged paragraphs hit the paragraph cache. Turning the mode off restores the buffer exactly. While active, the standard **EKP** menu exposes formatting, protection, and window-fit diagnostic commands; `C-h m` describes the same workflow. The buffer is treated as a live document, not just a canvas: - **Saving** writes the *logical* text — soft line breaks, glue spaces and break hyphens never reach disk; the on-screen buffer stays justified even if writing fails or is interrupted. This guarantee applies to whole-buffer saves; an explicit region-only `write-region` writes the selected physical buffer representation. - **Searching** (isearch) sees the logical text, so CJK phrases and hyphenated words are found across the layout. - **Copying** puts the logical text on the kill ring, so pasted text carries words, not pixel spacing. Existing mode/user substring filters remain active and are restored when the final layout span is removed. - Merely enabling the mode never marks the buffer modified (no stray lock files or auto-saves), and `undo` is not fought by the re-flow timer. `ekp-region-margin-pixel` (default 2) is subtracted from the window width as a rounding safety margin. Large buffers (over `ekp-auto-justify-lazy-threshold` characters, default 20 000) re-flow visible-first: the portion on screen updates synchronously and the rest follows in idle background chunks, with a per-tick time budget (`ekp-auto-justify-tick-budget`) and priority for whatever you scroll to. Mode presets for verbatim protection — one call each: ```elisp (add-hook 'org-mode-hook #'ekp-org-setup) (add-hook 'markdown-mode-hook #'ekp-markdown-setup) ``` `ekp-auto-justify-mode` also applies the matching preset automatically in Org and Markdown buffers when you have not configured your own. ### Protecting code and other verbatim text - Block level: paragraphs carrying the `ekp-verbatim` text property (`M-x ekp-verbatim-region`), wearing a face listed in `ekp-region-skip-faces` (e.g. `org-block`, `markdown-code-face`), or matched by the buffer-local function `ekp-region-skip-predicate` pass through completely untouched. - Inline level: spans carrying `ekp-no-break` (`M-x ekp-no-break-region`) become rigid atoms — never broken, never hyphenated, spacing kept literal — ideal for inline code, product names, or numbers with units. Manual properties are deliberately **current-buffer-session only**: plain-text saving and reopening do not persist them. Use `M-x ekp-allow-break-region` / `ekp-clear-verbatim-region` to remove them. For protection derived from persistent document syntax, use mode faces or the buffer-local `ekp-region-skip-predicate` (the Org/Markdown presets do this automatically). ## Typography - **Alignment** — `ekp-alignment`: `justify` (default), `ragged-right`, `ragged-left`, or `center`. Non-justify modes keep word spacing natural while Knuth-Plass still minimizes raggedness within `ekp-ragged-stretch-pixel` (≈2 em) per line. - **Hanging punctuation** — set `ekp-protrusion` to `t` and line-final punctuation (。、」 as well as periods, commas and break hyphens) hangs past the flush edge by `ekp-protrusion-ratios`. The 0.5 default for fullwidth closers is visually equivalent to CLREQ line-end punctuation compression. `ekp-auto-justify-mode` reserves the protrusion width automatically. - **Paragraph shapes** — `ekp-first-line-indent` (`t` = 2 em) for the CJK paragraph convention, or full TeX-style `ekp-parshape` with per-line `(INDENT . WIDTH)`. First-line indent runs on the fast 1D path and the C engine; only full `ekp-parshape` and `ekp-looseness` fall back to the Elisp-only 2D dynamic program. - **Unbreakables** — NO-BREAK SPACE, NARROW NBSP, FIGURE SPACE and WORD JOINER characters keep their neighbors together out of the box. - Kinsoku covers full- *and* halfwidth punctuation: a line never starts with `。、」!?` or a lone `.,;:!?`, never ends with `「(` etc. Japanese line-start prohibition also covers small kana, the prolonged sound mark and iteration marks (`っ ょ ー 々`), configurable via `ekp-cjk-no-line-start-extra`. Limitations worth knowing: mid-line CLREQ punctuation *compression* (e.g. 「字。下」 squeezed inside a line) cannot be rendered — Emacs cannot shrink a glyph's advance — which is why line-edge compression is delivered via protrusion instead; left-edge protrusion is likewise not renderable (text cannot start before the line origin). ## Configuration ### Hyphenation language ```elisp (setq ekp-latin-lang "de_DE") ; default "en_US" ``` Short codes like `"de"` resolve to the first matching dictionary. Each supported dictionary's own `LEFTHYPHENMIN` / `RIGHTHYPHENMIN` are honored (English keeps ≥2 letters before and ≥3 after a break); pass explicit margins to `ekp-hyphen-create` to override. EKP supports ordinary Liang patterns. It fails closed with `ekp-hyphen-unsupported-pattern` for `eo`, `ca`, `hu_HU`, and `sq_AL` because those files contain slash/replacement rules that conditionally rewrite glyphs at a chosen break. Treating them as ordinary positions would produce linguistically wrong text and incorrect DP widths. The exact inventory, SHA-256 checksums, pinned source paths, and license evidence live in `dictionaries/MANIFEST.tsv` and `dictionaries/LICENSES.md`. ### Spacing parameters Three glue classes control spacing (all values in pixels): | Group | Between | |:--------|:---------------------------| | `lws-*` | two Latin words | | `mws-*` | a Latin word and a CJK char| | `cws-*` | two CJK characters | Each class has an ideal width, a maximum stretch and a maximum shrink: ```elisp (ekp-param-set lws-ideal lws-stretch lws-shrink mws-ideal mws-stretch mws-shrink cws-ideal cws-stretch cws-shrink) ;; e.g. (ekp-param-set 7 3 2 5 2 1 0 2 0) ``` - If you never call `ekp-param-set`, defaults are derived automatically from the font of each string. - Explicit parameters **persist** until you call `ekp-param-reset`, which returns to automatic per-string defaults. ### Algorithm parameters | Variable | Default | Meaning | |:--------------------------------|:--------|:--------| | `ekp-line-penalty` | 10 | Base cost per line; higher prefers fewer lines | | `ekp-hyphen-penalty` | 50 | Cost of a hyphenated break (added as penalty²) | | `ekp-adjacent-fitness-penalty` | 100 | Cost when adjacent lines differ in tightness by >1 class | | `ekp-consecutive-hyphen-penalty`| 100 | Multiplier for runs of hyphenated lines (× count²) | | `ekp-last-line-min-ratio` | 0.5 | Minimum fill ratio for the last line | | `ekp-last-line-short-penalty` | 50 | Cost multiplier for a too-short last line | | `ekp-looseness` | 0 | Target line count offset: +1 = one line more than optimal, −1 = one fewer | Both engines implement these parameters: before an actual C computation, the Elisp side synchronizes their current values. The DP cache signature includes every parameter in the table, so changes take effect on the next call without manually clearing caches. `ekp-looseness` is part of the cache key and uses a dedicated Elisp path (the C module is bypassed automatically while it is non-zero). ### Caching Tokenization, measurement, and DP results are cached per paragraph; box widths are additionally cached session-wide, so a glyph shared across paragraphs is measured only once. Both explicit spacing values and the automatic `ekp-default-cws-stretch-pixel` input participate in paragraph cache identity. The same complete structural key also governs the same-string fast path, so adding or removing layout properties such as `ekp-no-break` takes effect immediately on an already cached string object. - `ekp-para-cache-limit` (default 256): max cached paragraphs; the cache is flushed when the limit is reached. - `M-x ekp-clear-caches` clears everything (use after changing fonts or themes that affect glyph widths). ## Performance Measured on the bundled sample texts (`tests/ekp-bench.el`), batch Emacs 30.2, Apple Silicon; see DEVELOPER.md for methodology: | Case (text-zh.txt ≈ 3.6 KB) | Elisp (byte-compiled) | C module | |:-----------------------------|----------------------:|---------:| | justify, width 200px | 150 ms | 41 ms | | optimal-width search 340–380 | 529 ms | 106 ms | | DP only, width 400px | 30 ms | 2.5 ms | **Byte-compile the package** — the Elisp engine is ~10× faster compiled. Both engines produce identical output; the C module pays off most for optimal-width search and long multi-paragraph texts. (Absolute numbers vary with machine and power state; the ratios are the point.) ## Known Limitations - Measurement follows the current buffer's face remappings (`text-scale-mode`, themes, `ekp-org-setup`-style tweaks) and reserves the truncation-indicator column in windows without fringes, so justified lines fit the real display. If lines ever look truncated or short in an exotic setup, run `M-x ekp-diagnose` in that buffer — it reports whether measurement matches rendering. The full fit matrix is a developer tool in `tests/ekp-gui-verify.el`; load that file before invoking `M-x ekp-gui-verify`. - One font is assumed per Latin/CJK script per paragraph when computing spacing defaults; mixed-font paragraphs work but spacing defaults come from the first font found. - `ekp-pixel-range-justify` minimizes average demerits with a ternary search plus a local scan; cost is not perfectly unimodal in width, so the result is a very good, but not guaranteed global, optimum. - In batch/tty Emacs, pixel widths degrade to character columns (the full pipeline still works; useful for testing). ## Interactive Demo ```bash emacs -Q -L /path/to/emacs-kp -l tests/ekp-showcase.el -f ekp-showcase ``` One buffer, live keys: `-`/`+` change the pixel width (per-reflow time in the header line), `d` runs an animated width sweep with an fps report, `a` cycles alignment, `p` toggles hanging punctuation, `i` first-line indent, `s` a wedge parshape, `c` compares the C engine with pure Elisp, `w` follows the window width via `ekp-auto-justify-mode`. The sample text includes a protected code block, an inline no-break atom and NBSP-joined numbers. ## Testing ```bash tests/run-tests.sh /path/to/emacs # batch-safe ERT suite # Full interactive GUI fit matrix emacs -Q -L /path/to/emacs-kp -L /path/to/emacs-kp/tests \ -l /path/to/emacs-kp/tests/ekp-gui-verify.el \ -f ekp-gui-verify-matrix ``` The matrix prints every row and exits with status 1 if any fit check fails, so the same command can gate local release automation. The verifier is a developer tool under `tests/`; it is not loaded by `(require 'ekp-region)`. ## Credits - **Core algorithm**: ["Breaking Paragraphs into Lines"](https://gwern.net/doc/design/typography/tex/1981-knuth.pdf) by Donald E. Knuth and Michael F. Plass (1981) - **Hyphenation**: Frank Liang's algorithm, adapted from [Pyphen](https://github.com/Kozea/Pyphen) - **Dictionaries**: hyphenation patterns from the [LibreOffice dictionaries](https://github.com/LibreOffice/dictionaries); see `dictionaries/MANIFEST.tsv`, `dictionaries/LICENSES.md`, and the bundled per-dictionary notices for exact source, checksum, and license evidence.