ci: add lint, sanitizer, macOS and snapshot jobs; docs and changelog

- CI: package-lint + checkdoc job, ASan/UBSan fuzz job (make DEBUG=1),
  macOS dylib build + full suite, Emacs snapshot (advisory)
- readme: installation section, new commands and editor-integration
  behavior, JIS kinsoku option, HYPHENMIN note, refreshed perf table
- add CHANGELOG.md and CONTRIBUTING.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.github/workflows/ci.yml

@@ -12,11 +12,13 @@ jobs:
   test:
     name: elisp-only (Emacs ${{ matrix.emacs-version }})
     runs-on: ubuntu-latest
+    continue-on-error: ${{ matrix.emacs-version == 'snapshot' }}
     strategy:
       fail-fast: false
       matrix:
-        # 29.1 is the declared floor (Package-Requires); 30.1 is current stable
+        # 29.1 is the declared floor (Package-Requires); 30.1 is current
-        emacs-version: ['29.1', '30.1']
+        # stable; snapshot is advisory (allowed to fail)
+        emacs-version: ['29.1', '30.1', 'snapshot']
     steps:
       - uses: actions/checkout@v4
       - uses: purcell/setup-emacs@master
@@ -30,6 +32,40 @@ jobs:
       - name: Run ERT suite (C-module tests auto-skip)
         run: tests/run-tests.sh emacs
 
+  lint:
+    name: package-lint + checkdoc
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: purcell/setup-emacs@master
+        with:
+          version: '30.1'
+      - name: Install package-lint
+        run: |
+          emacs -Q --batch \
+            --eval "(require 'package)" \
+            --eval "(add-to-list 'package-archives '(\"melpa\" . \"https://melpa.org/packages/\") t)" \
+            --eval "(package-initialize)" \
+            --eval "(package-refresh-contents)" \
+            --eval "(package-install 'package-lint)"
+      - name: package-lint
+        run: |
+          emacs -Q --batch \
+            --eval "(require 'package)" \
+            --eval "(package-initialize)" \
+            --eval "(require 'package-lint)" \
+            --eval "(setq package-lint-main-file \"ekp.el\")" \
+            -f package-lint-batch-and-exit \
+            ekp.el ekp-utils.el ekp-hyphen.el ekp-region.el
+      - name: checkdoc
+        run: |
+          emacs -Q --batch \
+            --eval "(setq sentence-end-double-space t)" \
+            --eval "(dolist (f '(\"ekp.el\" \"ekp-utils.el\" \"ekp-hyphen.el\" \"ekp-region.el\"))
+                      (checkdoc-file f))" \
+            2>&1 | tee checkdoc.log
+          test ! -s checkdoc.log
+
   test-c-module:
     name: full suite + C parity (apt Emacs)
     runs-on: ubuntu-latest
@@ -51,3 +87,43 @@ jobs:
         run: tests/run-tests.sh emacs
       - name: Property fuzz (300 cases, C vs elisp parity)
         run: emacs -Q --batch -L . -l tests/ekp-fuzz.el
+
+  test-c-sanitizers:
+    name: C module under ASan/UBSan
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Emacs
+        run: sudo apt-get update && sudo apt-get install -y emacs-nox gcc
+      - name: Ensure emacs-module.h is findable
+        run: |
+          if [ ! -e /usr/include/emacs-module.h ]; then
+            HDR=$(dpkg -L emacs-common emacs-bin-common 2>/dev/null | grep -m1 '/emacs-module\.h$' || true)
+            [ -n "$HDR" ] || HDR=$(sudo find /usr -name emacs-module.h 2>/dev/null | head -1)
+            test -n "$HDR" || { echo 'emacs-module.h not found'; exit 1; }
+            sudo install -m 644 "$HDR" /usr/local/include/
+          fi
+      - name: Build C module with sanitizers
+        run: make -C ekp_c DEBUG=1
+      - name: Property fuzz under ASan/UBSan
+        # Emacs itself is not ASan-instrumented: preload the runtime and
+        # skip leak checking (the Emacs process "leaks" by design).
+        run: |
+          LIBASAN=$(gcc -print-file-name=libasan.so)
+          LD_PRELOAD="$LIBASAN" ASAN_OPTIONS=detect_leaks=0 \
+            emacs -Q --batch -L . -l tests/ekp-fuzz.el
+
+  test-macos:
+    name: macOS (dylib build + full suite)
+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: purcell/setup-emacs@master
+        with:
+          version: '30.1'
+      - name: Build C module
+        run: make -C ekp_c
+      - name: Run ERT suite (with C module)
+        run: tests/run-tests.sh emacs
+      - name: Property fuzz (300 cases, C vs elisp parity)
+        run: emacs -Q --batch -L . -l tests/ekp-fuzz.el

CHANGELOG.md

@@ -0,0 +1,102 @@
+# Changelog
+
+All notable changes to emacs-kp are documented here.  The format
+follows [Keep a Changelog](https://keepachangelog.com/), and the
+project aims to follow [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+### Editor integration
+
+- Saving an auto-justified buffer now writes the **logical** text to
+  disk; soft line breaks, glue spaces and break hyphens are layout,
+  not content, and no longer reach the file.
+- `isearch` searches the logical text, so CJK phrases and hyphenated
+  words are found across the justified layout.
+- Copying/killing puts the logical text on the kill ring (via a
+  buffer-local `filter-buffer-substring-function`).
+- Justification no longer flips `buffer-modified-p` on an unmodified
+  buffer — no stray lock files or auto-saves — and re-flow no longer
+  fights `undo`.
+- Text typed adjacent to synthesized glue/hyphens no longer inherits
+  the marker properties (`rear-nonsticky`), so it is never mistaken
+  for layout and deleted on the next re-flow.
+- Switching major mode or turning the minor mode off restores the
+  logical text cleanly, widening first so narrowing leaves no
+  justified orphans.
+- Paragraphs containing `field`/`read-only` text (comint prompts) are
+  left verbatim.
+
+### New commands and options
+
+- `ekp-justify-buffer` / `ekp-unjustify-buffer`.
+- `ekp-justify-region` / `ekp-unjustify-region` act on the paragraph
+  at point when no region is active (DWIM).
+- `ekp-org-setup` / `ekp-markdown-setup` wire the verbatim-protection
+  presets in one call; `ekp-auto-justify-mode` applies the matching
+  one automatically in Org/Markdown buffers.
+- `fill-paragraph` is remapped to `ekp-refill-paragraph` while the
+  mode is on.
+- User options are now `defcustom`s under the `ekp` / `ekp-region`
+  groups.
+- `ekp-cjk-no-line-start-extra`: Japanese line-start prohibition for
+  small kana, the prolonged sound mark and iteration marks (JIS X
+  4051).
+- `ekp-auto-justify-tick-budget`: time budget per background re-flow
+  tick.
+
+### Typography
+
+- First-line indent (`ekp-first-line-indent`) now runs on the fast 1D
+  dynamic program and the C engine, instead of the heavy 2D path.
+- Dictionary `LEFTHYPHENMIN` / `RIGHTHYPHENMIN` are honored (English
+  now keeps ≥2 before and ≥3 after a break, per its dictionary).
+
+### Performance
+
+- Session-global box-width cache eliminates cross-paragraph duplicate
+  measurement (≈40% of `string-pixel-width` calls on the Chinese
+  sample); byte-compiled justify of that sample dropped ~40%.
+- Cache keys ignore volatile `fontified` bookkeeping, restoring the
+  paragraph-cache hit rate in font-locked buffers.
+- Large-buffer re-flow: huge edits chunk in the background, scrolled
+  regions get priority, background ticks honor a time budget.
+
+### Fixed
+
+- The DP cache no longer returns a stale layout after `ekp-looseness`
+  changes (results are keyed by looseness).
+- Changing a style variable (`ekp-alignment`, …) invalidates the
+  same-string fast path.
+- Missing `dictionaries/` directory only disables hyphenation instead
+  of breaking `(require 'ekp)`.
+
+### C module (1.4 → 1.5)
+
+- **Breaking:** the unused experimental C tokenization path
+  (`ekp-c-break-lines`, `ekp-c-hyphenate`, `ekp-c-load-hyphenator`,
+  `ekp-c-set-spacing`, and the `ekp_paragraph.c` / `ekp_hyphen.c`
+  sources) is removed.  It carried a heap overflow reachable from
+  Lisp.  Rebuild with `make -C ekp_c clean all`.
+- `ekp-c-break-with-arrays` gains a 15th argument, `FIRST-LINE-WIDTH`,
+  supporting first-line indent in C.
+- Thread pool sized to the machine's cores, created lazily; a full
+  queue blocks instead of dropping tasks.
+- Allocation failures and bad arguments fail the call cleanly (Elisp
+  fallback) instead of silently producing a different layout;
+  extracted integers are clamped to int32.
+
+### Packaging
+
+- Added `COPYING` (GPL-3.0-or-later) and license headers to all
+  sources; real author/maintainer metadata.
+- Autoload cookies on all interactive commands.
+- Removed internal working files and an unused demo GIF from the repo.
+
+## Historical
+
+Earlier work (the core KP overhaul and the P1/P2 typography wave —
+optimal line breaking, CJK kinsoku, hyphenation, pixel-exact
+justification, alignment modes, hanging punctuation, parshape,
+no-break atoms, verbatim protection, the C module) predates this
+changelog; see the git history.

CONTRIBUTING.md

@@ -0,0 +1,59 @@
+# Contributing to emacs-kp
+
+Thanks for your interest.  This document covers the essentials for
+making a change that will pass review and CI.
+
+## Development setup
+
+```bash
+EMACS=emacs   # or /path/to/Emacs
+
+# Byte-compile with warnings as errors (CI does this)
+$EMACS -Q --batch -L . \
+  --eval '(setq byte-compile-error-on-warn t)' \
+  -f batch-byte-compile ekp.el ekp-utils.el ekp-hyphen.el ekp-region.el
+
+# Run the ERT suite (C-module tests auto-skip if not built)
+tests/run-tests.sh $EMACS
+
+# Build the C module (required for the parity tests and fuzz)
+make -C ekp_c              # add DEBUG=1 for ASan/UBSan
+
+# Property fuzz: 300 random cases, asserts C and Elisp agree byte-for-byte
+$EMACS -Q --batch -L . -l tests/ekp-fuzz.el
+```
+
+## Ground rules
+
+- **The two engines must produce byte-identical output.** Any change
+  to the demerits or line-metric formulas must touch both
+  `ekp--dp-run-1d` (Elisp) and `dp_process_position` (`ekp_c/ekp_kp.c`),
+  and the fuzz suite must stay at 300/300.
+- **The layout is lossless.** The renderer's marker properties
+  (`ekp-glue`, `ekp-soft-break`, `ekp-soft-hyphen`, `ekp-hidden`) must
+  round-trip exactly through `ekp-unjustify-region`.
+- Any C-module API change bumps `EKP_VERSION_MINOR` and the matching
+  `ekp-c-module-required-version`, and rebuilds the module.
+- New behavior needs an ERT test.  Buffer-level behavior (save,
+  isearch, undo, kill/yank, mode interactions) goes in
+  `tests/ekp-region-tests.el`.
+
+## Style
+
+- `lexical-binding: t` everywhere; keep byte-compilation warning-free.
+- `checkdoc` clean (CI enforces it): imperative docstring first lines,
+  arguments mentioned in uppercase, two spaces after a sentence.
+- `package-lint` clean: the `ekp-` / `ekp-region-` namespaces, proper
+  autoload cookies on interactive entry points.
+- Match the surrounding code; keep comments about *why*, not *what*.
+
+## Commits
+
+Conventional Commits (`feat:`, `fix:`, `perf:`, `refactor!:`,
+`docs:`, `test:`, `chore:`).  Explain the reasoning in the body, not
+just the change.
+
+## License
+
+By contributing you agree that your contributions are licensed under
+GPL-3.0-or-later, matching the project (see `COPYING`).

readme.md

@@ -32,11 +32,24 @@ typesetting, entirely inside Emacs.
 - Emacs **29.1+** (uses `string-pixel-width` and `object-intervals`)
 - Optional, for the C module: a C11 compiler and pthreads
 
-## Quick Start
+## Installation
+
+Clone the repository and add it to your `load-path` (the
+`dictionaries/` directory must sit next to the `.el` files):
 
 ```elisp
 (add-to-list 'load-path "/path/to/emacs-kp")
 (require 'ekp)
+(require 'ekp-region)   ; buffer/region commands
+```
+
+Byte-compiling is strongly recommended — the Elisp engine is about
+10× faster compiled.
+
+## Quick Start
+
+```elisp
+(require 'ekp)
 
 ;; Justify a paragraph to 600 pixels
 (insert (ekp-pixel-justify "Your paragraph text here..." 600))
@@ -73,12 +86,15 @@ loading refuses with a message asking you to rebuild.
 ```
 
 - `M-x ekp-justify-region` — justify the region to the window text
-  width (with a numeric prefix argument, to that many pixels).
+  width (with a numeric prefix argument, to that many pixels).  With
-- `M-x ekp-unjustify-region` — restore the original text **exactly**,
+  no active region, it justifies the paragraph at point.
-  including collapsed whitespace runs.  Justification is lossless: every
+- `M-x ekp-justify-buffer` — justify the whole buffer.
-  synthesized space, soft line break, and soft hyphen carries the
+- `M-x ekp-unjustify-region` / `ekp-unjustify-buffer` — restore the
-  original text it replaced, so restoring is a structural transform that
+  original text **exactly**, including collapsed whitespace runs.
-  also works after you edited the justified text.
+  Justification is lossless: every synthesized space, soft line break,
+  and soft hyphen carries the original text it replaced, so restoring
+  is a structural transform that also works after you edited the
+  justified text.
 - `M-x ekp-auto-justify-mode` — keep the whole buffer justified to the
   window width.  Re-flows (debounced by
   `ekp-auto-justify-resize-delay`) when the window width changes, and
@@ -86,24 +102,38 @@ loading refuses with a message asking you to rebuild.
   (`ekp-auto-justify-edit-delay`), so unchanged paragraphs hit the
   paragraph cache.  Turning the mode off restores the buffer exactly.
 
+The buffer is treated as a live document, not just a canvas:
+
+- **Saving** writes the *logical* text — soft line breaks, glue
+  spaces and break hyphens never reach disk; the on-screen buffer
+  stays justified.
+- **Searching** (isearch) sees the logical text, so CJK phrases and
+  hyphenated words are found across the layout.
+- **Copying** puts the logical text on the kill ring, so pasted text
+  carries words, not pixel spacing.
+- Merely enabling the mode never marks the buffer modified (no stray
+  lock files or auto-saves), and `undo` is not fought by the re-flow
+  timer.
+
 `ekp-region-margin-pixel` (default 2) is subtracted from the window
 width as a rounding safety margin.
 
 Large buffers (over `ekp-auto-justify-lazy-threshold` characters,
 default 20 000) re-flow visible-first: the portion on screen updates
-synchronously (~15 ms) and the rest follows in idle background chunks.
+synchronously and the rest follows in idle background chunks, with a
+per-tick time budget (`ekp-auto-justify-tick-budget`) and priority
+for whatever you scroll to.
 
-Mode presets for verbatim protection:
+Mode presets for verbatim protection — one call each:
 
 ```elisp
-(add-hook 'org-mode-hook
+(add-hook 'org-mode-hook      #'ekp-org-setup)
-          (lambda ()
+(add-hook 'markdown-mode-hook #'ekp-markdown-setup)
-            (setq-local ekp-region-skip-faces ekp-region-org-skip-faces)))
-(add-hook 'markdown-mode-hook
-          (lambda ()
-            (setq-local ekp-region-skip-faces ekp-region-markdown-skip-faces)))
 ```
 
+`ekp-auto-justify-mode` also applies the matching preset automatically
+in Org and Markdown buffers when you have not configured your own.
+
 ### Protecting code and other verbatim text
 
 - Block level: paragraphs carrying the `ekp-verbatim` text property
@@ -130,12 +160,16 @@ Mode presets for verbatim protection:
   the protrusion width automatically.
 - **Paragraph shapes** — `ekp-first-line-indent` (`t` = 2 em) for the
   CJK paragraph convention, or full TeX-style `ekp-parshape` with
-  per-line `(INDENT . WIDTH)`.  Both are Elisp-only paths (the C
+  per-line `(INDENT . WIDTH)`.  First-line indent runs on the fast 1D
-  module is bypassed, as with `ekp-looseness`).
+  path and the C engine; only full `ekp-parshape` and `ekp-looseness`
+  fall back to the Elisp-only 2D dynamic program.
 - **Unbreakables** — NO-BREAK SPACE, NARROW NBSP, FIGURE SPACE and
   WORD JOINER characters keep their neighbors together out of the box.
 - Kinsoku covers full- *and* halfwidth punctuation: a line never
   starts with `。、」!?` or a lone `.,;:!?`, never ends with `「(` etc.
+  Japanese line-start prohibition also covers small kana, the
+  prolonged sound mark and iteration marks (`っ ょ ー 々`), configurable
+  via `ekp-cjk-no-line-start-extra`.
 
 Limitations worth knowing: mid-line CLREQ punctuation *compression*
 (e.g. 「字。下」 squeezed inside a line) cannot be rendered — Emacs
@@ -152,7 +186,10 @@ not renderable (text cannot start before the line origin).
 ```
 
 Any `dictionaries/hyph_<lang>.dic` works; short codes like `"de"`
-resolve to the first matching dictionary.
+resolve to the first matching dictionary.  Each dictionary's own
+`LEFTHYPHENMIN` / `RIGHTHYPHENMIN` are honored (English keeps ≥2
+letters before and ≥3 after a break); pass explicit margins to
+`ekp-hyphen-create` to override.
 
 ### Spacing parameters
 
@@ -197,7 +234,9 @@ is non-zero).
 
 ### Caching
 
-Tokenization, measurement, and DP results are cached per paragraph.
+Tokenization, measurement, and DP results are cached per paragraph;
+box widths are additionally cached session-wide, so a glyph shared
+across paragraphs is measured only once.
 
 - `ekp-para-cache-limit` (default 256): max cached paragraphs; the
   cache is flushed when the limit is reached.
@@ -211,13 +250,15 @@ Emacs 30.2, Apple Silicon; see DEVELOPER.md for methodology:
 
 | Case (text-zh.txt ≈ 3.6 KB)  | Elisp (byte-compiled) | C module |
 |:-----------------------------|----------------------:|---------:|
-| justify, width 200px         |                 96 ms |    57 ms |
+| justify, width 200px         |                150 ms |    41 ms |
-| optimal-width search 340–380 |                294 ms |    75 ms |
+| optimal-width search 340–380 |                529 ms |   106 ms |
-| DP only, width 400px         |                 15 ms |   1.3 ms |
+| DP only, width 400px         |                 30 ms |   2.5 ms |
 
 **Byte-compile the package** — the Elisp engine is ~10× faster
 compiled.  Both engines produce identical output; the C module pays
 off most for optimal-width search and long multi-paragraph texts.
+(Absolute numbers vary with machine and power state; the ratios are
+the point.)
 
 ## Known Limitations