geekinney/ekp
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

refactor!: per-gap break permissions — punctuation as first-class boxes

Browse Source 
Kinsoku moves from merge-based box attachment to a general break-
permission layer, the foundation for no-break spans, NBSP, verbatim
atoms and punctuation protrusion:

- tokenizer: every CJK char (punctuation included) is its own box;
  the hold/attach machinery is gone.  Fixes two latent bugs: a second
  consecutive closer (字。」) could start a line, and an opener held
  across spaces reordered box content vs the original string
- ekp--str-type: cjk-punct splits into cjk-open (Ps/Pi) / cjk-close
- ekp-para gains breaks-allowed (bool-vector) + forbidden-positions
  (sparse, for C); glue at unbreakable punct gaps is nws, so
  punctuation hugs its content and justification stretch no longer
  opens gaps at 「x or x。
- halfwidth kinsoku: boxes consisting purely of .,;:!?)]}’”»›… may
  not start a line ((&[{‘“«‹ may not end one) — an ASCII comma after
  a CJK char no longer dangles at line start (old behavior violated
  CLREQ)
- DP (elisp 1D, elisp loose 2D, C): forbidden gaps are skipped as
  candidates while the line keeps extending; the emergency fallback
  generalizes from single boxes to atomic runs (no permitted break
  inside), recording real gap counts so multi-box emergency lines
  render correctly
- C module 1.2: ekp-c-break-with-arrays 11→12 args
  (forbidden-positions), batch vectors 12 elements, version gate bumped

Hyphenation improvement: 「Hello / Hello」 previously failed the
word regexp as merged boxes and were never hyphenated; as separate
boxes they hyphenate normally.

Tests: 48 ERT green (split tests migrated to the new box contract;
new unit test for break permissions and a rendered-output kinsoku
sweep across widths); 300-case fuzz 0 failures with C 1.2 parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Kinneyzhang 2026-07-26 21:05:50 +08:00
parent 33981a647e
commit 64eb2f38d5
7 changed files with 347 additions and 95 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
.phrase/phases/phase-p1p2-20260726/NOTES.md Normal file
View File

@ -0,0 +1,71 @@
# P1+P2 功能阶段笔记(2026-07-26 起)
> 用户指令:高质量完成 P1(标点挤压、ragged 模式、no-break API)+ P2(悬挂、
> parshape/首行缩进、连续标点);代码块等特殊文本需正确处理;**通用机制优先,
> 万不得已才做场景特化**。
## 总体架构决策
1. **地基 = 逐间隙断行许可(breaks-allowed)+ 标点独立成盒**。
禁则从"吞噬式附着"迁移为 DP 层的断点禁止;所有后续特性(NBSP、
no-break 区间、行内 verbatim 原子、标点类别)都是这套机制的实例。
2. **Emacs 显示引擎约束(已确证)**:无法缩减字形 advance(无负宽
display)→ CLREQ 行中标点挤压不可渲染;行首/行尾挤压视觉上等价于
"悬挂"(protrusion)→ P1-1 + P2-4 + P2-6 统一为**边缘突出机制**,
按字符类配比率(可扩展到拉丁连字符突出 = microtype)。
3. ragged-right/left/center:DP 侧 = 刚性 glue(stretch/shrink 数组置零)
+ 每行额外伸展量 R(badness 以 R 为 flexibility);渲染侧分派剩余量
(右/左/对半)。C 只需 +1 标量。
4. parshape/首行缩进:每行宽依赖行号 → 复用 looseness 的 2D DP,
elisp-only(C 自动旁路,同 looseness 先例)。
5. verbatim:段落级豁免(region 层谓词/属性)+ 行内原子
(ekp-no-break 属性 → 禁断点 + 刚性 glue + 禁断词)。
## 关键实现事实(读码结论)
- tokenizer 附着逻辑在 ekp-utils.el `ekp--handle-cjk-char/latin-char`
(开放标点 hold-and-prepend;闭合标点 append-to-prev)。
**已知老 bug**:连续闭合标点(字。」)第二个独立成盒且断点未禁止 →
」可出现行首;开放标点跨空格 hold 还会导致盒序与原文顺序不一致。
迁移后两者都根治。
- `ekp--str-type` 返回 space/latin/cjk/cjk-punct → 拆成 cjk-open
(opening-punct-p:general-category Ps/Pi)/ cjk-close(fw-punct-p
且非 open)。“” 特例保持 'cjk。
- `ekp--glue-type` 新矩阵(保持旧拓扑等价):space→nws;
before=open→nws(原盒内);after=close→nws(原盒内;close-close
从 cws 改为 nws,属有意修正);latin-latin→lws;cjk-cjk→cws;
cjk/latin 混→mws;其余含标点→cws。
- breaks-allowed 规则:`allowed[k] = !(tail(box[k-1])=open || head(box[k])=close)`,
k∈[1,n-1];k=n(段末)恒可。存 bool-vector(elisp DP 用)+
forbidden-positions int 向量(C 打包用,稀疏,仿 hyphen-positions)。
- DP 改动(elisp `ekp--dp-run-1d` + C `dp_process_position` 镜像):
候选 k 需 allowed;不 allowed 时**不 throw**继续延伸;
紧急兜底从 single-box 推广为 atomic-run(i 到 k 间无允许断点);
多盒紧急行需记录 gaps 计数(渲染 normal 路径 clamp ≥0 自然溢出)。
- C 桥:`ekp-c-break-with-arrays` 11→12 参(forbidden-positions),
batch 向量同步;`ekp-c-set-penalties` 后续 ragged 加 extra-stretch
标量;protrusion 再加两数组(head/tail protrude px)。每次 API 变
动 bump EKP_VERSION_MINOR + `ekp-c-module-required-version`。
- 隐性收益:「Hello / Hello」 之前整盒无法匹配断词正则(左右标点类
不含 CJK 引号)→ 拆盒后可正常断词。
- 测试影响:tests/ekp-tests.el 里 split-* 结构测试要改为新盒契约;
新增行为级禁则测试(任意宽度:行首无 close、行尾无 open、
字。」不拆)。fuzz 断言与引擎无关,应保持 0 失败。
## 阶段与提交计划
- [ ] A 地基:标点成盒 + breaks-allowed + DP/C(1.2)+ 测试迁移
- [ ] B no-break API:ekp-no-break 属性、NBSP/WJ/2060/202F、刚性 glue
- [ ] C 对齐模式:ekp-alignment(justify|ragged-right|ragged-left|center)
+ C extra-stretch(1.3)
- [ ] D 突出/悬挂:ekp-protrusion(类→左右比率)+ C 两数组(1.4)
+ region 层宽度补偿
- [ ] E parshape:ekp-parshape + ekp-first-line-indent(2D,elisp-only)
- [ ] F verbatim:段落豁免(region 谓词/属性)+ 行内原子(含禁断词)
- [ ] G 文档(readme×2 DEVELOPER×2)+ GUI 目检 + 记忆更新
## 验证清单(每阶段)
byte-compile 零警告(error-on-warn)→ 47+ ERT → C 重建 + parity →
fuzz 300 → 提交。改 ekp_c/ 后必须 make clean && make。
Emacs: /Applications/Emacs.app/Contents/MacOS/Emacs

61
ekp-utils.el
View File

@ -198,48 +198,28 @@ Return (new-state new-latin-word new-cjk-char new-boxes)."
(if (= state 1)
;; Already in latin mode: accumulate
(list 1 (concat latin-word str) nil boxes)
;; Was in CJK mode: flush CJK char, switch to latin
;; If held cjk-char is opening punct, prepend it to the latin word
(if (and cjk-char (ekp-cjk-opening-punct-p cjk-char))
(list 1 (concat cjk-char str) nil boxes)
(list 1 str nil (ekp--flush-cjk-char cjk-char boxes)))))
;; Was in CJK mode: flush held CJK char, switch to latin
(list 1 str nil (ekp--flush-cjk-char cjk-char boxes))))
(defun ekp--handle-cjk-char (str state latin-word cjk-char boxes)
"Handle a CJK (width=2) character.
Return (new-state new-latin-word new-cjk-char new-boxes)."
Return (new-state new-latin-word new-cjk-char new-boxes).
Every CJK character — punctuation included — becomes its own box.
Kinsoku is enforced by the DP through per-gap break permissions
(`ekp-para-breaks-allowed'), not by merging boxes."
(if (= state 1)
;; Was in latin mode: flush latin word, push CJK directly
(let ((new-boxes (ekp--flush-latin-word latin-word boxes)))
(if (ekp-cjk-opening-punct-p str)
;; Opening punct: hold as cjk-char (will attach to next char)
(list 2 nil str new-boxes)
(list 2 nil nil (cons str new-boxes))))
;; Already in CJK mode
(cond
((ekp-cjk-opening-punct-p str)
;; Opening punct: cannot end a line (kinsoku rule).
;; If previous held char is also opening punct, concatenate them.
;; Otherwise flush previous and hold this opening punct.
(if (and cjk-char (ekp-cjk-opening-punct-p cjk-char))
(list 2 nil (concat cjk-char str) boxes)
(list 2 nil str (ekp--flush-cjk-char cjk-char boxes))))
((ekp-cjk-fw-punct-p str)
;; Closing/other punct: attaches to previous CJK char
(list 2 nil nil (cons (concat cjk-char str) boxes)))
(t
;; Regular CJK char: prepend any held opening punct
(if (and cjk-char (ekp-cjk-opening-punct-p cjk-char))
;; Previous was opening punct: combine with current char and hold.
;; Now last char is regular, so this won't be detected as opening.
(list 2 nil (concat cjk-char str) boxes)
;; Normal case: flush previous, hold current
(list 2 nil str (ekp--flush-cjk-char cjk-char boxes)))))))
;; Was in latin mode: flush latin word, hold current CJK char
(list 2 nil str (ekp--flush-latin-word latin-word boxes))
;; Already in CJK mode: flush held char, hold current
(list 2 nil str (ekp--flush-cjk-char cjk-char boxes))))
(defun ekp-split-to-boxes (string)
"Split STRING into typographic boxes.
Latin words become single boxes; CJK chars are individual boxes.
Whitespace runs are preserved as separate boxes; CJK punctuation
attaches to its neighboring char per kinsoku rules."
Latin words become single boxes; CJK chars — punctuation included —
are individual boxes. Whitespace runs are preserved as separate
boxes. Kinsoku is enforced later via per-gap break permissions, not
by merging boxes."
(if (string-blank-p string)
(vector string)
(with-temp-buffer
@ -268,12 +248,9 @@ attaches to its neighboring char per kinsoku rules."
(t (setq latin-word str state 1))))
;; Whitespace or other zero-width: flush content, accumulate spaces
((or (string-blank-p str) (= 0 width))
;; Don't flush opening punct - keep it held for attachment to next char
(if (and cjk-char (ekp-cjk-opening-punct-p cjk-char))
nil ; keep cjk-char as-is
(setq boxes (ekp--flush-cjk-char cjk-char boxes))
(when cjk-char (setq prev-state 2))
(setq cjk-char nil))
(setq boxes (ekp--flush-cjk-char cjk-char boxes))
(when cjk-char (setq prev-state 2))
(setq cjk-char nil)
(setq boxes (ekp--flush-latin-word latin-word boxes))
(when latin-word (setq prev-state 1))
(setq latin-word nil)
@ -365,7 +342,7 @@ after CALLBACK returns."
(defalias 'ekp-c-module-reload #'ekp--module-reload
"Load MODULE from a temp copy to allow rebuilding.")
(defconst ekp-c-module-required-version "1.1"
(defconst ekp-c-module-required-version "1.2"
"Minimum C module version compatible with this Elisp code.")
(defun ekp-c-module-load ()

142
ekp.el
View File

@ -116,6 +116,11 @@ when non-zero the C module is bypassed automatically.")
;; indentation is preserved); trail-spaces[k] = total width of
;; consecutive space boxes ending at box k-1.
lead-spaces trail-spaces
;; Per-gap break permission: breaks-allowed[k] non-nil iff a line may
;; end after box k-1 (kinsoku, no-break spans). n+1 bool-vector;
;; index n (paragraph end) is always allowed. forbidden-positions is
;; the same information as a sparse int vector for the C bridge.
breaks-allowed forbidden-positions
;; Glue params snapshot at para creation time (plist)
glue-params
(dp-cache nil :type hash-table))
@ -247,14 +252,19 @@ Returns (boxes-vector . hyphen-positions-vector)."
(defun ekp--str-type (str)
"Classify single-character string STR.
Returns one of `space', `latin', `cjk', `cjk-punct'."
Returns one of `space', `latin', `cjk', `cjk-open', `cjk-close'.
`cjk-open' must not end a line; `cjk-close' must not start one
\(kinsoku) — enforced via `ekp-para-breaks-allowed'."
(cond
;; Whitespace or zero-width characters
((or (string-blank-p str) (= (string-width str) 0)) 'space)
;; a half-width cjk punct
((or (string= "“" str) (string= "”" str)) 'cjk)
((= (string-width str) 1) 'latin)
((ekp-cjk-fw-punct-p str) 'cjk-punct)
;; Opening punctuation (Ps/Pi), e.g. 「『(《
((ekp-cjk-opening-punct-p str) 'cjk-open)
;; Closing/other full-width punctuation, e.g. 。、」!?
((ekp-cjk-fw-punct-p str) 'cjk-close)
;; double-width (or wider): CJK-like content, including emoji
(t 'cjk)))
@ -292,14 +302,52 @@ mws between cjk and latin; nws means no whitespace. Space boxes
(cond
;; Space boxes: no additional glue needed
((or (eq before 'space) (eq after 'space)) 'nws)
;; Punctuation hugs its content: no glue after an opener,
;; none before a closer (these gaps are also unbreakable).
((eq before 'cjk-open) 'nws)
((eq after 'cjk-close) 'nws)
((and (eq before 'latin) (eq after 'latin)) 'lws)
((and (eq before 'cjk) (eq after 'cjk)) 'cws)
((or (and (eq before 'cjk) (eq after 'latin))
(and (eq before 'latin) (eq after 'cjk)))
'mws)
((or (eq before 'cjk-punct) (eq after 'cjk-punct)) 'cws))
;; Remaining punctuation adjacency (after a closer, or before
;; an opener): CJK spacing.
((or (eq before 'cjk-close) (eq after 'cjk-open)) 'cws))
'nws)))
(defconst ekp--no-line-start-chars ".,;:!?)]}%’”»›…·"
"Halfwidth/neutral punctuation that must not start a line.
Applies to boxes consisting solely of these characters (a lone comma
after a CJK char), never to words that merely begin with one
\(\".emacs\"). Fullwidth closers are covered by the `cjk-close'
class instead.")
(defconst ekp--no-line-end-chars "([{‘“«‹"
"Halfwidth/neutral punctuation that must not end a line.
Same box-level rule as `ekp--no-line-start-chars'; fullwidth openers
are covered by the `cjk-open' class.")
(defun ekp--box-pure-set-p (box set)
"Non-nil when BOX is non-empty and every char is a member of SET."
(let ((len (length box)) (chars (append set nil)) (i 0) (all t))
(when (> len 0)
(while (and all (< i len))
(unless (memq (aref box i) chars)
(setq all nil))
(setq i (1+ i)))
all)))
(defun ekp--box-no-line-start-p (box box-type)
"Non-nil if BOX must not appear at the start of a line."
(or (eq (car box-type) 'cjk-close)
(ekp--box-pure-set-p box ekp--no-line-start-chars)))
(defun ekp--box-no-line-end-p (box box-type)
"Non-nil if BOX must not appear at the end of a line."
(or (eq (cdr box-type) 'cjk-open)
(ekp--box-pure-set-p box ekp--no-line-end-chars)))
(defun ekp--compute-glue-types (boxes boxes-types hyphen-positions)
"Compute glue types for BOXES. Positions after HYPHEN-POSITIONS are `nws'."
(let* ((n (length boxes))
@ -454,7 +502,24 @@ Computes ALL data in one pass: text, params, and prefix arrays."
(mws-prefixs (make-vector (1+ n) 0))
(cws-prefixs (make-vector (1+ n) 0))
(lead-spaces (make-vector (1+ n) 0))
(trail-spaces (make-vector (1+ n) 0)))
(trail-spaces (make-vector (1+ n) 0))
(breaks-allowed (make-bool-vector (1+ n) t))
(forbidden nil))
;; Kinsoku via break permissions: a line may not end with an
;; opening-punct box, nor start with a closing-punct box — full-
;; and halfwidth alike. Punctuation also hugs its content: those
;; unbreakable gaps carry no glue.
(let ((k 1))
(while (< k n)
(when (or (ekp--box-no-line-end-p (aref boxes (1- k))
(aref boxes-types (1- k)))
(ekp--box-no-line-start-p (aref boxes k)
(aref boxes-types k)))
(aset breaks-allowed k nil)
(push k forbidden)
(unless (eq (aref glues-types k) 'nws)
(aset glues-types k 'nws)))
(setq k (1+ k))))
;; Single loop for all prefix computations
(dotimes (i n)
(let* ((box-w (aref boxes-widths i))
@ -510,6 +575,8 @@ Computes ALL data in one pass: text, params, and prefix arrays."
:cws-prefixs cws-prefixs
:lead-spaces lead-spaces
:trail-spaces trail-spaces
:breaks-allowed breaks-allowed
:forbidden-positions (vconcat (nreverse forbidden))
:glue-params (list :lws-ideal ekp-lws-ideal-pixel
:lws-stretch ekp-lws-stretch-pixel
:lws-shrink ekp-lws-shrink-pixel
@ -720,6 +787,7 @@ unreachable (only possible when ALLOW-EMERGENCY is nil)."
(cws-prefixs (ekp-para-cws-prefixs para))
(lead-spaces (ekp-para-lead-spaces para))
(trail-spaces (ekp-para-trail-spaces para))
(breaks-ok (ekp-para-breaks-allowed para))
(params (ekp-para-glue-params para))
(lws-stretch (plist-get params :lws-stretch))
(mws-stretch (plist-get params :mws-stretch))
@ -746,11 +814,20 @@ unreachable (only possible when ALLOW-EMERGENCY is nil)."
(lead-glue-min (- lead-glue-ideal (aref glue-shrinks i)))
(lead-glue-max (+ lead-glue-ideal (aref glue-stretches i)))
(lead-space (aref lead-spaces i))
(saw-allowed nil)
(k (1+ i)))
(catch 'break
(while (<= k n)
(if (not (or (= k n) (aref breaks-ok k)))
;; Break forbidden here (kinsoku, no-break span):
;; not a candidate; keep extending the line.
(setq k (1+ k))
(let* ((is-last (= k n))
(single-box (= k (1+ i)))
;; No permitted break strictly inside [i, k): the
;; run is atomic and eligible for emergency
;; handling, like a single box.
(atomic-run (not saw-allowed))
(end-with-hyphenp (aref hyph-flags (1- k)))
(hyph-w (if end-with-hyphenp hyphen-pixel 0))
(raw-ideal (- (aref ideal-prefixs k) ip-i lead-glue-ideal))
@ -764,16 +841,17 @@ unreachable (only possible when ALLOW-EMERGENCY is nil)."
space-w)
hyph-w)))
(cond
;; Line already too long: emergency-record single box,
;; Line already too long: emergency-record atomic run,
;; then stop extending.
((or (> minw line-pixel)
(and is-last (> ideal line-pixel)))
(when (and single-box allow-emergency)
(when (and atomic-run allow-emergency)
(ekp--dp-relax-emergency
demerits backptrs rests gaps hyphen-counts
fitness-classes i k prev-dem
(- line-pixel ideal) end-with-hyphenp
prev-hyphen-count))
prev-hyphen-count
(unless single-box (ekp--gaps-between para i k))))
(throw 'break nil))
;; Valid break point
((or (<= minw line-pixel maxw)
@ -841,15 +919,17 @@ unreachable (only possible when ALLOW-EMERGENCY is nil)."
(aset gaps k line-gaps)
(aset fitness-classes k fitness)
(aset hyphen-counts k new-hyphen)))))
;; Invalid single box (rigid underfull): emergency
;; Invalid atomic run (rigid underfull): emergency
;; record so the DP cannot dead-end (2nd pass only).
((and single-box allow-emergency)
((and atomic-run allow-emergency)
(ekp--dp-relax-emergency
demerits backptrs rests gaps hyphen-counts
fitness-classes i k prev-dem
(- line-pixel ideal) end-with-hyphenp
prev-hyphen-count)))
(setq k (1+ k))))))))
prev-hyphen-count
(unless single-box (ekp--gaps-between para i k)))))
(setq saw-allowed t)
(setq k (1+ k)))))))))
;; Extract solution (nil when end unreachable in the strict pass)
(when (aref demerits n)
(let ((breaks (ekp--dp-trace-breaks backptrs n)))
@ -861,9 +941,12 @@ unreachable (only possible when ALLOW-EMERGENCY is nil)."
(defun ekp--dp-relax-emergency (demerits backptrs rests gaps hyphen-counts
fitness-classes i k prev-dem rest
end-with-hyphenp prev-hyphen-count)
"Record an emergency (over/underfull single-box) break at K from I.
end-with-hyphenp prev-hyphen-count
&optional line-gaps)
"Record an emergency (over/underfull atomic-run) break at K from I.
REST is line-pixel minus the line's ideal width (may be negative).
LINE-GAPS is the (lws mws cws) gap-count list for multi-box runs
\(nil for single boxes, which render via the single-box path).
Only replaces an existing entry when strictly better."
(let ((total (+ prev-dem
(expt (+ ekp-line-penalty ekp--infinite-badness) 2)
@ -873,7 +956,7 @@ Only replaces an existing entry when strictly better."
(aset demerits k total)
(aset backptrs k i)
(aset rests k rest)
(aset gaps k nil)
(aset gaps k line-gaps)
(aset fitness-classes k 3)
(aset hyphen-counts k
(if end-with-hyphenp (1+ prev-hyphen-count) 0)))))
@ -922,6 +1005,7 @@ breaks when no valid layout exists."
(glue-stretches (ekp-para-glue-stretches para))
(lead-spaces (ekp-para-lead-spaces para))
(trail-spaces (ekp-para-trail-spaces para))
(breaks-ok (ekp-para-breaks-allowed para))
(params (ekp-para-glue-params para))
(lws-stretch (plist-get params :lws-stretch))
(mws-stretch (plist-get params :mws-stretch))
@ -947,11 +1031,16 @@ breaks when no valid layout exists."
(lead-glue-min (- lead-glue-ideal (aref glue-shrinks i)))
(lead-glue-max (+ lead-glue-ideal (aref glue-stretches i)))
(lead-space (aref lead-spaces i))
(saw-allowed nil)
(k (1+ i)))
(catch 'break
(while (<= k n)
(if (not (or (= k n) (aref breaks-ok k)))
;; Break forbidden here: keep extending the line.
(setq k (1+ k))
(let* ((is-last (= k n))
(single-box (= k (1+ i)))
(atomic-run (not saw-allowed))
(end-with-hyphenp
(ekp--hyphenate-p hyphen-positions (1- k)))
(hyph-w (if end-with-hyphenp hyphen-pixel 0))
@ -970,12 +1059,15 @@ breaks when no valid layout exists."
(cond
((or (> minw line-pixel)
(and is-last (> ideal line-pixel)))
(when (and single-box allow-emergency)
(when (and atomic-run allow-emergency)
(setq candidate
(list (+ (expt (+ ekp-line-penalty
ekp--infinite-badness) 2)
(* (float adjustment) adjustment))
adjustment nil 3
adjustment
(unless single-box
(ekp--gaps-between para i k))
3
(if end-with-hyphenp
(1+ prev-hyphen-count) 0)))
(ekp--dp-loose-relax states counts-at k (1+ lc) i
@ -1031,17 +1123,21 @@ breaks when no valid layout exists."
adjustment line-gaps fitness nh)))))
(ekp--dp-loose-relax states counts-at k (1+ lc) i
prev-dem candidate))
((and single-box allow-emergency)
((and atomic-run allow-emergency)
(setq candidate
(list (+ (expt (+ ekp-line-penalty
ekp--infinite-badness) 2)
(* (float adjustment) adjustment))
adjustment nil 3
adjustment
(unless single-box
(ekp--gaps-between para i k))
3
(if end-with-hyphenp
(1+ prev-hyphen-count) 0)))
(ekp--dp-loose-relax states counts-at k (1+ lc) i
prev-dem candidate)))
(setq k (1+ k))))))))
(setq saw-allowed t)
(setq k (1+ k)))))))))
;; Select final state: line count closest to (optimal + looseness).
;; nil when the end is unreachable (strict pass only).
(when-let* ((end-counts (aref counts-at n)))
@ -1150,7 +1246,7 @@ If `ekp-use-c-module' is non-nil and the C module is available (and
dp-result))
(defun ekp--prepare-para-for-c (para line-pixel)
"Prepare PARA data as an 11-element vector for the C batch API."
"Prepare PARA data as a 12-element vector for the C batch API."
(vector (ekp-para-ideal-prefixs para)
(ekp-para-min-prefixs para)
(ekp-para-max-prefixs para)
@ -1161,7 +1257,8 @@ If `ekp-use-c-module' is non-nil and the C module is available (and
(ekp-para-hyphen-pixel para)
line-pixel
(ekp-para-lead-spaces para)
(ekp-para-trail-spaces para)))
(ekp-para-trail-spaces para)
(ekp-para-forbidden-positions para)))
(defun ekp--dp-cache-via-c (para line-pixel)
"Compute breaks using the C module with PARA's precomputed arrays.
@ -1179,7 +1276,8 @@ runs the pure DP. Falls back to Elisp when the C call fails."
(ekp-para-hyphen-pixel para)
line-pixel
(ekp-para-lead-spaces para)
(ekp-para-trail-spaces para)))
(ekp-para-trail-spaces para)
(ekp-para-forbidden-positions para)))
(c-breaks (car result))
(c-cost (cdr result)))
(if (null c-breaks)

59
ekp_c/ekp.c
View File

@ -323,7 +323,7 @@ static emacs_value Fekp_c_break_with_arrays(emacs_env *env, ptrdiff_t nargs,
{
(void)data;
if (!ekp_global || nargs < 11)
if (!ekp_global || nargs < 12)
return env->intern(env, "nil");
/* Get prefix array sizes (n+1 elements) */
@ -383,6 +383,18 @@ static emacs_value Fekp_c_break_with_arrays(emacs_env *env, ptrdiff_t nargs,
int32_t hyph_width = env->extract_integer(env, args[7]);
int32_t line_width = env->extract_integer(env, args[8]);
/* Forbidden break positions (sorted gap indices, may be empty) */
ptrdiff_t forb_count = env->vec_size(env, args[11]);
int32_t *forb_pos = NULL;
if (forb_count > 0) {
forb_pos = malloc(forb_count * sizeof(int32_t));
if (forb_pos) {
for (ptrdiff_t i = 0; i < forb_count; i++) {
forb_pos[i] = env->extract_integer(env, env->vec_get(env, args[11], i));
}
}
}
/* Call the pure DP function */
ekp_result_t *result = ekp_break_with_prefixes(
ideal_prefix, min_prefix, max_prefix,
@ -390,12 +402,14 @@ static emacs_value Fekp_c_break_with_arrays(emacs_env *env, ptrdiff_t nargs,
n,
hyph_pos, hyph_count > 0 ? (size_t)hyph_count : 0,
hyph_width, line_width,
lead_spaces, trail_spaces);
lead_spaces, trail_spaces,
forb_pos, (forb_pos && forb_count > 0) ? (size_t)forb_count : 0);
free(ideal_prefix); free(min_prefix); free(max_prefix);
free(glue_ideals); free(glue_shrinks); free(glue_stretches);
free(lead_spaces); free(trail_spaces);
free(hyph_pos);
free(forb_pos);
if (!result)
return env->intern(env, "nil");
@ -428,7 +442,8 @@ static bool extract_paragraph_data(
int32_t **glue_ideals, int32_t **glue_shrinks, int32_t **glue_stretches,
int32_t **hyph_pos, size_t *n, ptrdiff_t *hyph_count,
int32_t *hyph_width, int32_t *line_width,
int32_t **lead_spaces, int32_t **trail_spaces)
int32_t **lead_spaces, int32_t **trail_spaces,
int32_t **forb_pos, ptrdiff_t *forb_count)
{
ptrdiff_t prefix_len = env->vec_size(env, args[0]);
if (prefix_len <= 1)
@ -482,6 +497,17 @@ static bool extract_paragraph_data(
*hyph_width = env->extract_integer(env, args[7]);
*line_width = env->extract_integer(env, args[8]);
*forb_count = env->vec_size(env, args[11]);
*forb_pos = NULL;
if (*forb_count > 0) {
*forb_pos = malloc(*forb_count * sizeof(int32_t));
if (*forb_pos) {
for (ptrdiff_t i = 0; i < *forb_count; i++) {
(*forb_pos)[i] = env->extract_integer(env, env->vec_get(env, args[11], i));
}
}
}
return true;
}
@ -519,13 +545,14 @@ static emacs_value Fekp_c_break_batch(emacs_env *env, ptrdiff_t nargs,
int32_t **all_hyph = calloc(para_count, sizeof(int32_t *));
int32_t **all_lead = calloc(para_count, sizeof(int32_t *));
int32_t **all_trail = calloc(para_count, sizeof(int32_t *));
int32_t **all_forb = calloc(para_count, sizeof(int32_t *));
if (!inputs || !all_ideal || !all_min || !all_max ||
!all_glue_i || !all_glue_sh || !all_glue_st || !all_hyph ||
!all_lead || !all_trail) {
!all_lead || !all_trail || !all_forb) {
free(inputs); free(all_ideal); free(all_min); free(all_max);
free(all_glue_i); free(all_glue_sh); free(all_glue_st); free(all_hyph);
free(all_lead); free(all_trail);
free(all_lead); free(all_trail); free(all_forb);
return env->intern(env, "nil");
}
@ -533,14 +560,14 @@ static emacs_value Fekp_c_break_batch(emacs_env *env, ptrdiff_t nargs,
for (ptrdiff_t p = 0; p < para_count; p++) {
emacs_value para_vec = env->vec_get(env, args[0], p);
/* Extract 11 arguments from this paragraph's vector */
emacs_value para_args[11];
for (int i = 0; i < 11; i++) {
/* Extract 12 arguments from this paragraph's vector */
emacs_value para_args[12];
for (int i = 0; i < 12; i++) {
para_args[i] = env->vec_get(env, para_vec, i);
}
size_t n;
ptrdiff_t hyph_count;
ptrdiff_t hyph_count, forb_count;
int32_t hyph_width, line_width;
if (!extract_paragraph_data(env, para_args,
@ -548,16 +575,18 @@ static emacs_value Fekp_c_break_batch(emacs_env *env, ptrdiff_t nargs,
&all_glue_i[p], &all_glue_sh[p], &all_glue_st[p],
&all_hyph[p], &n, &hyph_count,
&hyph_width, &line_width,
&all_lead[p], &all_trail[p])) {
&all_lead[p], &all_trail[p],
&all_forb[p], &forb_count)) {
/* Cleanup on failure */
for (ptrdiff_t j = 0; j < p; j++) {
free(all_ideal[j]); free(all_min[j]); free(all_max[j]);
free(all_glue_i[j]); free(all_glue_sh[j]); free(all_glue_st[j]);
free(all_hyph[j]); free(all_lead[j]); free(all_trail[j]);
free(all_forb[j]);
}
free(inputs); free(all_ideal); free(all_min); free(all_max);
free(all_glue_i); free(all_glue_sh); free(all_glue_st); free(all_hyph);
free(all_lead); free(all_trail);
free(all_lead); free(all_trail); free(all_forb);
return env->intern(env, "nil");
}
@ -574,6 +603,9 @@ static emacs_value Fekp_c_break_batch(emacs_env *env, ptrdiff_t nargs,
inputs[p].line_width = line_width;
inputs[p].lead_spaces = all_lead[p];
inputs[p].trail_spaces = all_trail[p];
inputs[p].forbidden_positions = all_forb[p];
inputs[p].forbidden_count =
(all_forb[p] && forb_count > 0) ? (size_t)forb_count : 0;
}
/* Process all paragraphs in parallel */
@ -584,10 +616,11 @@ static emacs_value Fekp_c_break_batch(emacs_env *env, ptrdiff_t nargs,
free(all_ideal[p]); free(all_min[p]); free(all_max[p]);
free(all_glue_i[p]); free(all_glue_sh[p]); free(all_glue_st[p]);
free(all_hyph[p]); free(all_lead[p]); free(all_trail[p]);
free(all_forb[p]);
}
free(inputs); free(all_ideal); free(all_min); free(all_max);
free(all_glue_i); free(all_glue_sh); free(all_glue_st); free(all_hyph);
free(all_lead); free(all_trail);
free(all_lead); free(all_trail); free(all_forb);
if (!results)
return env->intern(env, "nil");
@ -697,7 +730,7 @@ MEASURE-FUNC: function that takes a string and returns pixel width\n\n\
Returns (BREAKS . TOTAL-COST) where BREAKS is list of break positions.\n\n\
(fn STRING HYPHENATOR-INDEX LINE-WIDTH MEASURE-FUNC)");
defun(env, "ekp-c-break-with-arrays", 11, 11, Fekp_c_break_with_arrays,
defun(env, "ekp-c-break-with-arrays", 12, 12, Fekp_c_break_with_arrays,
"Break lines using Elisp's pre-computed prefix arrays (preferred API).\n\n\
IDEAL-PREFIX: vector of ideal width prefix sums (n+1 elements)\n\
MIN-PREFIX: vector of min width prefix sums (n+1 elements)\n\

60
ekp_c/ekp_kp.c
View File

@ -138,6 +138,11 @@ typedef struct {
size_t hyphen_count;
int32_t hyphen_width;
/* Forbidden break positions (kinsoku, no-break spans): sorted gap
* indices where a line may NOT end. Nullable. */
const int32_t *forbidden_positions;
size_t forbidden_count;
/* Space-box run widths (nullable, n+1 elements each):
* lead_spaces[i] = width of space-box run starting at box i
* trail_spaces[k] = width of space-box run ending at box k-1
@ -186,6 +191,28 @@ static inline bool dp_is_hyphen(const dp_input_t *in, size_t pos)
return (size_t)in->hyphen_positions[lo] == pos;
}
/*
* Shared forbidden-break check for dp_input_t (binary search).
*/
static inline bool dp_is_forbidden(const dp_input_t *in, size_t pos)
{
if (!in->forbidden_positions || in->forbidden_count == 0)
return false;
size_t lo = 0;
size_t hi = in->forbidden_count - 1;
while (lo < hi) {
size_t mid = lo + (hi - lo) / 2;
if ((size_t)in->forbidden_positions[mid] < pos)
lo = mid + 1;
else
hi = mid;
}
return (size_t)in->forbidden_positions[lo] == pos;
}
/*
* Core DP algorithm - shared by both entry points
* Processes position i, trying all end positions k.
@ -244,9 +271,21 @@ static void dp_process_position(
int32_t lead_space = in->lead_spaces ? in->lead_spaces[i] : 0;
/* Try extending to each position k > i */
bool saw_allowed = false;
for (size_t k = i + 1; k <= n; k++) {
bool is_last = (k == n);
/* Break forbidden here (kinsoku, no-break span): not a
* candidate; keep extending the line. */
if (!is_last && dp_is_forbidden(in, k))
continue;
bool is_single_box = (k == i + 1);
/* No permitted break strictly inside [i, k): the run is atomic
* and eligible for emergency handling, like a single box. */
bool atomic_run = !saw_allowed;
saw_allowed = true;
bool end_hyphen = dp_is_hyphen(in, k - 1);
int32_t hyph_w = end_hyphen ? in->hyphen_width : 0;
@ -266,7 +305,7 @@ static void dp_process_position(
/* Too long? (last line is never shrunk below its ideal) */
if (min_w > line_width || (is_last && ideal > line_width)) {
if (is_single_box && in->allow_emergency)
if (atomic_run && in->allow_emergency)
dp_relax_emergency(in, i, k, prev_dem, prev_hyph, prev_lines,
line_width - ideal, end_hyphen,
demerits, backptrs, rest_pixels,
@ -279,9 +318,9 @@ static void dp_process_position(
(is_last && ideal <= line_width);
if (!valid) {
/* Rigid underfull single box: emergency-record so the
/* Rigid underfull atomic run: emergency-record so the
* position after it stays reachable (2nd pass only). */
if (is_single_box && in->allow_emergency)
if (atomic_run && in->allow_emergency)
dp_relax_emergency(in, i, k, prev_dem, prev_hyph, prev_lines,
line_width - ideal, end_hyphen,
demerits, backptrs, rest_pixels,
@ -602,7 +641,9 @@ ekp_result_t *ekp_break_with_prefixes(
int32_t hyphen_width,
int32_t line_width,
const int32_t *lead_spaces,
const int32_t *trail_spaces)
const int32_t *trail_spaces,
const int32_t *forbidden_positions,
size_t forbidden_count)
{
if (!ideal_prefix || !min_prefix || !max_prefix || n == 0 || line_width <= 0)
return NULL;
@ -651,6 +692,8 @@ ekp_result_t *ekp_break_with_prefixes(
.hyphen_positions = hyphen_positions,
.hyphen_count = hyphen_count,
.hyphen_width = hyphen_width,
.forbidden_positions = forbidden_positions,
.forbidden_count = forbidden_count,
.lead_spaces = lead_spaces,
.trail_spaces = trail_spaces,
.n = n,
@ -771,7 +814,8 @@ static void batch_worker(void *arg)
in->n,
in->hyphen_positions, in->hyphen_count,
in->hyphen_width, in->line_width,
in->lead_spaces, in->trail_spaces);
in->lead_spaces, in->trail_spaces,
in->forbidden_positions, in->forbidden_count);
}
/*
@ -799,7 +843,8 @@ ekp_result_t **ekp_break_batch(ekp_batch_input_t *inputs, size_t count)
in->n,
in->hyphen_positions, in->hyphen_count,
in->hyphen_width, in->line_width,
in->lead_spaces, in->trail_spaces);
in->lead_spaces, in->trail_spaces,
in->forbidden_positions, in->forbidden_count);
}
return results;
}
@ -816,7 +861,8 @@ ekp_result_t **ekp_break_batch(ekp_batch_input_t *inputs, size_t count)
in->n,
in->hyphen_positions, in->hyphen_count,
in->hyphen_width, in->line_width,
in->lead_spaces, in->trail_spaces);
in->lead_spaces, in->trail_spaces,
in->forbidden_positions, in->forbidden_count);
}
return results;
}

11
ekp_c/ekp_module.h
View File

@ -16,7 +16,7 @@
/* Version */
#define EKP_VERSION_MAJOR 1
#define EKP_VERSION_MINOR 1
#define EKP_VERSION_MINOR 2
/* Limits */
#define EKP_MAX_PATTERN_LEN 64
@ -237,6 +237,9 @@ void ekp_result_destroy(ekp_result_t *r);
* starting at box i; entry 0 must be 0 (indentation kept)
* trail_spaces: (n+1 elements, nullable) width of the space-box run
* ending at box k-1
* forbidden_positions: sorted array of gap indices where a line may
* NOT end (kinsoku, no-break spans); nullable
* forbidden_count: length of forbidden_positions
*/
ekp_result_t *ekp_break_with_prefixes(
const int32_t *ideal_prefix,
@ -251,7 +254,9 @@ ekp_result_t *ekp_break_with_prefixes(
int32_t hyphen_width,
int32_t line_width,
const int32_t *lead_spaces,
const int32_t *trail_spaces);
const int32_t *trail_spaces,
const int32_t *forbidden_positions,
size_t forbidden_count);
/*
* Batch input for parallel processing
@ -270,6 +275,8 @@ typedef struct {
int32_t line_width;
const int32_t *lead_spaces; /* nullable, n+1 elements */
const int32_t *trail_spaces; /* nullable, n+1 elements */
const int32_t *forbidden_positions; /* nullable, sorted gap indices */
size_t forbidden_count;
} ekp_batch_input_t;
/*

38
tests/ekp-tests.el
View File

@ -114,16 +114,36 @@ Used to verify no content is lost by justification."
(should (equal (append (ekp-split-to-boxes "中文排版") nil)
'("中" "文" "排" "版"))))
(ert-deftest ekp-test-split-cjk-punct-attaches ()
"Closing CJK punctuation attaches to the preceding char (kinsoku)."
(let ((boxes (append (ekp-split-to-boxes "中文，排版。") nil)))
(should (member "文，" boxes))
(should (member "版。" boxes))))
(ert-deftest ekp-test-split-cjk-punct-own-boxes ()
"CJK punctuation is its own box; kinsoku lives in break permissions."
(should (equal (append (ekp-split-to-boxes "中文，排版。") nil)
'("中" "文" "，" "排" "版" "。")))
(should (equal (append (ekp-split-to-boxes "看《中文》吧") nil)
'("看" "《" "中" "文" "》" "吧"))))
(ert-deftest ekp-test-split-cjk-opening-punct-holds ()
"Opening CJK punctuation attaches to the following char (kinsoku)."
(let ((boxes (append (ekp-split-to-boxes "看《中文》吧") nil)))
(should (member "《中" boxes))))
(ert-deftest ekp-test-breaks-allowed-kinsoku ()
"Break permissions forbid line-initial closers and line-final openers."
(let* ((para (ekp--get-para "看《中文》吧，好。」的"))
(boxes (append (ekp-para-boxes para) nil))
(ok (ekp-para-breaks-allowed para)))
;; boxes: 看 《 中 文 》 吧 , 好 。 」 的
(should (equal boxes '("看" "《" "中" "文" "》" "吧" "，"
"好" "。" "」" "的")))
;; forbidden: after 《 (idx 2), before 》 (idx 4), before , (6),
;; before 。 (8), before 」 (9)
(dolist (k '(2 4 6 8 9))
(should-not (aref ok k)))
;; allowed elsewhere, e.g. 看|《, 》|吧, ,|好, 」|的
(dolist (k '(1 5 7 10))
(should (aref ok k)))))
(ert-deftest ekp-test-kinsoku-rendered-output ()
"No rendered line starts with a closer or ends with an opener."
(let ((text "他说:「今天天气很好。」然后就离开了这里,再也没有回来过。"))
(dolist (w (number-sequence 30 200 7))
(dolist (line (split-string (ekp-pixel-justify text w) "\n"))
(should-not (string-match-p "\\`[。、,;:」』)》!?]" line))
(should-not (string-match-p "[「『(《]\\'" line))))))
(ert-deftest ekp-test-split-fullwidth-alnum-not-punct ()
"Fullwidth letters/digits are content, not punctuation."