ekp/.phrase/phases/phase-kp-overhaul-20260726/issue_logical_text_api_20260728.md

Issue: Logical Text API Boundary 2026-07-28

issue011 [ ] Direct Elisp buffer APIs observe physical layout text

• Status: Runtime fix and complete repository gate implemented by task019–task022. The issue remains open only until the user accepts the visible behavior.
• Summary: The former buffer renderer wrote synthesized glue spaces, soft newlines, discretionary hyphens, and hidden payload text into the real buffer. Boundary adapters made save/copy/search appear logical, but ordinary Elisp character APIs still observed the physical layout.
• Environment: Text-property renderer and synchronous live flow after task021; Emacs 30.2.
• Historical Repro:
  1. Insert text into a temporary buffer.
  2. Run the pre-task020 ekp-justify-region with a narrow width.
  3. Compare the original text with buffer-string or buffer-substring-no-properties.
  4. Compare the same region through filter-buffer-substring.
• Historical Evidence: The pre-fix batch probe produced a 39-character physical representation from a 30-character logical string. buffer-string did not equal the original, while the boundary-filtered substring did.
• Expected vs Current:
  • Expected: Ordinary Elisp buffer-text APIs observe the original character sequence while EKP's justified presentation remains visible and editable.
  • Current implementation: buffer-string, buffer-substring-no-properties, direct character access, ordinary search, syntax, markers, and saving operate on the unchanged source characters. APIs that preserve or inspect text properties may still observe EKP's display metadata; copy/kill strips that owned metadata.
• Investigation:
  • The former ekp-justify-region deleted source text and inserted the physical rendered string.
  • Reversible ekp-glue, ekp-soft-break, ekp-soft-hyphen, and ekp-hidden markers made restoration possible but did not keep direct buffer reads logical.
  • Save/copy/isearch adapters compensated at individual boundaries and could not cover arbitrary Elisp consumers.
  • The correct owner is the buffer representation: source characters must remain authoritative, and layout must be presentational state only.
• 2026-07-29 Design Evidence:
  • The user ruled out overlays and selected text properties on existing source characters as the only projection mechanism.
  • (min-width ((PIXELS))) can add an exact absolute-pixel minimum width without adding buffer characters.
  • (space-width FACTOR) can shrink or stretch existing ASCII spaces. Combined with min-width, it rendered a natural 7px space at exact 3px and 20px targets in a clean GUI Emacs 30.2 probe while source text remained "a b".
  • line-prefix can provide display-only indentation for non-continuation lines. Native continuation lines still require wrap-prefix.
  • A replacing display string on an existing grapheme can show the grapheme followed by a discretionary hyphen and visual newline. A GUI probe displayed d-\n from the source d in abcdefgh; the source stayed eight characters, two screen lines were produced, one vertical-motion landed on source position 5, and a 20px line-prefix applied to both rows.
  • The proposed Latin hyphen owner is the complete grapheme immediately before the selected break. Its display string reproduces that grapheme, appends the hyphen and newline, and uses the cursor property to keep point on the source glyph.
  • Full reasoning, source links, limitations, and verification gates are recorded in tech-refer_text_property_layout_20260729.md.
• Root Cause: Logical document state and rendered layout shared one mutable character sequence. Reversible markers made the physical representation recoverable, not logical.
• Required Outcome: Remove the leak at the representation owner rather than adding per-command adapters. Ordinary Elisp character consumers must see logical content without losing justified display, exact round-trip, or seamless editing.
• Implemented Design:
  • Logical source characters stay in the real buffer.
  • The buffer renderer creates no overlay and inserts no layout character.
  • Existing ASCII spaces use combined space-width and absolute-pixel min-width; zero-source CJK/mixed glue uses min-width on the preceding complete grapheme; indentation uses line-prefix.
  • CJK breaks and Latin discretionary hyphens use replacing display strings on an existing complete grapheme.
  • Projection values have exact EKP owner properties, are nonsticky, and update under with-silent-modifications.
  • Foreign display, line-prefix, wrap-prefix, composition, and invisible owners make only the affected hard paragraph natural and diagnostic.
  • One buffer uses the narrowest live window as its authoritative width; simultaneous independent per-window plans are not claimed.
  • Live editing uses a natural unfinished line, bounded overflow/pullback, a stable active-flow anchor, line-signature convergence, IME deferral, stale-generation rejection, and bounded automatic paragraph work.
• Fix: ekp.el now exposes a reusable semantic layout plan while retaining the compatible returned-string API. ekp-buffer.el consumes that plan as a source-clean text-property projection and runs a distinct synchronous live-flow state machine.
• Verification to Date:
  • Buffer ERT passes 79/79, including unchanged source/tick/undo/modified state, direct logical reads, save/copy/isearch, foreign ownership, no-overlay checks, exact property forms, overflow, deletion pullback, exact type/delete break restoration, immediate active-line edge whitespace, active/inactive mark preservation, anchor lifecycle, IME, undo, resize, multiwindow width, and overload behavior.
  • A clean Emacs 30.2 static probe passes exact 1–64px ASCII and zero-source CJK glue, display-only breaks/hyphens, line-prefix, point, vertical motion, region, mouse, unchanged source, and zero overlays. The seven-case scale/remap/fringe/width matrix passes.
  • A 31.71s dynamic recording passes before/immediate/settled overflow and deletion checkpoints. Immediate and settled source/projection hashes match; deletion restores the original source/projection hashes and break count; every assertion is true and temporal review passes.
  • Final-source default and seeded-permuted ERT pass 170/170; every one of the 170 tests passes in a fresh Emacs process; C/Elisp fuzz passes 300/300.
  • Warning-as-error Elisp compilation, checkdoc, pinned package-lint, release/dictionary/static gates, all four C build profiles, full diff review, and anti-slop cleanup pass.
  • Independent final code review returns APPROVE with no findings; independent architecture review returns CLEAR.
• User Confirmation: On 2026-07-28, the user explicitly defined the target as clean text through Elisp APIs and requested recording only, with implementation deferred. On 2026-07-29, the user resumed design, excluded overlays, selected text properties, identified min-width, space-width, and line-prefix, requested a non-polluting hyphen implementation, then explicitly requested complete implementation under the persistent goal model. Final visible-behavior acceptance is pending.
• Resolved At: Unresolved.
• Resolved By: Pending.
• Commit: Pending.
• Related: ekp.el, ekp-buffer.el, task017–task022, spec_text_property_layout_engine_20260729.md, tech-refer_text_property_layout_20260729.md, postmortem/20260729-text-property-live-layout.md.