2.7 KiB
Reproducible benchmark guide
This document describes the benchmark runner shipped with TP 1.0. It measures the current retained/reactive runtime; it is not a historical TP 0.3 stack benchmark and it does not impose a release threshold.
Run
Use the Makefile entry point:
make benchmark
The equivalent batch command is:
emacs -Q --batch -L . -l tp-benchmark.el -f tp-benchmark-run
The runner uses fixed seeds 1, 7, 42, 747555 and generated seed 8675309. Each scenario performs a correctness assertion before timing the operation. Record the Emacs version, machine, seed, and full output when comparing runs.
Scenarios
For every seed, the runner executes:
| Scenario | Fixtures |
|---|---|
| large-text | strings of 100,000 and 1,000,000 characters; set and presence-aware search |
| fragmented | 1,000, 10,000, and 50,000 alternating property intervals |
| retained-keyed-reconcile | retained content with 10, 100, and 1,000 stable keyed entries |
| signal-sparse-update | one target binding beside 1, 100, and 10,000 unrelated bindings |
| transaction-batch | 1, 100, and 10,000 writes to one retained surface |
| equal-write-noop | the same write counts, all equal to the committed value |
The retained scenarios verify stable object reconciliation, correct published text, dependency-local recomputation, one publication for a batch, and no revision change for equal writes. A failed assertion aborts the run instead of producing misleading timing evidence.
Output
Each row is a whitespace-separated key/value record. The stable fields are:
scenario, status, fixture, seed, requested, actual, operations, objects, subscribers, invalidated, recomputed, skipped, text-operations, property-operations, touched, revision, published, elapsed, gcs, and note.
The output is deliberately machine-readable enough for local comparison, but it is not a compatibility format. Interpret it together with the scenario source in tp-benchmark.el.
Interpretation
These measurements are advisory. Runtime, garbage collection, Emacs build, machine load, and buffer implementation details affect absolute timings. Compare like-for-like runs, inspect correctness failures first, and use the reports/counters to explain a regression:
- fragmented measures expose interval-run scaling;
- retained-keyed-reconcile measures keyed object reuse and publication work;
- signal-sparse-update checks that unrelated bindings are not recomputed;
- transaction-batch measures deduplicated recomputation and one surface commit;
- equal-write-noop checks that equal values do not publish a new revision.
For the contracts behind these scenarios, read API semantics, architecture, and the public API reference.