70 lines
2.7 KiB
Markdown
70 lines
2.7 KiB
Markdown
# Reproducible benchmark guide
|
|
|
|
This document describes the benchmark runner shipped with TP 1.0. It measures
|
|
the current retained/reactive runtime; it is not a historical TP 0.3 stack
|
|
benchmark and it does not impose a release threshold.
|
|
|
|
## Run
|
|
|
|
Use the Makefile entry point:
|
|
|
|
~~~sh
|
|
make benchmark
|
|
~~~
|
|
|
|
The equivalent batch command is:
|
|
|
|
~~~sh
|
|
emacs -Q --batch -L . -l tp-benchmark.el -f tp-benchmark-run
|
|
~~~
|
|
|
|
The runner uses fixed seeds 1, 7, 42, 747555 and generated seed 8675309. Each
|
|
scenario performs a correctness assertion before timing the operation. Record
|
|
the Emacs version, machine, seed, and full output when comparing runs.
|
|
|
|
## Scenarios
|
|
|
|
For every seed, the runner executes:
|
|
|
|
| Scenario | Fixtures |
|
|
| --- | --- |
|
|
| large-text | strings of 100,000 and 1,000,000 characters; set and presence-aware search |
|
|
| fragmented | 1,000, 10,000, and 50,000 alternating property intervals |
|
|
| retained-keyed-reconcile | retained content with 10, 100, and 1,000 stable keyed entries |
|
|
| signal-sparse-update | one target binding beside 1, 100, and 10,000 unrelated bindings |
|
|
| transaction-batch | 1, 100, and 10,000 writes to one retained surface |
|
|
| equal-write-noop | the same write counts, all equal to the committed value |
|
|
|
|
The retained scenarios verify stable object reconciliation, correct published
|
|
text, dependency-local recomputation, one publication for a batch, and no
|
|
revision change for equal writes. A failed assertion aborts the run instead of
|
|
producing misleading timing evidence.
|
|
|
|
## Output
|
|
|
|
Each row is a whitespace-separated key/value record. The stable fields are:
|
|
|
|
scenario, status, fixture, seed, requested, actual, operations, objects,
|
|
subscribers, invalidated, recomputed, skipped, text-operations,
|
|
property-operations, touched, revision, published, elapsed, gcs, and note.
|
|
|
|
The output is deliberately machine-readable enough for local comparison, but
|
|
it is not a compatibility format. Interpret it together with the scenario
|
|
source in tp-benchmark.el.
|
|
|
|
## Interpretation
|
|
|
|
These measurements are advisory. Runtime, garbage collection, Emacs build,
|
|
machine load, and buffer implementation details affect absolute timings.
|
|
Compare like-for-like runs, inspect correctness failures first, and use the
|
|
reports/counters to explain a regression:
|
|
|
|
- fragmented measures expose interval-run scaling;
|
|
- retained-keyed-reconcile measures keyed object reuse and publication work;
|
|
- signal-sparse-update checks that unrelated bindings are not recomputed;
|
|
- transaction-batch measures deduplicated recomputation and one surface commit;
|
|
- equal-write-noop checks that equal values do not publish a new revision.
|
|
|
|
For the contracts behind these scenarios, read [API semantics](API-SEMANTICS.md),
|
|
[architecture](ARCHITECTURE.md), and the [public API reference](API-REFERENCE.md).
|