dongsheng123132/dsh-benchmark
30Last commit Aug 15, 2026
dsh-benchmark DSH plugin
dsh-benchmark defines an evidence protocol around fixed cases: explicit target and suite revisions, file-derived fingerprints, bounded subprocesses, raw measurements, versioned scoring, content-addressed reports, and baseline regression comparison. It is a formal Codex plugin and standalone MCP server, exposing inspect, run, and compare commands. The project deliberately stays at the deterministic execution-evidence layer, not evaluating subjective LLM quality.
How to install the dsh-benchmark DSH plugin
dsh plugin --profile benchmark add github:dongsheng123132/dsh-benchmarkCopying does not run this command. Review the repository and version before installing the dsh-benchmark DSH plugin.
dsh-benchmark DSH plugin data source
dsh-benchmark DSH plugin snapshot date: Aug 16, 2026
discovered
What the dsh-benchmark DSH plugin can do
- Registers three DSH tools: dsh_benchmark_inspect, dsh_benchmark_run, dsh_benchmark_compare
- Uses a manifest-based evidence model with target fingerprints, bounded subprocesses, and versioned scoring
- Provides a safety model with shell:false, path traversal prevention, and secret-bearing field rejection
- Includes a standalone MCP server with benchmark_manifest_lint and benchmark_report_address tools
- Supports CLI usage with inspect, run, and compare subcommands
Where the dsh-benchmark DSH plugin fits
- Benchmark a fixed JSONL runner to verify its performance against a baseline
- Run regression tests in CI for DSH plugins or tools with deterministic pass/fail criteria
- Compare two versions of a command-line tool using the same manifest and baseline
- Validate that a plugin update does not regress execution latency or output stability
- Generate content-addressed reports for audit or compliance purposes
Who the dsh-benchmark DSH plugin is for
- DSH plugin developers who need reproducible benchmarks for their tools
- Maintainers of DSH tools or plugins that require deterministic execution evidence
- Teams using DeepSeek Harness who want to track performance regression across releases
dsh-benchmark DSH plugin limitations
- Only evaluates commands and JSONL runners, not subjective LLM quality or skill output
- Requires Node.js 22+ and does not provide an OS sandbox for malicious code – only run trusted executables
- The first release focuses on deterministic measures; raw business output is excluded from reports
- MCP server only accepts bounded inline JSON and never executes commands or reads the filesystem
dsh-benchmark DSH plugin: from the repository README
Quoted from the dongsheng123132/dsh-benchmark README, the upstream source of the dsh-benchmark DSH plugin. Copyright remains with the original authors.
Reproducible, deterministic benchmark evidence for [DeepSeek Harness](https://github.com/deepseek-ai/DeepSeek-Harness) tools and plugins. This project deliberately does **not** duplicate `dsh-batch-regression`, which runs one shell command repeatedly for median/distribution statistics. `dsh-benchmark` defines an evidence protocol around fixed cases: explicit target and suite revisions, file-derived target fingerprints, bounded argv-only subprocesses, raw measurements, versioned deterministic scoring, content-addressed reports, and baseline regression comparison. The first release evaluates commands and JSONL runners, not subjective LLM quality. Version 0.2.0 is a formal Codex plugin and standalone proof-only MCP server, and uses the namespace export shape required by the stock DSH Web Loader. A real Cordis boot regression test guards that loader contract. Adjacent benchmark skills often grade Skill or LLM quality. This project stays at the deterministic execution-evidence layer: fixed target revisions and cases, raw bounded measurements without raw business output, versioned scoring, content-addressed reports, and baseline regression decisions. ## Evidence model An explicit m
Read the full READMERepository license: MIT
dsh-benchmark DSH plugin questions
How do I install dsh-benchmark?
Run `dsh plugin --profile benchmark add github:dongsheng123132/dsh-benchmark` in your DSH environment. This assumes you have DSH CLI installed and configured. The plugin registers three tools and a MCP server.
What is the evidence model in dsh-benchmark?
The evidence model uses a manifest that freezes target revision, suite name, executable, bounds (timeout, output cap, concurrency), fixed argv, expected exit code and hashes, scorer version, and threshold rules. Each run produces a content-addressed report with raw measurements and expectation checks, excluding command input and output bodies.
How do I create a benchmark manifest?
You can write a JSON manifest following the example at `examples/benchmark.example.json` in the repository. The manifest must define a suite, target, executable, cases with argv and optional JSONL stdin, scorer settings, and baseline thresholds. Run `dsh_benchmark_inspect` to validate your manifest.
What do the exit codes mean?
Exit code 0 means all tests passed. Exit code 2 means a report or comparison was written but the scorer failed (e.g., regression detected). Exit code 1 means a manifest or operational error (e.g., invalid manifest, missing file).
Is dsh-benchmark safe to run with untrusted manifests?
The safety model prevents shell interpolation, PATH lookups, workspace escape, and secret leakage. However, it is not an OS sandbox – you should only run trusted benchmark executables. The plugin rejects secret-bearing manifest fields and enforces bounded output and concurrency.