hccccc01333/dsh-eval
10Last commit Aug 14, 2026
dsh-eval DSH plugin
dsh-eval is an installable DeepSeek Harness evaluation plugin that runs benchmark cases through headless dsh profiles, harvests session logs as traces, calculates run metrics, and produces JSON and Markdown reports. It also supports paired run comparisons, LLM judging, recorded-log replay, and importing Codex or Claude Code logs.
How to install the dsh-eval DSH plugin
dsh plugin --profile eval add dsh-evalCopying does not run this command. Review the repository and version before installing the dsh-eval DSH plugin.
dsh-eval DSH plugin data source
dsh-eval DSH plugin snapshot date: Aug 18, 2026
indexed
What the dsh-eval DSH plugin can do
- Runs YAML-defined benchmarks with headless dsh subprocesses across cases and trials
- Harvests persisted session logs as trial traces, including merged subagent logs
- Calculates task, tool, tool-selection, step, token, context, latency, cost, retry, and invalid-tool-call metrics
- Supports scripted task checks and expected-tool matching
- Uses optional LLM judge configuration for final-answer scoring and hallucination flags
- Generates JSON run artifacts and Markdown reports
- Compares two runs with signed B-minus-A deltas and win/lose/tie statistics
- Replays recorded logs and imports Codex or Claude Code session logs
Where the dsh-eval DSH plugin fits
- Regression testing DeepSeek Harness agents or skills
- Comparing two agent, prompt, model, or configuration variants
- Running repeatable benchmark suites in CI using recorded logs
- Analyzing tool selection, task completion, latency, token use, and estimated cost
- Converting Codex or Claude Code session logs into evaluation runs
Who the dsh-eval DSH plugin is for
- DeepSeek Harness plugin and agent developers
- Teams maintaining agent benchmark suites
- Engineers evaluating changes to agent configurations or models
dsh-eval DSH plugin limitations
- The package targets official @deepseek-ai releases with 0.1.0-rc.6 peer dependencies.
- Benchmark execution requires a dsh launcher and runs one headless dsh subprocess per case and trial.
- Task success and tool-selection metrics depend on benchmark expected.check and expected.tool configuration.
- LLM-based final-answer scores and hallucination flags require judge configuration.
- Parallel trial execution and a web dashboard are listed as roadmap items, not current features.
Objective checks
These signals come from public metadata scans of the dsh-eval DSH plugin repository. They are not a security certification.
- info
license declared
GitHub reports the repository license as MIT.
- info
package manifest not found
No root package.json was captured in the latest GitHub snapshot.
- info
source available
Public GitHub source metadata is available for this registry snapshot.
dsh-eval DSH plugin alternatives
dsh-eval DSH plugin questions
What is the dsh-eval DSH plugin?
dsh-eval is a project included in the current DSH plugin snapshot. This page organizes repository metadata and reviewed notes, but you should still inspect the upstream repository before running it.
How do I install the dsh-eval DSH plugin?
The page shows the install command captured from the source record. Copying does not run it, so confirm the package, profile, and version against upstream documentation first.
Which DSH profile does dsh-eval use?
The snapshot records eval as the dsh-eval profile. DSH conventions can change quickly, so verify the manifest and README before installation.
Is dsh-eval compatible with the current DSH release?
The source data does not include a verified compatibility matrix. Check recent commits, releases, and DSH breaking changes, then test in a profile you can roll back.
What should I do if dsh-eval fails to load?
Check the install command, selected profile, dependency logs, and Cordis startup output first. Keep the exact error string, then compare it with this site's fix pages and upstream issues.
What are the closest alternatives to dsh-eval?
Related plugins on this page come from the current snapshot and are only candidates. Compare capabilities, maintenance activity, installation method, and limitations before choosing one.