dsh-eval DSH plugin

dsh-eval is an installable DeepSeek Harness evaluation plugin that runs benchmark cases through headless dsh profiles, harvests session logs as traces, calculates run metrics, and produces JSON and Markdown reports. It also supports paired run comparisons, LLM judging, recorded-log replay, and importing Codex or Claude Code logs.

ui-productivity terminal developer-tools security-policy skills-workflows remote-executioneval

How to install the dsh-eval DSH plugin

dsh plugin --profile eval add dsh-eval

Copying does not run this command. Review the repository and version before use.

Data source

Snapshot date: Aug 15, 2026

indexed

What the dsh-eval DSH plugin can do

Runs YAML-defined benchmarks with headless dsh subprocesses across cases and trials
Harvests persisted session logs as trial traces, including merged subagent logs
Calculates task, tool, tool-selection, step, token, context, latency, cost, retry, and invalid-tool-call metrics
Supports scripted task checks and expected-tool matching
Uses optional LLM judge configuration for final-answer scoring and hallucination flags
Generates JSON run artifacts and Markdown reports
Compares two runs with signed B-minus-A deltas and win/lose/tie statistics
Replays recorded logs and imports Codex or Claude Code session logs

Where the dsh-eval DSH plugin fits

Regression testing DeepSeek Harness agents or skills
Comparing two agent, prompt, model, or configuration variants
Running repeatable benchmark suites in CI using recorded logs
Analyzing tool selection, task completion, latency, token use, and estimated cost
Converting Codex or Claude Code session logs into evaluation runs

Who the dsh-eval DSH plugin is for

DeepSeek Harness plugin and agent developers
Teams maintaining agent benchmark suites
Engineers evaluating changes to agent configurations or models

dsh-eval DSH plugin limitations

The package targets official @deepseek-ai releases with 0.1.0-rc.6 peer dependencies.
Benchmark execution requires a dsh launcher and runs one headless dsh subprocess per case and trial.
Task success and tool-selection metrics depend on benchmark expected.check and expected.tool configuration.
LLM-based final-answer scores and hallucination flags require judge configuration.
Parallel trial execution and a web dashboard are listed as roadmap items, not current features.

Objective checks

These signals come from public metadata scans. They are not a security certification.

license declared
GitHub reports the repository license as MIT.
info
package manifest not found
No root package.json was captured in the latest GitHub snapshot.
info
source available
Public GitHub source metadata is available for this registry snapshot.
info

dsh-eval DSH plugin alternatives

dsh-automation

Schedule standalone coding tasks in fresh DeepSeek Harness Agent sessions, with workspace-scoped controls and durable run history.

dsh-agent-messaging

Cross-session agent-to-agent messaging for DeepSeek Harness — address another session by name and deliver a message into its inbox.

dsh-mnemon

Cross-agent, local-first persistent memory plugin for DeepSeek Harness (DSH), powered by Mnemon. It shares long-term memory across Mnemon-enabled agents and adds runtime memory, searchable project documents, semantic recall, knowledge graph, and a Sidebar UI.

awesome-deepseek-harness

A curated list of plugins, skills, MCP servers, patch/profile layers, orchestrators & UIs for DeepSeek Harness (DSH). Visualization · PPT · Coding · Agents · Loops (auto-research) and more. #dsh

dsh-eval DSH plugin questions

What is the dsh-eval DSH plugin?

dsh-eval is a project included in the current DSH plugin snapshot. This page organizes repository metadata and reviewed notes, but you should still inspect the upstream repository before running it.

How do I install the dsh-eval DSH plugin?

The page shows the install command captured from the source record. Copying does not run it, so confirm the package, profile, and version against upstream documentation first.

Which DSH profile does dsh-eval use?

The snapshot records eval as the dsh-eval profile. DSH conventions can change quickly, so verify the manifest and README before installation.

Is dsh-eval compatible with the current DSH release?

The source data does not include a verified compatibility matrix. Check recent commits, releases, and DSH breaking changes, then test in a profile you can roll back.

What should I do if dsh-eval fails to load?

Check the install command, selected profile, dependency logs, and Cordis startup output first. Keep the exact error string, then compare it with this site's fix pages and upstream issues.

What are the closest alternatives to dsh-eval?

Related plugins on this page come from the current snapshot and are only candidates. Compare capabilities, maintenance activity, installation method, and limitations before choosing one.