BiBoyang/dsh-eval-harness
51Last commit Aug 14, 2026
dsh-eval-harness DSH plugin
dsh-eval-harness is a DSH plugin that provides a regression evaluation workflow for DSH plugin and skill development. It allows you to write YAML test cases, run them headlessly against a real DSH agent, parse session traces, assert expected behaviors, and compare results against a baseline to produce PASS/WARN/FAIL reports with CI-compatible exit codes. It is designed to be integrated into CI pipelines to catch regressions automatically.
How to install the dsh-eval-harness DSH plugin
dsh plugin --profile headless add dsh-eval-harnessCopying does not run this command. Review the repository and version before installing the dsh-eval-harness DSH plugin.
dsh-eval-harness DSH plugin data source
dsh-eval-harness DSH plugin snapshot date: Aug 16, 2026
discovered
What the dsh-eval-harness DSH plugin can do
- Runs eval_run to execute all YAML test cases in a directory, driving a headless DSH agent and collecting session traces.
- Supports multiple assertion types: turn_end, tools_called, output_contains, max_steps, max_tokens, no_tool_errors, tools_exact, tools_not_called, output_not_contains, output_matches, tool_args_contains, tool_result_contains, and LLM-as-judge (output_judge).
- Provides eval_gate to compare current report against a baseline report, outputting overall verdict (PASS/WARN/FAIL) and exit code for CI integration.
- Includes a built-in zero-dependency YAML subset parser for test case files.
- Supports concurrent test execution (concurrency option) with per-case isolated session directories and workspaces.
- Handles both zstd-compressed and plain JSONL session traces automatically.
Where the dsh-eval-harness DSH plugin fits
- Run regression tests for a DSH plugin or skill before merging a pull request.
- Integrate into CI pipeline to automatically gate deployments based on test results.
- Use LLM-as-judge to evaluate semantic correctness of agent responses when structural assertions are insufficient.
- Compare token consumption across runs to detect unexpected cost increases.
- Maintain a baseline of expected behavior and receive alerts when changes cause regressions.
Who the dsh-eval-harness DSH plugin is for
- DSH plugin developers who need to ensure their plugins don't break existing functionality.
- DSH skill authors who want to validate skill behavior with automated tests.
- CI/CD pipeline maintainers setting up regression gates for DSH-based projects.
dsh-eval-harness DSH plugin limitations
- Requires a real DSH environment with a headless profile and a valid DSH binary or npx fallback.
- LLM-as-judge depends on OpenAI-compatible API (default DeepSeek) and requires EVAL_JUDGE_API_KEY or DEEPSEEK_API_KEY environment variable; without it, judge calls will error.
- Only supports a subset of YAML: no anchors, multi-document, or complex flow structures; parsing errors are reported with line numbers.
- Concurrency is limited to 1 by default; higher concurrency may require careful resource management.
- Baseline report must be manually updated via a separate workflow after artifact changes.
dsh-eval-harness DSH plugin: from the repository README
Quoted from the BiBoyang/dsh-eval-harness README, the upstream source of the dsh-eval-harness DSH plugin. Copyright remains with the original authors.
DSH 插件/skill 作者的回归评测门禁：写 yaml 用例 → headless 驱动真实 agent 跑 → 解析 session trace 断言 → 对比 baseline 出 PASS/WARN/FAIL 报告与 CI 退出码。 ## 简介 给 DSH 插件/skill 的回归评测流程提供一个可进 CI 的门禁工具： 1. 用 yaml 写评测用例（prompt + 期望行为断言）； 2. `eval_run` 逐条 fork `dsh --profile headless --patch <overlay> <prompt>` 子进程跑真实 agent（overlay 把会话落盘切到隔离目录，每条用例独立 workspace），解析落盘的 `session.jsonl` / `session.jsonl.zstd` trace（多帧 zstd 直读），执行断言，写 `report.json` + `report.md`； 3. `eval_gate` 把本次报告与 baseline 报告对比，输出 `OVERALL=PASS|WARN|FAIL|N/A` 与退出码，供 CI 拦截回归。 ## 安装 已发布到 npm（[`dsh-eval-harness`](https://www.npmjs.com/package/dsh-eval-harness)）： ```sh dsh plugin --profile headless add dsh-eval-harness # 或从 GitHub 源码安装： # dsh plugin --profile headless add github:boyang/dsh-eval-harness # 验证挂载 dsh --profile headless --dump-config | grep dsh-eval-harness ``` ## 能力面 ### Tools | 工具 | 说明 | | --- | --- | | `eval_run` | 跑 cases_dir 下全部用例：headless 驱动真实 agent → 采集 session trace → 断言 → 写 report.json/report.md | | `eval_gate` | 对比 baseline 与本次报告，输出门禁判定（OVERALL/EXIT_CODE），strict 模式收紧 WARN 退出码 | ### Skills | Skill | 作用 | | --- | --- | | `eval` | 教模型帮用户编写评测用例（用例格式、断言编写要点、解析子集约束） | ## 用例格式（cases/*.yml） 一个文件一条用例： ```yaml name: 用例名
Read the full READMEThe repository declares no license. Check with the authors before using it.
dsh-eval-harness DSH plugin questions
How do I install dsh-eval-harness?
Install it as a DSH plugin using the command: `dsh plugin --profile headless add dsh-eval-harness`. You can also install from GitHub source: `dsh plugin --profile headless add github:boyang/dsh-eval-harness`. After installation, use `dsh --profile headless --dump-config | grep dsh-eval-harness` to verify the plugin is mounted.
How do I write a test case?
Create a YAML file in your cases directory following the format described in the README. Each file contains one case with fields like `name`, `prompt`, optional `require_plugins`, `tags`, `retries`, and an `assert` section. The `assert` section supports multiple assertion types such as `turn_end`, `tools_called`, `output_contains`, `max_steps`, `max_tokens`, etc. You can also use `output_judge` for LLM-based semantic evaluation.
How do I configure the LLM judge?
The judge uses OpenAI-compatible chat completions API. Set the environment variable `EVAL_JUDGE_API_KEY` (falls back to `DEEPSEEK_API_KEY`), `EVAL_JUDGE_BASE_URL` (default `https://api.deepseek.com`), and `EVAL_JUDGE_MODEL` (default `deepseek-chat`). If these are not set, judge calls will error. The judge is only invoked after all structural assertions pass, to save tokens.
How do I update the baseline report?
Use the provided GitHub Actions workflow `update-baseline.yml` (trigger manually from Actions tab). It will run all cases, overwrite `baseline/report.json`, create a PR with the summary for human review. Do not auto-merge. After updating the baseline, subsequent eval_gate comparisons will use the new baseline.
What is the gate verdict logic?
The verdict is determined by a priority order: if any case goes from PASS to FAIL/error, or a new case is FAIL/error, the verdict is FAIL (exit code 1). If any case improves from FAIL/error to PASS, or case count changes (added/removed), verdict is WARN (exit code 0, or 2 in strict mode). If token total increases beyond threshold, it's WARN. If everything matches baseline, verdict is PASS (exit code 0). If no baseline, verdict is N/A (exit code 2).