dsh-vision-primitives DSH plugin

dsh-vision-primitives is a native DSH plugin that gives text-only agents precise visual reasoning capabilities. It uses Set-of-Mark numbered grids, deterministic pixel coordinate math, and a set of tools (capture, zoom, annotate, measure, diff, color, OCR) to bridge visual perception with exact pixel coordinates. Optionally integrates with MiMo V2.5 multimodal model for description and visual grounding.

vision developer-tools

How to install the dsh-vision-primitives DSH plugin

dsh plugin --profile <name> add github:zouyuanqing/dsh-vision-primitives

This source command needs manual review. Copying does not run it.

dsh-vision-primitives DSH plugin data source

dsh-vision-primitives DSH plugin snapshot date: Aug 16, 2026

discovered

What the dsh-vision-primitives DSH plugin can do

Vision_capture: Screenshot (full/region, multi-monitor) or read workspace PNG into session frames.
Vision_grid: Overlay Set-of-Mark numbered grid, return numbered image and precise box/center per cell.
Vision_zoom: Lossless nearest-neighbor zoom with coordinate mapping chain preserved.
Vision_analyze: Structured output (caption, layout, elements) with normalized/pixel boxes, SOM cell numbers, and screen coordinates.
Vision_ocr: Windows native OCR returning word text boxes with frame/screen coordinates.
Vision_diff: Frame differencing for deterministic change detection (bbox, ratio, highlight image).

Where the dsh-vision-primitives DSH plugin fits

Enable a text-only LLM to visually inspect a UI screenshot and click on a specific button by resolving grid coordinates.
Automate desktop testing: capture screen, detect changes via diff, and validate pixel positions of UI elements.
Provide structured visual evidence (caption, layout, elements) to a reasoning agent for multi-step visual tasks.
Use OCR to extract text from a screenshot region and then measure distances between detected elements.
Combine vision_capture, vision_grid, and vision_zoom to precisely locate and annotate a target in a high-resolution image.

Who the dsh-vision-primitives DSH plugin is for

DSH users who want to add visual reasoning capabilities to their agents without external MCP servers.
Developers building automated visual testing, screen scraping, or GUI automation workflows on Windows.
Users of MiMo multimodal model who need a seamless integration with DSH for visual grounding.

dsh-vision-primitives DSH plugin limitations

Screen capture and native OCR are currently Windows-only (PowerShell and WinRT APIs).
Frame files are stored under sandboxPolicy.workspaceRoot/.vispri.
MiMo tools (vision_describe, vision_locate, vision_analyze) require an API key (MIMO_API_KEY) and internet access.
Dynamic plugin sandbox (cordis_define) cannot activate the client half, so WebUI config card is only available in bundle installation.

dsh-vision-primitives DSH plugin: from the repository README

Quoted from the zouyuanqing/dsh-vision-primitives README, the upstream source of the dsh-vision-primitives DSH plugin. Copyright remains with the original authors.

**Native interactive visual-reasoning plugin for DeepSeek Harness (DSH).** 给纯文本智能体装上"精确的眼睛":以 **Set-of-Mark 编号网格 + 确定性像素坐标数学** 为核心,让 Harness 智能体对屏幕/图片做精确到像素的视觉交互推理 —— 全程零外部 MCP 服务器,视觉推理内核 100% 在 DSH Host 运行时内以纯 JS 执行。 Design inspired by [vision-primitives-mcp](https://github.com/zouyuanqing/vision-primitives-mcp), re-implemented as a **native DSH plugin** (official profile bundle: host half + WebUI client half). ## 特性 | | | |---|---| | 🧠 **智能体即视觉模型** | 插件产出图片路径 + 确定性坐标数学;Harness 多模态智能体(或内置 MiMo 后端)看图决策,插件把"模糊感知"换算成"精确像素" | | 🎯 **SOM 编号网格** | `vision_grid` 叠加编号网格 → `vision_resolve(cell)` 得格子中心精确坐标,消除视觉模型坐标误差 | | 🔍 **局部无损放大** | `vision_zoom` 最近邻放大(像素级保真),保留到原帧的坐标映射链 | | 📐 **几何验证** | `vision_annotate` / `vision_measure` / `vision_diff` / `vision_find_color` / `vision_ocr` 确定性验证 | | 🖥️ **MiMo V2.5 后端** | `vision_describe` / `vision_locate`(多模态理解 + 视觉定位),并注册 `mimo` 模型路由(LlmAdapter,流式/函数调用/图像输入全支持) | | 📋 **聊天框贴图(paste-to-path)** | 纯文本模型下聊天框粘贴图片 → 自动转为"文件路径 + 视觉证据"文本注入(社区 paste-to-path 方案原生实现,默认关闭);视觉模型保持原生图片附件不受影响 | | 🔁 **发送时图片桥接** | 默认开启:纯文本模型也能粘贴/拖动图片(原生缩略图,无"模型不支持图片"报错);发送时图片缓存为工作区文件,以 `[Attached image: 路径]` 文本交给模型,模型可按需 `read_image` / `vision_analyze` —— 灵感来自社区 [dsh-image-

Read the full READMERepository license: MIT

dsh-vision-primitives DSH plugin questions

How do I install dsh-vision-primitives?

Run `dsh plugin --profile <name> add github:zouyuanqing/dsh-vision-primitives` in your DSH terminal. You can also pin a specific commit with `#<commit-sha>`. After installation, restart DSH and the plugin will register vision_* tools automatically.

Do I need an API key to use all features?

No. The core visual primitives (grid, zoom, annotate, measure, diff, color, OCR) work completely offline without any API key. Only the MiMo-powered tools (vision_describe, vision_locate, vision_analyze) require a MIMO_API_KEY. You can set it via `dsh credentials set MIMO_API_KEY <your-key>` or in the WebUI configuration card.

Does it work on macOS or Linux?

Currently, screen capture and native OCR are Windows-only due to dependencies on PowerShell and WinRT APIs. Other features that work on existing image files (like vision_grid, vision_zoom, vision_analyze) may work on other platforms if the image is provided by another source, but the plugin is primarily designed for Windows.

How do I enable paste-to-path for text-only models?

Paste-to-path is disabled by default. To enable it, go to Settings → Plugin Config → Vision Primitives and set `pasteToPath` to true. You can also configure it via CLI or cordis.patch.yml. When enabled, pasting an image into the chat will automatically insert its file path and a MiMo summary.

Can I use this plugin with a multimodal model instead of MiMo?

The plugin registers a `mimo` model route that works with Xiaomi's MiMo V2.5 API. If you have a different multimodal model, you can potentially use the vision_analyze tool with a custom integration, but the built-in multimodal features are tied to MiMo. The core visual primitives remain model-agnostic.