← All skills
eval-loop
92 / 100 · Low risk
Fine to try. Some of what it can do is expected: a data skill usually connects to databases or online services with your credentials.
By malloydata/publisher (GitHub) · data · 132 stars · scanned at commit bb8ce0d
Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files: the set in the model package's evals/ directory, runs in the set's workdir; checkpoints are git commits of the model repo. `eval.py` runs each step from the set's eval.toml. Use to score a model, diagnose failures, improve behin…
Works with any agent that supports SKILL.md skills.
What this skill can do
Runs commands on your computer
This is disclosed by the author in the skill's SKILL.md or README.
Found in 15 places.
Show 12 more
- skills/eval-loop/scripts/agent_harness.py, line 260
except subprocess.TimeoutExpired: - skills/eval-loop/scripts/build_run_package_test.py, line 76
p = subprocess.run( - skills/eval-loop/scripts/build_run_package_test.py, line 166
p = subprocess.run( - skills/eval-loop/scripts/build_run_package_test.py, line 199
p = subprocess.run( - skills/eval-loop/scripts/build_run_package_test.py, line 232
return subprocess.run( - skills/eval-loop/scripts/run_baseline.py, line 321
head = subprocess.run(["git", "-C", str(d), "rev-parse", "HEAD"], - skills/eval-loop/scripts/run_baseline.py, line 328
dirty = subprocess.run(cmd, capture_output=True, text=True, timeout=10) - skills/eval-loop/scripts/run_baseline.py, line 1173
r = subprocess.run(cmd, timeout=a.timeout * 2) - skills/eval-loop/scripts/run_baseline_test.py, line 1594
run = lambda *a: subprocess.run(a, cwd=self.repo, capture_output=True) - skills/eval-loop/scripts/serve.py, line 604
p = subprocess.Popen( - skills/eval-loop/scripts/serve.py, line 646
except (ProcessLookupError, subprocess.TimeoutExpired): - skills/eval-loop/scripts/serve_test.py, line 482
real_popen = serve.subprocess.Popen
Connects to the internet
The author does not mention this. Expected for a data skill.
Found in 25 places.
Show 17 more
- skills/eval-loop/scripts/build_run_package.py, line 618
tail = [f"curl -sS -X DELETE {base}/api/v0/environments/{env}/packages/{name}"] - skills/eval-loop/scripts/build_run_package.py, line 621
f"curl -sS -X POST {base}/api/v0/environments/{env}/packages \\", - skills/eval-loop/scripts/build_run_package.py, line 637
req = urllib.request.Request( - skills/eval-loop/scripts/build_run_package.py, line 641
with urllib.request.urlopen(req, timeout=120): - skills/eval-loop/scripts/build_run_package_test.py, line 286
self.assertIn("curl -sS -X POST http://localhost:4000/api/v0/environments/" - skills/eval-loop/scripts/build_run_package_test.py, line 288
self.assertIn("curl -sS -X DELETE http://localhost:4000/api/v0/" - skills/eval-loop/scripts/run_baseline.py, line 74
import urllib.request - skills/eval-loop/scripts/run_baseline.py, line 848
urllib.request.urlopen(urllib.request.Request(url, method="GET"), - skills/eval-loop/scripts/run_baseline.py, line 917
req = urllib.request.Request( - skills/eval-loop/scripts/run_baseline.py, line 921
with urllib.request.urlopen(req, timeout=timeout) as r: - skills/eval-loop/scripts/run_baseline_test.py, line 2250
return mock.patch.object(rb.urllib.request, "urlopen", - skills/eval-loop/scripts/serve.py, line 100
import urllib.request - skills/eval-loop/scripts/serve.py, line 109
with urllib.request.urlopen(f"http://localhost:{port}/api/v0/projects", - skills/eval-loop/scripts/serve.py, line 234
with urllib.request.urlopen(f"{package_url}/embedding-index", - skills/eval-loop/scripts/serve.py, line 240
with urllib.request.urlopen(package_url, timeout=10) as r: - skills/eval-loop/scripts/serve.py, line 260
req = urllib.request.Request( - skills/eval-loop/scripts/serve.py, line 264
urllib.request.urlopen(req, timeout=timeout).read()
and 5 more places
Install
Paste this into your terminal. It downloads exactly the version we scanned, not whatever is newest.
if [ -e ~/.agents/skills/'eval-loop' ] || [ -L ~/.agents/skills/'eval-loop' ]; then echo 'A skill folder named eval-loop already exists — not replacing it' >&2; false; else d=$(mktemp -d) && git clone --quiet --filter=blob:none --no-checkout 'https://github.com/malloydata/publisher' "$d" && git -C "$d" checkout --quiet 'bb8ce0db61c90d45273e4b65c5ccfc6475a9193a' -- 'skills/eval-loop' && mkdir -p ~/.agents/skills && cp -R "$d"/'skills/eval-loop' ~/.agents/skills/'eval-loop' && rm -rf "$d"; fi
Codex reads ~/.agents/skills directly. For Claude Code, Cursor, Copilot, Gemini CLI, Antigravity or OpenCode, also link it. For Claude Code: ln -s ~/.agents/skills/'eval-loop' ~/.claude/skills/'eval-loop' (other agents: ~/.cursor/skills, ~/.copilot/skills, ~/.gemini/skills, ~/.gemini/config/skills, ~/.config/opencode/skills). Or install through the finder skill, which does both.
Badge for your README
The badge for malloydata/publisher shows the score of its lowest-scoring skill.
[](https://zalavijaysinh.github.io/skilltrust/repos/malloydata/publisher/)
Score history
| Date | Version | Score |
|---|
| 2026-11-14 | bb8ce0d | 92 |