Research
Yang et al. proposed treating markdown skill files as trainable parameters (SkillOpt, 2026). We built the optimizer and ran it against five production skills across five models. Average improvement: +13.4%. Frontier models showed near-perfect portability. The self-hosted 26B model showed where skills stop transferring. Total cost: under $5.
May 2026 | Dry Ground AI Research
+13.4%
Avg Improvement
across 5 production skills
1.8
Mean Convergence
steps to ceiling
5 × 5
Portability Matrix
Sonnet 4.6, Opus 4.6, GPT-5, Grok 4.3, Gemma 4
24
Total Edits Accepted
across all 5 skills
35%
Acceptance Rate
within 20-40% healthy range
<$1
Cost Per Run
per skill optimization
The Problem
Every agent framework has converged on the same pattern: a markdown file that tells the model how to behave. OpenClaw calls them SKILL.md. Claude Code uses CLAUDE.md. Cursor has .cursorrules. Codex has AGENTS.md. Different names, same mechanism.
These files are functionally the weights of a frozen model system. They determine what the agent sees and how it responds. They're the highest-leverage artifact in any deployment.
And almost everyone writes them by hand, iterates through trial and error, and has no way to measure whether version N+1 is actually better than version N.
SkillOpt changes that. It treats the skill file as a trainable parameter and runs a proper optimization loop: execute tasks, analyze failures, propose bounded edits, validate against a held-out set, accept only strict improvements. The original paper showed it works on benchmarks. We tested whether it works on production skills.
Implementation
We implemented SkillOpt as an OpenClaw agent skill. The optimizer itself is a skill that optimizes other skills. The mechanical parts are handled by five Python scripts. The creative parts (failure analysis, edit proposals) are handled by the agent.
Four atomic operations: append, insert_after, replace, delete. Enforces protected section boundaries. No edit can touch content between SLOW_UPDATE markers. Validates that no two edits target overlapping regions.
The validation gate. Candidate score must be strictly greater than current score. Ties are rejected. This single rule prevents the drift that kills most self-improving systems.
Deterministic scoring via string containment checks. Does the code review mention "SQL injection"? Does the humanized text still contain "leverages"? Binary, fast, reproducible. LLM-as-judge scoring available for qualitative tasks.
Task splitting (2:1:7 train/val/test) with deterministic seeding. Full audit trail with versioned snapshots, gate decision logging, and one-command rollback.
The optimization loop itself is agent-orchestrated. Sub-agents run tasks in parallel, a stronger model (Claude Opus) handles reflection and edit proposals, and the gate script makes the accept/reject decision. Each step produces an auditable JSON log entry with the exact edits proposed, applied, and their impact on validation scores.
Test Subjects
We chose skills that span five distinct task types: text transformation, code analysis, system evaluation, style transfer, and structured output. Each has 50 test tasks with deterministic grading criteria.
Removes AI writing patterns: inflated vocabulary, em dashes, rule-of-three, vague attributions, sycophantic tone. 50 tasks across 11 pattern categories, 6 domains, 5 formats.
Reviews code for SQL injection, XSS, hardcoded secrets, eval, path traversal, N+1 queries, and error handling gaps. 50 tasks across Python, JavaScript, TypeScript, Go, and SQL.
Evaluates system designs for scalability, security, reliability, data design, and API quality. 50 tasks with realistic architecture descriptions and known anti-patterns.
Drafts business emails matching an executive's voice: concise, direct, no corporate buzzwords, no formal sign-offs. 50 tasks across meeting follow-ups, client comms, internal team, and sales contexts. Tests style transfer and personalization.
Extracts structured JSON from unstructured text: contacts, action items, financial data, document metadata. 50 tasks from emails, meeting transcripts, invoices, and LinkedIn bios. Tests precision and format compliance.
Experiment 1
We ran the full SkillOpt loop on each skill with Claude Sonnet 4.6 as the target model and Claude Opus 4.6 as the optimizer. Maximum 4 edits per step, strict validation gating, 4 epochs.
| Skill | Baseline | Final | Improvement | Steps | Edits | Accept Rate |
|---|---|---|---|---|---|---|
| Text Humanizer | 0.980 | 1.000 | +2.0% | 3 / 8 | 12 | 37.5% |
| Code Security Review | 0.900 | 1.000 | +11.1% | 1 / 8 | 2 | 12.5% |
| Architecture Review | 0.579 | 1.000 | +42.1% | 2 / 4 | 5 | 50.0% |
| Email Drafting | 0.947 | 0.987 | +4.2% | 1 / 4 | 1 | 25.0% |
| Data Extraction | 0.929 | 1.000 | +7.6% | 2 / 4 | 4 | 50.0% |
| Mean | 0.867 | 0.997 | +13.4% | 1.8 | 5.8 | 35.0% |
The weakest baseline (Architecture Review, 0.579) showed the largest improvement (+42.1 percentage points). Code Review converged in a single step with just 2 edits. Data Extraction revealed a skill design error: its own normalization rules were destroying expected content, turning "555-0311" into "5550311" and "July 15" into "2025-07-15." The optimizer caught it and fixed it by switching to "preserve as written." After hitting ceiling, the strict gate correctly rejected all subsequent proposals as ties, preventing drift.
The acceptance rate across all experiments was 35%, within the 20-40% range the original paper identifies as healthy. Lower means the gate is working. Higher means it might be too loose.
Experiment 2
We tested whether skills optimized on Claude Sonnet 4.6 transfer to other frontier models. For each skill, we ran 15 held-out test tasks on Claude Opus 4.6, GPT-5, and Grok 4.3.
| Skill | Claude Sonnet 4.6 | Claude Opus 4.6 | GPT-5 | Grok 4.3 | Gemma 4 26B |
|---|---|---|---|---|---|
| Text Humanizer | 1.000 | 1.000 | 1.000 | 0.975 | 0.133 |
| Code Security Review | 1.000 | 1.000 | 1.000 | 0.820 | 0.467 |
| Architecture Review | 1.000 | 1.000 | 1.000 | 0.664 | 0.000 |
| Email Drafting | 1.000 | 1.000 | 1.000 | 0.730 | 0.733 |
| Data Extraction | 0.930 | 1.000 | 1.000 | 0.800 | 0.867 |
Claude Sonnet 4.6, Claude Opus 4.6, and GPT-5 scored 1.0 on 14 of 15 skill-model combinations. The one exception: Data Extraction on Sonnet scored 0.93 due to a date normalization edge case. Among frontier models, procedural knowledge in markdown transfers cleanly.
Grok 4.3 tells a more interesting story. The humanizer skill checks for absence of AI patterns, so there's no strict keyword matching. It transferred at 97.5%. Code review dropped to 82% and architecture review to 66%. Grok identified every vulnerability correctly. The problem: Grok writes "XSS" where our grader expects "cross-site scripting." It says "SSRF" instead of "server-side request forgery." The skills transferred. The scoring couldn't tell.
Gemma 4 26B (a self-hosted 26B open-weight model) tells a different story entirely. Data extraction transferred well (87%) and email drafting held at 73%. But architecture review scored 0% and humanizer scored 13%. These aren't grader artifacts. Gemma couldn't follow the humanizer's vocabulary blacklist, and it missed architectural patterns the skill explicitly listed. Structured output tasks ("extract these fields") transfer down the capability ladder. Complex reasoning tasks ("identify every security anti-pattern in this system design") do not.
The Grok results point to a grader limitation. The Gemma results point to a real capability boundary. Both findings matter: if your grader can't distinguish a terminology preference from a reasoning failure, your portability numbers are noise. And if you're deploying skills to smaller models, test on the actual target before assuming transfer.
Edit Analysis
Across all five skills, accepted edits fell into six categories. The most impactful were meta-instructions and rule corrections.
Adding missing terms to detection or review lists
Example: Humanizer: +8 AI vocabulary terms (nuances, compelling, vanguard, catalyzed...)
Meta-instructions about not reintroducing the same problems during task execution
Example: "When rewriting, do not introduce new instances of the same patterns you are removing"
Fixing skill rules that sound correct but break in practice
Example: Data Extraction: changed "normalize to E.164" to "preserve as written" after normalization destroyed expected values
Adding entire missing categories to review checklists
Example: Arch Review: +tenant isolation, row-level security, distributed rate limiting
Ensuring specific names, dates, and numbers from the input appear in output
Example: Email Drafting: added "include all specific names, company names, and dates from the scenario"
Replacing general guidance with concrete, testable instructions
Example: Code Review: adding Go blank-identifier pattern to error handling section
Key Findings
The humanizer skill started at 0.98 and reached 1.0 after three accepted steps. But in our initial 20-task pilot, both baseline and optimized versions passed every deterministic check identically. The real improvements (vocabulary reintroduction prevention, em dash enforcement) only showed up when we added LLM-as-judge scoring. Task set quality bounds optimization quality. Invest in evaluation before optimization.
The edit budget (max 4 per step) forced high-signal changes. Code Review improved 11.1% with 2 edits. A full rewrite would have been riskier, harder to validate, and impossible to audit. Every accepted change has a rationale, a diff, and a score delta. This matters in production where unexplained behavioral changes are unacceptable.
In text transformation skills, models correctly remove flagged terms but substitute other AI-sounding terms not on the list. "Leverages" becomes "harnesses." "Transformative" becomes "game-changing." The fix: expand the vocabulary list AND add a meta-instruction warning against the pattern itself. The meta-instruction ("do not introduce new instances of the same patterns you are removing") was the single highest-impact edit across all five skills.
Mean convergence: 1.8 steps. After hitting ceiling, the strict gate rejected all subsequent proposals. No drift, no degradation. The 35% acceptance rate means the optimizer proposes aggressively and the gate filters tightly. That's the dynamic you want.
Among frontier models (Sonnet, Opus, GPT-5), transfer is near-perfect. Grok 4.3 scored lower on paper (66-97%), but those gaps were mostly abbreviation preferences. Gemma 4 26B showed real degradation: 0% on architecture review, 13% on humanizing. The pattern is clear. Structured output tasks (data extraction, email drafting) transfer down the capability ladder. Complex reasoning tasks (architectural analysis, vocabulary enforcement) do not. Test on your actual deployment target.
Limitations
The 5-task validation sets saturated at 1.0 fast. Once you're there, the gate can't tell good from excellent. Production deployments need 20+ tasks minimum. Deterministic scoring rewards exact terminology over conceptual understanding, which showed up clearly in the Grok 4.3 results: correct answers scored as failures because of abbreviation preferences. We tested five models (Claude Sonnet 4.6, Claude Opus 4.6, GPT-5, Grok 4.3, Gemma 4 26B). Additional open-weight models like Llama and Mistral would add coverage. The task sets were constructed for this study; production workload distributions may differ.
This is a single-run validation per configuration. Different random seeds would give variance estimates. We've published the implementation and data if you want to reproduce it.
Bottom Line
SkillOpt works on production skills. At under a dollar per skill and 1-3 steps to convergence, you can afford to run it on everything. Every change is auditable: diff, rationale, score delta. The strict gate prevents the drift that kills most self-improving systems. Optimize once and it runs across models.
Start with deterministic scoring criteria. Write 50 test tasks for your most important skill, then run the loop. The biggest gains come from your weakest skills; our Architecture Review jumped 42 percentage points. Once the obvious gaps are closed, move to LLM-as-judge scoring to catch the subtler issues.
The numbers matter less than the mindset shift. Once you have a gate, "I think this skill is better" becomes "I measured it." That's the line between prompt engineering and context engineering.
What This Means for You
SkillOpt handles the optimization side. CompanyClaw's Skills Hub handles the distribution side: sync, curation, install enforcement, and a marketplace that keeps only repo-sourced skills in front of client agents. The two systems were built to work together.
Skills Hub users get access to skills that have already been run through this optimization loop. Every skill in the curated catalog has a version history, a score baseline, and a gate log. "We think this is better" is replaced with "we measured it, here's the diff."
Every Dry Ground AI client gets the same pipeline applied to their deployments. We optimize the skills, validate portability across the models running in their stack, and deliver a full audit trail. The goal is the same one Denise put well when she reviewed this article: the distribution layer and the optimization layer are finally talking to each other. That's the gap most teams are still working around.
We run SkillOpt on client agent deployments as part of our AI solutions practice. Measurable improvement, full audit trail, your data stays on your infrastructure.
Talk to UsWe use cookies to improve your experience. Cookie Policy · Privacy Policy