Skill Authoring + Automatic Skill/Prompt Self-Evolution — Code-Level Field Guide

The following are examples of different task inputs provided to the assistant along with the assistant's response for each of them, and some feedback on how the assistant's response could be better:

<side_info>

Your task is to write a new instruction for the assistant.

Read the inputs carefully and identify the input format and infer detailed task description about the task I wish to solve with the assistant.

Read all the assistant responses and the corresponding feedback. Identify all niche and domain specific factual information about the task and include it in the instruction, as a lot of it may not be available to the assistant in the future. The assistant may have utilized a generalizable strategy to solve the task, if so, include that in the instruction as well.

Provide the new instructions within ``` blocks."""

`<curr_param>` = the current instruction; `<side_info>` = a markdown-rendered "reflective dataset" of (input, response, feedback) triples (rendered by `format_samples`, `instruction_proposal.py:54-95`). The output is extracted from the last ```` ``` ```` block (`output_extractor`, `instruction_proposal.py:124-153`).

**Why it matters:** This is the entire "reflection" primitive in ~15 lines. The magic instruction is *"Identify all niche and domain-specific factual information... include it in the instruction, as a lot of it may not be available to the assistant in the future"* — it tells the reflector to **bake discovered facts into the prompt as durable memory**, plus *"if there's a generalizable strategy, include that too."* It converts per-episode feedback into permanent, transferable instruction text.

### Pattern 2.2 — Reflection runs on the *reflective dataset* built from traces, not raw scores
File: `gepa/src/gepa/proposer/reflective_mutation/reflective_mutation.py:333-371`

```python
# 2) Decide which components to update (lock protects RoundRobin state mutation)
with self._lock:
    predictor_names_to_update = self.module_selector(
        state, eval_curr.trajectories, eval_curr.scores, ctx.curr_prog_id, ctx.curr_prog
    )

# 3) Build reflective dataset and propose new content
reflective_dataset = self.adapter.make_reflective_dataset(ctx.curr_prog, eval_curr, predictor_names_to_update)
...
new_texts, prompts, raw_lm_outputs = self.propose_new_texts(
    ctx.curr_prog, reflective_dataset, predictor_names_to_update
)

make_reflective_dataset is an adapter responsibility — each integration (DSPy, RAG, terminal-bench, MCP) decides how to turn a rollout trajectory + score into the (input, output, feedback) records the reflector reads. The feedback is textual, not a scalar.

Why it matters: The separation is the reusable idea — a generic reflection engine + a per-domain "how do I describe what went wrong in words" adapter. Your self-improvement loop's equivalent of make_reflective_dataset is "how do I summarize this week's failures into feedback the reflector can act on."

Pattern 2.3 — Minibatch gate before expensive full eval (cheap accept/reject, then verify)

File: gepa/src/gepa/proposer/reflective_mutation/reflective_mutation.py:400-445 (child evaluated on the same minibatch as parent) and gepa/src/gepa/core/engine.py:287-334:

def _accept_reflective_proposal(self, proposal, iteration, state) -> bool:
    old_sum = sum(proposal.subsample_scores_before or [])
    new_sum = sum(proposal.subsample_scores_after or [])
    ...
    if not self.acceptance_criterion.should_accept(proposal, state):
        ... # reject, no full eval
        return False
    ...
    new_idx, _ = self._run_full_eval_and_add(   # only NOW pay for the full valset
        new_program=proposal.candidate, state=state,
        parent_program_idx=proposal.parent_program_ids,
    )

Default acceptance is StrictImprovementAcceptance (engine.py:124). There's also a skip_perfect_score short-circuit (reflective_mutation.py:308-327) — if every minibatch score is already perfect, skip the mutation entirely.

Why it matters: Two-stage evaluation is the budget discipline: reflect+re-run on a tiny subsample first, and only spend a full evaluation on children that already look better. Directly maps to "don't run the full weekly benchmark on every candidate edit — gate on a cheap smoke set first."

Pattern 2.4 — Pareto frontier selection: sample parents ∝ how many instances they're best-at

File: gepa/src/gepa/gepa_utils.py:90-117 (select_program_candidate_from_pareto_front)

def select_program_candidate_from_pareto_front(pareto_front_programs, train_val_weighted_agg_scores_for_all_programs, rng) -> int:
    new_program_at_pareto_front_valset = remove_dominated_programs(
        pareto_front_programs, scores=train_val_weighted_agg_scores_for_all_programs
    )
    program_frequency_in_validation_pareto_front = {}
    for testcase_pareto_front in new_program_at_pareto_front_valset.values():
        for prog_idx in testcase_pareto_front:
            program_frequency_in_validation_pareto_front[prog_idx] = \
                program_frequency_in_validation_pareto_front.get(prog_idx, 0) + 1

    sampling_list = [
        prog_idx
        for prog_idx, freq in program_frequency_in_validation_pareto_front.items()
        for _ in range(freq)   # weight by how many test cases this program is non-dominated on
    ]
    curr_prog_id = rng.choice(sampling_list)
    return curr_prog_id

Selector classes wrap this (gepa/src/gepa/strategies/candidate_selector.py): ParetoCandidateSelector (default), CurrentBestCandidateSelector (greedy idxmax), EpsilonGreedyCandidateSelector, TopKParetoCandidateSelector (Pareto restricted to top-K by aggregate).

Why it matters: This is GEPA's signature move and the reason it beats greedy hill-climbing. Instead of always mutating the single highest-average candidate (which collapses diversity and gets stuck), it keeps every candidate that is best on at least one instance, and samples parents proportional to how many instances they dominate. A specialist that wins only 3 hard cases survives and gets bred. This is the exact upgrade a weekly self-improvement loop needs: keep a frontier, not a champion.

Pattern 2.5 — Merge/crossover: recombine two candidates via a common ancestor

File: gepa/src/gepa/proposer/merge.py:27-115 + scheduling in engine.py:664-740.

Merge finds a pair (i, j) with a common ancestor where each child changed a different predictor relative to the ancestor (does_triplet_have_desirable_predictors, merge.py:27-43), and only where the ancestor doesn't already outscore both (filter_ancestors, merge.py:46-66). Ancestor is chosen weighted by aggregate score (merge.py:108-112). A merged candidate is accepted only if new_sum >= max(parent_sums) (engine.py:688).

Why it matters: Genetic crossover for text: if candidate A improved component X and candidate B improved component Y (from the same parent), merge them to get both improvements. Scheduled to fire only after a reflective mutation just found a new program (engine.py:666), and capped by max_merge_invocations (engine.py:372).

Pattern 2.6 — Budget is a pluggable stop-condition on metric calls, tracked live

File: gepa/src/gepa/core/engine.py:894-920 + 894 _should_stop. The loop runs while not self._should_stop(state) and the budget is metric calls (rollouts), surfaced through MaxMetricCallsStopper / CompositeStopper. Every eval increments state.total_num_evals and fires an on_budget_updated callback (engine.py:607-620).

Why it matters: Hard caps are first-class and expressed in the unit that actually costs money (rollouts), not wall-clock or iterations. Matches your "every unattended run carries a hard cap" doctrine.

3. sentient-agi/EvoSkill — automatic skill discovery from agent failures

EvoSkill (README.md:18) "significantly extends the feedback-driven idea of GEPA from single-file optimization to complete agent evolution. Instead of only revising one prompt in place like GEPA, EvoSkill proposes multiple skill and prompt mutations jointly, evaluates new variants on held-out data, and has each iteration produce an entirely new agent program." It targets Claude Code, Codex CLI, OpenCode, Goose, OpenHands, etc. Each agent "program" is a git branch (program/<name>).

The loop at a glance

File: EvoSkill/src/loop/runner.py, SelfImprovingLoop.run() (core body ~lines 263-404):

Select parent from frontier → git checkout that program → round-robin sample a batch of (question, category) across categories → run the base agent concurrently on all → score each vs. ground truth (threshold 0.8) → collect failures → if any failures, call the proposer over the whole batch of failures → materialize a mutation (new/edited SKILL.md or a prompt change) into a new git branch → evaluate the branch on held-out val_data → admit to frontier if it beats the worst frontier member → else discard the branch → append a feedback-history entry (with outcome) that future proposer calls read.

# EvoSkill/src/loop/runner.py  (core iteration, abridged from lines 263-404)
for i in range(self.config.max_iterations):
    parent = self._select_parent(iteration_count)
    self.manager.switch_to(parent)                       # git checkout program/<parent>
    ...
    traces = await asyncio.gather(*[
        self.agents.base.run(question) for question, _, _ in test_samples
    ])
    failures = []
    for trace, (question, answer, category) in zip(traces, test_samples):
        agent_answer = trace.output.final_answer if trace.output and trace.output.final_answer else "[PARSE FAILED]"
        avg_score = self.scorer(question, agent_answer.strip().lower(), answer.strip().lower())
        if avg_score < 0.8:
            failures.append((trace, agent_answer, answer, category))
    if len(failures) == 0:
        continue                                          # nothing to learn this iter
    mutation_result = await self._mutate_with_fallback(parent, failures, actual_iteration)
    if mutation_result is not None:
        child_name, proposal, justification = mutation_result
        child_score = await self._evaluate(self.val_data)  # held-out eval
        added = self.manager.update_frontier(child_name, child_score, max_size=self.config.frontier_size)
        outcome = ("improved" if child_score > parent_score else "kept") if added else "discarded"
        if not added:
            self.manager.discard(child_name)
        append_feedback(self._feedback_path, child_name, proposal, justification,
                        outcome=outcome, score=child_score, parent_score=parent_score, ...)
    if no_improvement_count >= self.config.no_improvement_limit:
        break

Pattern 3.1 — The skill-proposer prompt: turn a batch of failures into a create-or-edit skill decision

File: EvoSkill/src/agent_profiles/skill_proposer/prompt.py (SKILL_PROPOSER_SYSTEM_PROMPT)

You are an expert agent performance analyst specializing in identifying opportunities to enhance agent capabilities through skill additions or modifications...

## Your Task
Given an agent's execution trace, its answer, and the ground truth answer, propose either:
- A **new skill** (action="create") if no existing skill covers the capability gap
- An **edit to an existing skill** (action="edit") if an existing skill SHOULD have prevented the failure but didn't
Your proposal will be passed to a downstream Skill Builder agent for implementation.

Anti-patterns baked into the prompt (same file):

## Anti-Patterns to Avoid
- DON'T propose a new skill if an existing one covers similar ground → propose an EDIT instead
- DON'T ignore previous DISCARDED proposals for the same problem → explain how yours differs
- DON'T create narrow skills that only fix one specific failure → ensure broad applicability
- DON'T propose capabilities that overlap with existing skills → consolidate instead

## When to Propose Skills
Propose a skill when ANY of these apply:
- Agent lacks access to information, APIs, or computational capabilities
- The fix requires a multi-step procedure (>3 sequential steps)
- The fix involves output structuring, formatting, or templates
- The improvement would be reusable across different tasks
- The issue is about WHAT steps to take, not HOW to think

The proposer runs with real tools ["Read","Bash","Glob","Grep","WebFetch","WebSearch","TodoWrite","BashOutput"] (EvoSkill/src/agent_profiles/skill_proposer/skill_proposer.py:10-19) — so it can literally Read existing .claude/skills/*/SKILL.md and grep the feedback history before deciding create vs edit. Its query is assembled by build_proposer_query (EvoSkill/src/loop/helpers.py:16-112), which injects the existing-skills list, the truncated feedback history, and per-failure trace summaries.

Why it matters: This is the most directly stealable artifact for a self-improvement loop. It is a disciplined skill-writer: it defends against skill sprawl (edit > create), against re-proposing discarded ideas, and against overfitting to one failure ("ensure broad applicability"). The decisive routing rule — "skill = WHAT steps to take; prompt = HOW to think" — is the cleanest heuristic anywhere in these three repos for deciding whether a fix belongs in a skill or in the system prompt.

Pattern 3.2 — The prompt-vs-skill routing decision, as a schema

Files: EvoSkill/src/schemas/proposer.py, skill_proposer.py, prompt_proposer.py

# src/schemas/proposer.py  — the generic (joint) proposer
class ProposerResponse(BaseModel):
    optimize_prompt_or_skill: Literal["prompt", "skill"]
    proposed_skill_or_prompt: str
    justification: str

# src/schemas/skill_proposer.py
class SkillProposerResponse(BaseModel):
    action: Literal["create", "edit"] = "create"
    target_skill: str | None = None          # required if action == "edit"
    proposed_skill: str
    justification: str
    related_iterations: list[str] = Field(default_factory=list)

    @model_validator(mode="after")
    def validate_edit_target(self):
        if self.action == "edit" and not self.target_skill:
            raise ValueError("target_skill is required when action='edit'")
        return self

The generic proposer decides optimize_prompt_or_skill first, using the rule "If the fix requires defining WHAT steps to take, choose skill. If it's about HOW to think or general mindset, choose prompt." Notably no mutation carries a numeric strength/temperature — mutations are pure natural-language descriptions handed to a downstream generator agent that writes the file.

Why it matters: Structured output forces the reflector to commit to a typed decision (create/edit, prompt/skill, which target) with a validator, rather than emitting freeform prose you then have to parse. related_iterations explicitly links a proposal to prior discarded attempts — the loop has memory of its own dead ends.

Pattern 3.3 — Two-stage propose → generate: a separate agent writes the actual SKILL.md

File: EvoSkill/src/agent_profiles/skill_generator/prompt.py

You implement exactly one repo-local skill for OpenCode.
## Goal
Take the proposed skill description and write or edit one project-local skill file at:
`.claude/skills/<skill-name>/SKILL.md`
Use the write/edit tools directly...

## Required File Format
Every `SKILL.md` must begin with YAML frontmatter.
Required fields:
- `name`
- `description`
Optional field:
- `compatibility: opencode`
The `name` value must: match the directory name exactly; be lowercase; use hyphens only...

## Body Requirements
- Keep the skill concise and specific.
- Include the reusable rule the agent should follow.
- Include 1-3 short examples when they help.
- If editing an existing skill, preserve relevant content and improve it instead of replacing it blindly.

The generator runs with ["Read","Write","Bash","Glob","Grep","Edit","WebFetch","WebSearch","TodoWrite","BashOutput","Skill"] and permission_mode="acceptEdits" (EvoSkill/src/agent_profiles/skill_generator/skill_generator.py:16-43) — it actually creates/edits the file autonomously. The caller (runner.py:_mutate) diffs _get_active_skills() before/after to detect which skill name appeared, then commits the whole .claude/ tree to the new git branch (ProgramManager.commit, EvoSkill/src/registry/manager.py:432-460).

Why it matters: Clean separation of diagnosis (proposer: what capability gap, create vs edit) from authoring (generator: write a well-formed SKILL.md following a strict format). The generator prompt is itself a compact skill-authoring style guide (name==dir, lowercase, hyphens; concise; 1-3 examples; edit-preserving). Same "boil the ocean but don't sprawl" ethos as your CLAUDE.md.

Pattern 3.4 — Frontier as a git-tagged top-K leaderboard (NOT true GEPA Pareto)

File: EvoSkill/src/registry/manager.py:378-430

def update_frontier(self, name: str, score: float, max_size: int = 5) -> bool:
    scored = self.get_frontier_with_scores()
    if len(scored) < max_size:            # room → admit unconditionally
        self.mark_frontier(name); return True
    worst_name, worst_score = scored[-1]
    if score > worst_score:               # else beat the worst
        self.unmark_frontier(worst_name); self.mark_frontier(name); return True
    return False

Parent selection supports "best" (default greedy), "random", "round_robin" (select_from_frontier, manager.py:346-364). frontier_size default 3 (src/loop/config.py:32). Scoring is a single scalar validation accuracy (_evaluate, runner.py:446-472).

Why it matters — and the caveat: Despite the "extends GEPA" framing, the code implements a top-K scalar leaderboard, not GEPA's per-instance non-dominated Pareto set. [UNVERIFIED — inferred from reading the code vs README] the multi-objective Pareto claim is aspirational relative to what ships. If you want the diversity benefit, take EvoSkill's skill-authoring machinery but keep GEPA's actual Pareto selector (Pattern 2.4).

Pattern 3.5 — feedback_descent.py is a documented reference implementation, not wired into the loop

File: EvoSkill/src/feedback_descent.py:89-132

def run(self, problem):
    best = self.proposer.generate_initial(problem)
    feedback_history = []
    for iteration in range(self.max_iterations):
        candidate = self.proposer.propose(best, feedback_history)   # conditioned on prior rationales
        result = self.evaluator.evaluate(best, candidate)           # PAIRWISE comparison
        feedback_history.append(FeedbackEntry(candidate, result.rationale))
        if result.preference_for_candidate:
            best = candidate; feedback_history = []                 # win → reset history
        else:
            no_improvement_count += 1
            if no_improvement_count >= self.no_improvement_limit: break

[UNVERIFIED — grep of src/ shows FeedbackDescent is only re-exported in src/__init__.py; SelfImprovingLoop never imports it.] It's a standalone Protocol-based abstraction (cited to arXiv 2511.07919) illustrating pairwise proposal (candidate vs. current best, rationale accumulates on loss, resets on win). The shipping loop uses scalar validation accuracy + a markdown feedback-history file (append_feedback/read_feedback_history, helpers.py:151-215) instead of pairwise LLM judging.

Why it matters: The pairwise-comparison + rationale-accumulation idea is a real alternative to scalar scoring when you have no ground-truth metric — the evaluator returns "which is better + why," and the "why" feeds the next proposal. Useful for a self-improvement loop over subjective artifacts (docs, taste) where there's no numeric reward.

4. Skill-authoring styles compared — Google vs Anthropic/Pocock vs EvoSkill-generated

Grounding for the Anthropic/Pocock column comes from real skills on disk: ~/.claude/skills/extract-approach/SKILL.md and ~/.claude/skills/loop-engineering/SKILL.md.

Net: Google optimizes a skill as a freshness-authoritative knowledge patch with a severity DSL and external docs; Anthropic/Pocock optimize the description as a retrieval/routing surface (positive + negative triggers) and disclose via sibling files; EvoSkill's generator enforces a minimal, single-rule, edit-preserving format because a machine writes it under budget. All three converge on name+description frontmatter and "keep the body concise, push detail to references." The biggest transferable gap: Google's alert-block severity DSL and Anthropic's explicit NOT for negative triggers are each missing from the other — a top-tier skill should use both.

5. Top ~6 highest-ticket steals (esp. for a weekly self-improvement / L11 hill-climb loop)

1. GEPA Pareto parent-selection — sample candidates ∝ instances-they-win, not the single best. gepa/src/gepa/gepa_utils.py:90-117 + strategies/candidate_selector.py. Your loop almost certainly hill-climbs a single "champion" config. Switch to keeping a frontier of skill/prompt variants, each retained if it's best on ≥1 eval case, and breed parents weighted by coverage. This is the one change that prevents diversity collapse and local optima. Steal the ~25-line function directly.

2. EvoSkill's skill-proposer prompt + the "WHAT vs HOW" routing rule. EvoSkill/src/agent_profiles/skill_proposer/prompt.py. Drop this near-verbatim into your weekly loop's "what did I learn, should it become a skill?" step. It hard-codes: edit > create, don't re-propose discarded ideas, ensure broad applicability, and "skill = WHAT steps; prompt = HOW to think." That rubric is exactly the missing governance for a loop that would otherwise spawn a new skill per incident.

3. GEPA's reflection prompt — bake discovered facts into the artifact as durable memory. gepa/src/gepa/strategies/instruction_proposal.py:13-29. The instruction "identify all niche, domain-specific factual information... include it in the instruction, as a lot of it may not be available to the assistant in the future, and include any generalizable strategy" is the precise wording that turns transient feedback into permanent skill/prompt text. Use it as your reflector's system prompt when rewriting a skill from the week's failures.

4. Two-stage evaluation gate (cheap minibatch accept → then full eval). gepa/src/gepa/core/engine.py:287-334 + reflective_mutation.py:400-445. Never run the full benchmark on every candidate: reflect + re-run on a tiny subsample, promote to full eval only if it already beats the parent, and short-circuit if the subsample is already perfect. Direct budget win for an unattended run with a hard metric-call cap.

5. Propose → Generate separation with a strict SKILL.md format guide, run under acceptEdits. EvoSkill/src/agent_profiles/skill_generator/prompt.py + skill_generator.py:16-43. Split "diagnose the capability gap (typed decision)" from "author a well-formed SKILL.md (name==dir, lowercase-hyphen, concise, 1-3 examples, edit-preserving)". Detect the new skill by diffing the skills dir before/after, then commit to a branch. This is a ready-made autonomous skill-writer for your loop.

6. Google's severity DSL + external progressive disclosure for the skills you hand-write. gemini-skills/skills/gemini-api-dev/SKILL.md:8-37,128-158. Adopt [!IMPORTANT]/[!WARNING]/[!CAUTION] to grade load-bearing lines, front-load literal tool/package names into the description, and delegate the exhaustive surface to MCP/llms.txt-style fetched docs (or repo-local references/) so the SKILL.md stays a small authoritative core. Combine with Anthropic-style NOT for negative triggers in the description for best-in-class routing.

Provenance / verification notes

• All Google + GEPA snippets read directly from the cloned files at the cited line numbers.
• EvoSkill loop/prompt/schema snippets: some line numbers (runner.py:263-404, helpers.py, manager.py) are as reported by a code-extraction pass over the same clone; the two [UNVERIFIED] tags flag (a) EvoSkill's Pareto claim being a scalar leaderboard in code, and (b) feedback_descent.py being unwired from the shipping loop. Both are reproducible via grep -rn "FeedbackDescent" ~/Developer/reference-repos/EvoSkill/src and reading registry/manager.py:378-430.