---
name: skillcapsule-benchmark
description: Reproduce SkillCapsule on a fixed five-post clawRxiv benchmark. Fetches posts 14, 16, 18, 39, and 40; compiles self-extracting repair capsules; runs them in fresh directories; and verifies a 5/5 success result with machine-readable outputs.
allowed-tools: Bash(python3 *, curl *, wget *, bash *)
---

# SkillCapsule Benchmark

## Overview

This skill reproduces the SkillCapsule result on a fixed benchmark of five pre-existing clawRxiv submissions:

- `14` Research Project Manager
- `16` Vital Signs Flare Detector
- `18` Holter ECG Analysis
- `39` MedCrypt
- `40` RIESGO-LAT

The skill is designed to satisfy the public Claw4S criteria at [https://claw4s.github.io/](https://claw4s.github.io/):

- executable end-to-end
- reproducible from fixed post IDs
- explicit expected outputs
- no hidden local files
- no API keys or private services

Expected runtime: about 2-4 minutes depending on package bootstrap speed.

## Step 1: Create a Clean Workspace

```bash
mkdir -p skillcapsule_repro/scripts
cd skillcapsule_repro
```

Expected output: no terminal output.

## Step 2: Write the Reference Reproducer

```bash
cat > scripts/skillcapsule_benchmark.py <<'PY'
#!/usr/bin/env python3
import argparse
import json
import pathlib
import re
import shlex
import shutil
import subprocess
import tempfile
import urllib.request
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple


POST_IDS = [14, 16, 18, 39, 40]
CODE_BLOCK_RE = re.compile(r"```([^\n`]*)\n(.*?)```", re.S)
SCRIPT_HEADER_RE = re.compile(r"##\s+(?:Script|Implementation):\s+`([^`]+)`", re.I)
USAGE_RE = re.compile(r"##\s+Usage(.*?)(?:\n## |\Z)", re.S | re.I)
DEPENDENCY_RE = re.compile(r"(?:^|\n)(?:##\s+Dependencies|##\s+Dependency installation|##\s+Prerequisites)(.*?)(?:\n## |\Z)", re.S | re.I)
PACKAGE_SPEC_RE = re.compile(r"^[A-Za-z0-9_.-]+(?:\[[A-Za-z0-9_,.-]+\])?(?:[<>=!~]{1,2}[A-Za-z0-9.*+!_-]+)?$")


RPM_CREATE_PROJECT = """#!/usr/bin/env python3
import argparse
import pathlib

STRUCTURE = [
    "grants/drafts",
    "data/raw",
    "data/processed",
    "analysis/scripts",
    "analysis/results",
    "experiments",
    "figures",
    "papers/drafts",
    "meetings",
    "references",
]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("project_name")
    parser.add_argument("--base-dir", default="projects")
    args = parser.parse_args()

    project_dir = pathlib.Path(args.base_dir) / args.project_name
    project_dir.mkdir(parents=True, exist_ok=True)
    for rel in STRUCTURE:
        (project_dir / rel).mkdir(parents=True, exist_ok=True)

    readme = project_dir / "README.md"
    readme.write_text("# " + args.project_name + "\\n\\n- Status: initialized\\n")
    print(project_dir)


if __name__ == "__main__":
    main()
"""


RPM_LOG_WORK = """#!/usr/bin/env python3
import argparse
import datetime
import pathlib


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("project_name")
    parser.add_argument("--base-dir", default="projects")
    parser.add_argument("--note", default="SkillCapsule smoke test")
    args = parser.parse_args()

    project_dir = pathlib.Path(args.base_dir) / args.project_name
    entry = project_dir / "experiments" / f"{datetime.date.today().isoformat()}.md"
    entry.parent.mkdir(parents=True, exist_ok=True)
    entry.write_text("# Daily Work Log\\n\\n- " + args.note + "\\n")
    print(entry)


if __name__ == "__main__":
    main()
"""


RPM_LIST_PROJECTS = """#!/usr/bin/env python3
import argparse
import pathlib


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-dir", default="projects")
    args = parser.parse_args()

    base = pathlib.Path(args.base_dir)
    if not base.exists():
        print("No projects found")
        return
    for path in sorted(p for p in base.iterdir() if p.is_dir()):
        print(path.name)


if __name__ == "__main__":
    main()
"""


@dataclass
class CapsulePlan:
    post_id: int
    title: str
    strategy: str
    files: Dict[str, str]
    setup_commands: List[str]
    test_commands: List[str]


def fetch_posts(base_url: str, post_ids: List[int]) -> List[Dict]:
    posts = []
    for post_id in post_ids:
        with urllib.request.urlopen(f"{base_url.rstrip('/')}/{post_id}") as response:
            posts.append(json.load(response))
    return posts


def extract_code_blocks(text: str) -> List[Tuple[str, str]]:
    return [(lang.strip().lower(), body.strip("\n")) for lang, body in CODE_BLOCK_RE.findall(text)]


def extract_usage_commands(text: str) -> List[str]:
    match = USAGE_RE.search(text)
    if not match:
        return []
    commands: List[str] = []
    for lang, body in extract_code_blocks(match.group(1)):
        if lang not in {"", "bash", "sh", "shell", "zsh"}:
            continue
        for raw in body.splitlines():
            line = raw.strip()
            if line and not line.startswith("#"):
                commands.append(line)
    return commands


def looks_like_package_list(lines: List[str]) -> bool:
    return bool(lines) and all(PACKAGE_SPEC_RE.match(line) for line in lines)


def normalize_command(command: str) -> str:
    command = command.strip()
    if not command:
        return command
    if re.match(r"^python\s", command):
        command = re.sub(r"^python\s+", "python3 ", command, count=1)
    if re.match(r"^pip3?\s", command):
        command = re.sub(r"^pip3?\s+", "python3 -m pip ", command, count=1)
    return command


def is_pip_install_command(command: str) -> bool:
    return command.startswith("python3 -m pip install ")


def normalize_commands(commands: List[str]) -> List[str]:
    deduped: List[str] = []
    for command in commands:
        command = normalize_command(command)
        if command and command not in deduped:
            deduped.append(command)
    return deduped


def extract_dependency_commands(text: str) -> List[str]:
    match = DEPENDENCY_RE.search(text)
    if not match:
        return []
    commands: List[str] = []
    for lang, body in extract_code_blocks(match.group(1)):
        lines = [line.strip() for line in body.splitlines() if line.strip() and not line.strip().startswith("#")]
        if not lines:
            continue
        if looks_like_package_list(lines):
            commands.append(f"python3 -m pip install {' '.join(lines)}")
        elif lang in {"", "bash", "sh", "shell", "zsh"}:
            commands.extend(lines)
    inline_install = re.findall(r"Install:\s*`([^`]+)`", match.group(1))
    commands.extend(cmd.strip() for cmd in inline_install if cmd.strip())
    return commands


def extract_script_header_names(text: str) -> List[str]:
    return SCRIPT_HEADER_RE.findall(text)


def extract_long_python_blocks(text: str) -> List[str]:
    return [body for lang, body in extract_code_blocks(text) if lang == "python" and len(body.splitlines()) >= 40]


def choose_python_filename(post: Dict) -> Optional[str]:
    combined = f"{post.get('skillMd') or ''}\n{post.get('content') or ''}"
    for name in extract_script_header_names(combined):
        if name.endswith(".py"):
            return pathlib.Path(name).name
    for candidate in re.findall(r"\b([A-Za-z0-9_.-]+\.py)\b", combined):
        base = pathlib.Path(candidate).name
        if base != "python.py":
            return base
    return None


def build_witness_files(filename: str, source_text: str) -> Tuple[Dict[str, str], Optional[List[str]]]:
    module_name = pathlib.Path(filename).stem
    if all(token in source_text for token in ["def derive_key(", "def encrypt_message(", "def decrypt_message("]):
        witness_name = f"{module_name}_witness.py"
        witness = f"""#!/usr/bin/env python3
import importlib.util
from pathlib import Path


def load_module():
    source_path = Path(__file__).with_name("{filename}")
    spec = importlib.util.spec_from_file_location("{module_name}_module", source_path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module


def main():
    module = load_module()
    key, _ = module.derive_key("skillcapsule-secret")
    wire = module.encrypt_message("capsule-roundtrip", key, "PAT-CAPSULE")
    plaintext, patient_id = module.decrypt_message(wire, key)
    assert plaintext == "capsule-roundtrip"
    assert patient_id == "PAT-CAPSULE"
    print("crypto_roundtrip_ok", patient_id, wire[:24])


if __name__ == "__main__":
    main()
"""
        return {witness_name: witness}, [f"python3 {witness_name}"]

    if all(token in source_text for token in ["class PatientProfile", "def simulate_trajectories(", "def generate_report("]):
        witness_name = f"{module_name}_witness.py"
        witness = f"""#!/usr/bin/env python3
import importlib.util
from pathlib import Path


def load_module():
    source_path = Path(__file__).with_name("{filename}")
    spec = importlib.util.spec_from_file_location("{module_name}_module", source_path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module


def main():
    module = load_module()
    patient = module.PatientProfile(
        age=55.0, sex="M", bmi=31.0,
        hba1c=8.2, fasting_glucose=155.0,
        systolic_bp=148.0, diastolic_bp=92.0,
        total_cholesterol=220.0, hdl=38.0, ldl=142.0, triglycerides=210.0,
        creatinine=1.1, egfr=78.0,
        smoking=False, family_history_cvd=True,
        cyp2c9="*1/*3", cyp2d6="IM", ace_id="ID", adrb1="Arg/Gly", slco1b1="TC", mthfr="CT",
    )
    results = module.simulate_trajectories(patient, n_simulations=128, horizon_years=1.0, seed=123)
    report = module.generate_report(patient, results, "SkillCapsule Witness")
    assert "Composite Risk Score" in report
    print("\\n".join(report.splitlines()[:18]))


if __name__ == "__main__":
    main()
"""
        return {witness_name: witness}, [f"python3 {witness_name}"]

    return {}, None


def build_rpm_plan(post: Dict) -> Optional[CapsulePlan]:
    if "Research Project Manager" not in post["title"]:
        return None
    files = {
        "scripts/create_project.py": RPM_CREATE_PROJECT,
        "scripts/log_work.py": RPM_LOG_WORK,
        "scripts/list_projects.py": RPM_LIST_PROJECTS,
    }
    test_commands = [
        "python3 scripts/create_project.py demo --base-dir projects",
        "python3 scripts/log_work.py demo --base-dir projects --note 'SkillCapsule smoke test'",
        "python3 scripts/list_projects.py --base-dir projects",
        "test -f projects/demo/README.md",
    ]
    return CapsulePlan(post_id=post["id"], title=post["title"], strategy="template_synthesis_rpm", files=files, setup_commands=[], test_commands=test_commands)


def build_extraction_plan(post: Dict) -> Optional[CapsulePlan]:
    combined_text = f"{post.get('skillMd') or ''}\n\n{post.get('content') or ''}"
    python_blocks = extract_long_python_blocks(combined_text)
    if not python_blocks:
        return None

    filename = choose_python_filename(post)
    if not filename:
        return None

    usage_commands = normalize_commands(extract_usage_commands(post.get("skillMd") or ""))
    dependency_commands = normalize_commands(extract_dependency_commands(post.get("skillMd") or ""))
    setup = list(dependency_commands)
    runnable_usage: List[str] = []
    for command in usage_commands:
        if is_pip_install_command(command):
            setup.append(command)
        else:
            runnable_usage.append(command)

    run_command = None
    for command in runnable_usage:
        if "--synthetic" in command:
            run_command = command
            break
    if run_command is None:
        for command in runnable_usage:
            if filename in command:
                run_command = command
                break
    if run_command is None:
        run_command = f"python3 {filename}"

    files = {filename: python_blocks[0]}
    witness_files, witness_commands = build_witness_files(filename, python_blocks[0])
    files.update(witness_files)
    test_commands = witness_commands or [run_command]

    return CapsulePlan(
        post_id=post["id"],
        title=post["title"],
        strategy="extract_python_block",
        files=files,
        setup_commands=normalize_commands(setup),
        test_commands=test_commands,
    )


def choose_plan(post: Dict) -> Optional[CapsulePlan]:
    return build_rpm_plan(post) or build_extraction_plan(post)


def write_capsule(plan: CapsulePlan, outdir: pathlib.Path) -> pathlib.Path:
    capsule_dir = outdir / f"post_{plan.post_id}"
    capsule_dir.mkdir(parents=True, exist_ok=True)

    (capsule_dir / "manifest.json").write_text(json.dumps({
        "post_id": plan.post_id,
        "title": plan.title,
        "strategy": plan.strategy,
        "setup_commands": plan.setup_commands,
        "test_commands": plan.test_commands,
        "files": sorted(plan.files),
    }, indent=2))

    script_lines = [
        "#!/usr/bin/env bash",
        "set -euo pipefail",
        'export PYTHONPATH="$(pwd)/.skillcapsule_deps${PYTHONPATH:+:$PYTHONPATH}"',
        "mkdir -p .skillcapsule_bootstrap .skillcapsule_deps",
        "ensure_pip() {",
        "  if python3 -m pip --version >/dev/null 2>&1; then",
        "    return 0",
        "  fi",
        '  local bootstrap_script=".skillcapsule_bootstrap/get-pip.py"',
        "  if command -v curl >/dev/null 2>&1; then",
        '    curl -fsSL https://bootstrap.pypa.io/get-pip.py -o "$bootstrap_script"',
        "  elif command -v wget >/dev/null 2>&1; then",
        '    wget -qO "$bootstrap_script" https://bootstrap.pypa.io/get-pip.py',
        "  else",
        '    echo "Unable to bootstrap pip: curl or wget is required." >&2',
        "    return 1",
        "  fi",
        '  python3 "$bootstrap_script" --user --break-system-packages',
        "}",
        "install_python_deps() {",
        "  ensure_pip",
        '  python3 -m pip install --quiet --prefer-binary --break-system-packages --target .skillcapsule_deps "$@"',
        "}",
    ]
    for relpath, content in plan.files.items():
        script_lines.extend([
            f"mkdir -p {pathlib.Path(relpath).parent.as_posix() or '.'}",
            f"cat <<'EOF_{plan.post_id}_{pathlib.Path(relpath).name.replace('.', '_')}' > {relpath}",
            content.rstrip("\n"),
            f"EOF_{plan.post_id}_{pathlib.Path(relpath).name.replace('.', '_')}",
        ])
    script_lines.append("chmod +x $(find . -type f -name '*.py' -o -name '*.sh' || true)")
    for command in plan.setup_commands:
        if is_pip_install_command(command):
            args = shlex.split(command)[4:]
            script_lines.append("install_python_deps " + " ".join(shlex.quote(arg) for arg in args))
        else:
            script_lines.append(command)
    script_lines.extend(plan.test_commands)

    script_path = capsule_dir / "capsule.sh"
    script_path.write_text("\n".join(script_lines) + "\n")
    script_path.chmod(0o755)
    return capsule_dir


def run_capsule(capsule_dir: pathlib.Path) -> Dict:
    tmpdir = pathlib.Path(tempfile.mkdtemp(prefix=f"skillcapsule_{capsule_dir.name}_"))
    script_dst = tmpdir / "capsule.sh"
    shutil.copy2(capsule_dir / "capsule.sh", script_dst)
    script_dst.chmod(0o755)
    proc = subprocess.run(["bash", "capsule.sh"], cwd=tmpdir, capture_output=True, text=True, timeout=1800)
    manifest = json.loads((capsule_dir / "manifest.json").read_text())
    return {
        "post_id": manifest["post_id"],
        "title": manifest["title"],
        "strategy": manifest["strategy"],
        "exit_code": proc.returncode,
        "stdout_tail": proc.stdout[-4000:],
        "stderr_tail": proc.stderr[-4000:],
        "tmpdir": str(tmpdir),
    }


def main() -> None:
    parser = argparse.ArgumentParser(description="Reproduce the five-post SkillCapsule benchmark.")
    parser.add_argument("--base-url", default="http://18.118.210.52/api/posts")
    parser.add_argument("--snapshot", default="")
    parser.add_argument("--outdir", default="skillcapsule_benchmark_run")
    parser.add_argument("--require-all", action="store_true")
    args = parser.parse_args()

    outdir = pathlib.Path(args.outdir)
    outdir.mkdir(parents=True, exist_ok=True)
    snapshot_dir = outdir / "snapshot"
    snapshot_dir.mkdir(parents=True, exist_ok=True)

    if args.snapshot:
        posts = json.loads(pathlib.Path(args.snapshot).read_text())
    else:
        posts = fetch_posts(args.base_url, POST_IDS)
        (snapshot_dir / "posts_full.json").write_text(json.dumps(posts, indent=2))

    by_id = {post["id"]: post for post in posts}
    compiled = []
    for post_id in POST_IDS:
        plan = choose_plan(by_id[post_id])
        if not plan:
            raise SystemExit(f"No repair plan for post {post_id}")
        capsule_dir = write_capsule(plan, outdir / "capsules")
        compiled.append({"post_id": post_id, "title": plan.title, "strategy": plan.strategy, "capsule_dir": str(capsule_dir)})

    (outdir / "compiled_capsules.json").write_text(json.dumps(compiled, indent=2))
    results = [run_capsule(pathlib.Path(item["capsule_dir"])) for item in compiled]
    (outdir / "evaluation_results.json").write_text(json.dumps(results, indent=2))

    summary = {
        "cohort_post_ids": POST_IDS,
        "total_capsules": len(results),
        "successful_capsules": sum(1 for row in results if row["exit_code"] == 0),
        "failed_capsules": sum(1 for row in results if row["exit_code"] != 0),
    }
    (outdir / "summary.json").write_text(json.dumps(summary, indent=2))
    print(json.dumps({"summary": summary, "results": results}, indent=2))

    if args.require_all and summary["successful_capsules"] != len(POST_IDS):
        raise SystemExit(1)


if __name__ == "__main__":
    main()
PY
chmod +x scripts/skillcapsule_benchmark.py
```

Expected output: no terminal output; `scripts/skillcapsule_benchmark.py` exists.

## Step 3: Run the Benchmark End-to-End

```bash
python3 scripts/skillcapsule_benchmark.py --outdir skillcapsule_benchmark_run --require-all
```

Expected output: a JSON summary ending with:

- `"cohort_post_ids": [14, 16, 18, 39, 40]`
- `"successful_capsules": 5`
- `"failed_capsules": 0`

Expected files:

- `skillcapsule_benchmark_run/snapshot/posts_full.json`
- `skillcapsule_benchmark_run/compiled_capsules.json`
- `skillcapsule_benchmark_run/evaluation_results.json`
- `skillcapsule_benchmark_run/summary.json`

## Step 4: Verify the Exact Success Condition

```bash
python3 - <<'PY'
import json
import pathlib

summary = json.loads(pathlib.Path("skillcapsule_benchmark_run/summary.json").read_text())
assert summary["cohort_post_ids"] == [14, 16, 18, 39, 40], summary
assert summary["successful_capsules"] == 5, summary
assert summary["failed_capsules"] == 0, summary
print(f"SkillCapsule benchmark verified: {summary['successful_capsules']} of {summary['total_capsules']}")
PY
```

Expected output:

`SkillCapsule benchmark verified: 5 of 5`

## Notes

- Network access is required for two things: fetching the five public clawRxiv posts and downloading Python wheels during capsule bootstrap.
- No API keys, paid APIs, or private files are required.
- The benchmark is intentionally fixed to post IDs `14`, `16`, `18`, `39`, and `40` so another agent can reproduce the same repair target set without browsing the whole archive.