Overnight agent runs with budgets and a morning report

# AGENTS.md: unattended overnight agent runs

Agents in this repo may run unattended (headless, overnight, in CI). Unattended work follows these rules, because
nobody is there to stop a loop or answer a question.

## Every unattended task
- Has a **finish line**: one command that exits 0 when the task is done. Run it yourself before you stop.
- Runs in its **own git worktree on its own branch** (`night/<date>-<task>`), never in the main working tree.
- Has a **dollar cap** (`--max-budget-usd`) and a **turn cap** (`--max-turns`), inside a total for the night.
- Commits its work to its branch, then ends with one line: done, or what is blocking.

## Never, when unattended
- Push, merge, open or close pull requests, delete branches, or touch other worktrees.
- Edit, skip or delete tests to reach a finish line.
- Change budgets, permissions, CI config or settings files.
- Use secrets, production systems or paid services the task did not name.
- Wait for a human: if a decision is needed, commit what you have and say what the decision is.

## Running a batch
Use the `night-shift` skill: it creates the worktrees, sets the budgets and Claude Code's unattended retry settings,
runs each task, checks its finish line, and writes `.night-shift/report.md`. A human reviews every branch before merging.

# CLAUDE.md

@AGENTS.md

## Claude Code notes

- Skills from this loadout live in `.claude/skills/`: `night-shift`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From SetupBay "Overnight agent runs with budgets and a morning report" v1.0.0 — https://setupbay.com/loadouts/overnight-agent-runs/ -->

# GEMINI.md

@./AGENTS.md

## Gemini CLI notes

- Skills from this loadout live in `.agents/skills/`: `night-shift`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From SetupBay "Overnight agent runs with budgets and a morning report" v1.0.0 — https://setupbay.com/loadouts/overnight-agent-runs/ -->

---
name: night-shift
description: Queue coding tasks to run unattended overnight, each in its own git worktree with a dollar cap and a finish-line check, then read the morning report. Use when the user wants agents to work while they sleep, run a batch of tasks unattended, or asks what the overnight run did.
---

# Night shift

Unattended runs fail in two expensive ways: one stuck task eats the whole budget, or a usage limit leaves a request
hanging until morning with nothing to show. This skill gives every task a **finish line** (a command that exits 0
when it is done), its own worktree and branch, its own dollar cap inside a total for the night, and Claude Code's
unattended retry settings. Nothing is pushed or merged; the human reviews branches in the morning.

## 1. Write the plan with the human
Copy `references/night-shift.example.json` to `night-shift.json` at the repo root and fill it in together:
- **One task = one finish line.** "Make `pytest -q tests/test_dates.py` pass" is a task. "Improve the codebase" is not.
  If you cannot name the check command, the task is not ready for an unattended run.
- `budget_usd` per task and `budget_usd_total` for the night. The script never lets a task start with less than
  $0.50 left and passes Claude Code the smaller of the task budget and what is left as `--max-budget-usd`.
- Permissions: headless runs cannot answer prompts, so anything not allowed is refused. Allow only what the tasks
  need (`--allowedTools "Bash(pytest *)"`, `--permission-mode acceptEdits`). Never bypass permission checks
  (bypassPermissions mode or the "dangerously" flag) outside a container or VM; the script refuses both unless
  `NIGHT_SHIFT_SANDBOXED=1`.
- Keep `--max-turns` (it ends a run that loops) and `--output-format json` (the script reads the cost from it).

## 2. Start it
```bash
python3 .agents/skills/night-shift/scripts/night_shift.py run
```
(Claude Code installs the skill under `.claude/skills/`.) The main working tree must be committed first: each task
branches from `HEAD`. The script sets these unless they are already set:

| Variable | Value | Why |
|---|---|---|
| `CLAUDE_CODE_RETRY_WATCHDOG` | `1` | keep retrying 429 and 529 errors instead of failing the task |
| `CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS` | `1800000` | but end one request after 30 minutes of waiting (v2.1.295+) |
| `CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS` | `3600000` | end a request whose usage limit resets more than an hour away (v2.1.301+) |

The watchdog already fails at once on spend-limit and out-of-credits errors (v2.1.239+), so a hard limit stops the run.
For Codex, use `codex exec --json` as the agent and keep the budget in the spend-cap guard from the
`agent-spend-cap` loadout; Codex has no budget flag.

## 3. Read the morning report
`.night-shift/report.md` has one row per task: finish line, result, cost, turns, commits, branch, and the agent's last
line. `python3 .../night_shift.py report` rewrites it from `.night-shift/ledger.jsonl`. Then:
- For a passed task: review the diff (`git log -p main..night/<date>-<task>`), run the full suite, and only then merge.
  A passed finish line is not a review.
- For a task that did not reach the finish line: read its last note and the commits. Make the task smaller or the
  check clearer before queueing it again. Do not just raise the budget.
- Remove finished worktrees: `git worktree remove .night-shift/worktrees/<task>`.

## Never
- Push, merge, open PRs or delete branches from an unattended run.
- Queue tasks that need secrets, production access, or a human decision halfway through.
- Edit the plan's budget upwards during the night.

#!/usr/bin/env python3
"""Run a queue of agent tasks unattended: one git worktree and branch per task, a per-task and total
dollar budget, Claude Code's unattended retry settings, a finish-line check, and a morning report.

Usage:
  python3 night_shift.py run [--plan night-shift.json]   run every task in the plan, in order
  python3 night_shift.py report                           rewrite .night-shift/report.md from the ledger

Plan (night-shift.json):
  {
    "budget_usd_total": 6.00,
    "agent": ["claude", "-p", "--output-format", "json", "--max-turns", "40", "--permission-mode", "acceptEdits"],
    "tasks": [ {"id": "fix-flaky-date-test", "goal": "...", "check": "pytest -q tests/test_dates.py", "budget_usd": 2.00} ]
  }
The script appends --max-budget-usd (the smaller of the task budget and what is left) and the prompt.
Results go to .night-shift/ledger.jsonl and .night-shift/report.md. Branches are named night/<date>-<id>.
It never pushes, merges or deletes a branch. Exit 0 when every task ran, 3 when the budget stopped the queue.
Standard library only.
"""
import datetime
import json
import os
import subprocess
import sys

STATE = ".night-shift"
WATCHDOG_DEFAULTS = {
    "CLAUDE_CODE_RETRY_WATCHDOG": "1",                            # keep retrying 429/529 instead of failing at 3 a.m.
    "CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS": "1800000",          # but give up on one request after 30 minutes of waits
    "CLAUDE_CODE_RETRY_WATCHDOG_MAX_USAGE_LIMIT_WAIT_MS": "3600000",  # and end it if the usage limit resets more than 1 h away
}
MIN_TASK_USD = 0.50


def sh(cmd, cwd=None, timeout=None, env=None):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout, env=env)


def die(msg, code=1):
    print(f"night-shift: {msg}", file=sys.stderr)
    sys.exit(code)


def load_plan(path):
    try:
        plan = json.load(open(path))
    except (OSError, ValueError) as e:
        die(f"cannot read {path}: {e}")
    agent = plan.get("agent") or []
    if not agent:
        die("plan has no agent command")
    bypass = any(a.startswith("--dangerously") for a in agent) or "bypassPermissions" in agent
    if bypass and os.environ.get("NIGHT_SHIFT_SANDBOXED") != "1":
        die("refusing to run with permission checks bypassed outside a sandbox (set NIGHT_SHIFT_SANDBOXED=1 only inside a container or VM)")
    if "--max-budget-usd" in agent:
        die("leave --max-budget-usd out of the agent command; the script sets it per task from the budget")
    ids = [t.get("id", "") for t in plan.get("tasks", [])]
    if not ids or any(not i or not i.replace("-", "").isalnum() for i in ids) or len(set(ids)) != len(ids):
        die("every task needs a unique id made of letters, digits and hyphens")
    for t in plan["tasks"]:
        if not t.get("goal") or not t.get("check"):
            die(f"task {t['id']} needs a goal and a check command (the finish line)")
    return plan


def spent_tonight(ledger, night):
    return sum(r.get("cost_usd", 0) for r in ledger if r.get("night") == night)


def read_ledger():
    p = os.path.join(STATE, "ledger.jsonl")
    return [json.loads(l) for l in open(p)] if os.path.exists(p) else []


def prompt_for(task, branch):
    return (
        f"{task['goal']}\n\n"
        f"Finish line: `{task['check']}` exits 0 in this worktree. Run it yourself before you stop.\n"
        f"You are on branch {branch} in your own worktree. Commit your work to this branch with a clear message.\n"
        "Do not push, open pull requests, merge, or change files outside this worktree. Do not edit or skip tests to reach the finish line.\n"
        "If you cannot reach the finish line, commit what you have, then end with a short note: what is blocking, what you tried."
    )


def run(plan_path):
    plan = load_plan(plan_path)
    if sh(["git", "rev-parse", "--is-inside-work-tree"]).returncode != 0:
        die("run this from the root of a git repository")
    os.makedirs(os.path.join(STATE, "worktrees"), exist_ok=True)
    exclude = sh(["git", "rev-parse", "--git-path", "info/exclude"]).stdout.strip()
    if exclude and f"/{STATE}/" not in (open(exclude).read() if os.path.exists(exclude) else ""):
        os.makedirs(os.path.dirname(exclude), exist_ok=True)
        with open(exclude, "a") as fh:
            fh.write(f"\n/{STATE}/\n")  # keep worktrees and the ledger out of git status
    night = datetime.date.today().isoformat()
    total = float(plan.get("budget_usd_total", 0))
    ledger_path = os.path.join(STATE, "ledger.jsonl")
    stopped = False
    env = dict(os.environ)
    for k, v in WATCHDOG_DEFAULTS.items():
        env.setdefault(k, v)
    for task in plan["tasks"]:
        left = round(total - spent_tonight(read_ledger(), night), 4)
        budget = round(min(float(task.get("budget_usd", left)), left), 2)
        if budget < MIN_TASK_USD:
            print(f"STOP before {task['id']}: ${left:.2f} left of ${total:.2f} tonight")
            stopped = True
            break
        branch = f"night/{night}-{task['id']}"
        wt = os.path.join(STATE, "worktrees", task["id"])
        if os.path.exists(wt):
            die(f"{wt} already exists; review and remove it (git worktree remove) before re-running {task['id']}")
        r = sh(["git", "worktree", "add", "-q", "-b", branch, wt])
        if r.returncode != 0:
            die(f"git worktree add failed for {task['id']}: {r.stderr.strip()}")
        cmd = [*plan["agent"], "--max-budget-usd", f"{budget:.2f}", prompt_for(task, branch)]
        print(f"run {task['id']}: budget ${budget:.2f} (left ${left:.2f}), branch {branch}")
        started = datetime.datetime.now()
        try:
            a = sh(cmd, cwd=wt, timeout=int(task.get("timeout_min", 90)) * 60, env=env)
            out, code = a.stdout, a.returncode
        except subprocess.TimeoutExpired:
            out, code = "", "timeout"
        try:
            res = json.loads(out.strip().splitlines()[-1]) if out.strip() else {}
        except ValueError:
            res = {}
        cost = float(res.get("total_cost_usd") or 0)
        chk = sh(["bash", "-c", task["check"]], cwd=wt, timeout=1800)
        ahead = sh(["git", "rev-list", "--count", f"HEAD..{branch}"]).stdout.strip() or "0"
        row = {
            "night": night, "task": task["id"], "branch": branch, "agent_exit": code,
            "subtype": res.get("subtype", "no-json"), "turns": res.get("num_turns"), "cost_usd": round(cost, 4),
            "check": task["check"], "check_ok": chk.returncode == 0, "commits": int(ahead),
            "minutes": round((datetime.datetime.now() - started).total_seconds() / 60, 1),
            "note": (res.get("result") or "").strip().splitlines()[-1][:200] if res.get("result") else "",
        }
        with open(ledger_path, "a") as fh:
            fh.write(json.dumps(row) + "\n")
        print(f"  {task['id']}: {'PASS' if row['check_ok'] else 'FAIL'} check, {row['subtype']}, ${cost:.4f}, {row['commits']} commit(s)")
    write_report()
    return 3 if stopped else 0


def write_report():
    ledger = read_ledger()
    if not ledger:
        die("no runs in the ledger yet")
    night = ledger[-1]["night"]
    rows = [r for r in ledger if r["night"] == night]
    lines = [f"# Night shift {night}", "", "| Task | Finish line | Result | Cost | Turns | Commits | Branch |", "|---|---|---|---|---|---|---|"]
    for r in rows:
        result = "passed" if r["check_ok"] else "not reached"
        if r["subtype"] not in ("success",):
            result += f" ({r['subtype']})"
        lines.append(f"| {r['task']} | `{r['check']}` | {result} | ${r['cost_usd']:.2f} | {r['turns'] if r['turns'] is not None else '-'} | {r['commits']} | `{r['branch']}` |")
    spent = sum(r["cost_usd"] for r in rows)
    passed = sum(1 for r in rows if r["check_ok"])
    lines += ["", f"Spent ${spent:.2f} on {len(rows)} task(s); {passed} reached the finish line.", ""]
    for r in rows:
        if r["note"]:
            lines.append(f"- {r['task']}: {r['note']}")
    lines += ["", "Review each branch (`git log -p main..<branch>`) before merging. Nothing was pushed.",
              "Remove a worktree when done with it: `git worktree remove .night-shift/worktrees/<task>`."]
    with open(os.path.join(STATE, "report.md"), "w") as fh:
        fh.write("\n".join(lines) + "\n")
    print("\n".join(lines[:4 + len(rows) + 2]))


def main(argv):
    if not argv or argv[0] not in ("run", "report"):
        print(__doc__.strip(), file=sys.stderr)
        return 2
    if argv[0] == "report":
        write_report()
        return 0
    plan = argv[argv.index("--plan") + 1] if "--plan" in argv else "night-shift.json"
    return run(plan)


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

{
  "budget_usd_total": 6.00,
  "agent": ["claude", "-p", "--output-format", "json", "--max-turns", "40", "--permission-mode", "acceptEdits", "--allowedTools", "Bash(pytest *)", "Bash(git add *)", "Bash(git commit *)"],
  "tasks": [
    {
      "id": "fix-flaky-date-test",
      "goal": "tests/test_dates.py::test_week_start fails on Sundays. Find the cause in src/dates.py and fix it.",
      "check": "pytest -q tests/test_dates.py",
      "budget_usd": 2.00,
      "timeout_min": 60
    },
    {
      "id": "csv-export-header",
      "goal": "The CSV export in src/export.py has no header row. Add one, matching the column names in docs/export.md.",
      "check": "pytest -q tests/test_export.py",
      "budget_usd": 1.50
    },
    {
      "id": "type-errors-billing",
      "goal": "Make mypy pass for src/billing/ without adding type: ignore comments.",
      "check": "mypy src/billing",
      "budget_usd": 2.50
    }
  ]
}