pandas 3 guard for copy-on-write and removed APIs

# AGENTS.md: pandas 3 project (pandas 1 and 2 guard)

This project uses **pandas 3** (`pandas>=3.1,<4`). Most pandas code you have seen predates it. pandas 3 removed a
long list of deprecated APIs and changed two things that make old code wrong without an error:
**copy-on-write is always on**, and **strings are the `str` dtype, not `object`**.

## Version facts
- Check the pin first: `uv pip show pandas` or the `pandas` line in `pyproject.toml`/`uv.lock`.
- Copy-on-write: every indexing result behaves as a copy. Chained assignment (`df["a"][0] = 1`) and
  `df["a"].fillna(0, inplace=True)` change a temporary copy and leave `df` alone. pandas only warns.
- `pd.Series(["a"]).dtype` is `str`. `dtype == object` checks are now false for string columns, and a
  string column refuses non-string values.
- Offset aliases: `h`, `min`, `s`, `ms`, `us`, `ns`; month, quarter and year ends are `ME`, `QE`, `YE`.
- If you are unsure an API exists in pandas 3, check the API reference (https://pandas.pydata.org/docs/reference/)
  or the 3.0 release notes (https://pandas.pydata.org/docs/whatsnew/v3.0.0.html). Don't copy pandas 1 answers.

## Rules
- Write pandas 3 only. The `pandas-3-guard` skill has the full table; the common traps are:
  `DataFrame.map` (not `applymap`), `pd.concat` (not `append`), `.ffill()` (not `fillna(method=)`),
  `sep=r"\s+"` (not `delim_whitespace`), `date_format=` (not `date_parser`), `freq="h"`/`"ME"` (not `"H"`/`"M"`),
  `errors="coerce"` (not `"ignore"`), `pd.read_json(io.StringIO(text))` (not literal JSON).
- Assign with one `.loc` call: `df.loc[mask, "col"] = value`. Never chain `[...][...] =`.
- Reassign instead of `inplace=True` on a column: `df["col"] = df["col"].fillna(0)`.
- Test for strings with `pd.api.types.is_string_dtype(s)`, not `dtype == object`.
- Don't upgrade or downgrade pandas without asking the human.

## Before you say "done"
1. Run the `pandas-3-guard` scanner on the files you touched; it must report `0 error(s)`. Read every `warn`.
2. Run the tests with warnings as errors for pandas' own warnings:
   `uv run pytest -W error::pandas.errors.ChainedAssignmentError`.
3. Say which silent changes (copy-on-write writes, string dtype checks) you looked for and where.

# CLAUDE.md

@AGENTS.md

## Claude Code notes

- Skills from this loadout live in `.claude/skills/`: `pandas-3-guard`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From SetupBay "pandas 3 guard for copy-on-write and removed APIs" v1.0.0 — https://setupbay.com/loadouts/pandas-3-upgrade-guard/ -->

# GEMINI.md

@./AGENTS.md

## Gemini CLI notes

- Skills from this loadout live in `.agents/skills/`: `pandas-3-guard`. Use them when their description matches the task.
- Follow AGENTS.md above; where this file and AGENTS.md disagree, AGENTS.md wins.

<!-- From SetupBay "pandas 3 guard for copy-on-write and removed APIs" v1.0.0 — https://setupbay.com/loadouts/pandas-3-upgrade-guard/ -->

---
description: pandas 3 only; copy-on-write and the str dtype, no APIs removed in 3.0
globs: "**/*.py,**/*.ipynb"
alwaysApply: false
---

This project pins pandas 3. Copy-on-write is always on and strings are dtype `str`.

| Never (pandas 1 and 2) | Always (pandas 3) |
|---|---|
| `df["a"][0] = 1` | `df.loc[0, "a"] = 1` |
| `df["a"].fillna(0, inplace=True)` | `df["a"] = df["a"].fillna(0)` |
| `df.applymap(f)` | `df.map(f)` |
| `df.append(other)` | `pd.concat([df, other], ignore_index=True)` |
| `fillna(method="ffill")` | `.ffill()` |
| `read_csv(delim_whitespace=True)` | `read_csv(sep=r"\s+")` |
| `read_csv(date_parser=...)` | `read_csv(date_format="%Y-%m-%d")` |
| `freq="H"`, `"T"`, `"M"`, `"Y"` | `freq="h"`, `"min"`, `"ME"`, `"YE"` |
| `to_numeric(s, errors="ignore")` | `to_numeric(s, errors="coerce")` |
| `pd.read_json('[{...}]')` | `pd.read_json(io.StringIO(text))` |
| `s.dtype == object` for strings | `pd.api.types.is_string_dtype(s)` |
| `agg(np.sum)` | `agg("sum")` |

- No defensive `.copy()` to silence SettingWithCopyWarning; that warning no longer exists.
- After editing, run the `pandas-3-guard` skill's scanner and the tests.

---
name: pandas-3-guard
description: Check Python for pandas APIs removed in 3.0, chained assignment that copy-on-write ignores, and object-dtype string checks, before saying a pandas change is done. Use after writing or editing pandas code, when pandas raises an unexpected keyword or attribute error, warns ChainedAssignmentError, or when porting older pandas snippets.
---

# pandas 3 guard

pandas 3.0 (21 January 2026; 3.1 on 22 October 2026) removed what 2.x deprecated and turned on two behaviour
changes for good. Copy-on-write means a chained write such as `df["v"][0] = 1` changes a copy and leaves `df`
alone, with only a warning. The `str` dtype means string columns are no longer `object`. Removed names fail loudly.
Those two changes don't fail at all.

## 1. Confirm the version
- `uv run python -c "import pandas; print(pandas.__version__)"` must print `3.x`. If it prints `2.x`, stop and ask
  whether the project is moving to pandas 3. The pandas docs suggest getting clean on 2.3 first.

## 2. Scan
```bash
python3 .agents/skills/pandas-3-guard/scripts/pandas3_guard.py src tests notebooks
```
(Use the path the skill was installed to: `.claude/skills/...` for Claude Code, `.agents/skills/...` for the others.)

- Every `error` line is either an API pandas 3 removed or a write that copy-on-write throws away. Fix it with the
  replacement the line gives; `references/rename-table.md` has the table.
- The chained-assignment rules only look at names like `df`, `*_df`, `frame*` and `data`, so a frame called
  something else can slip through. The test run below catches those.
- `warn` lines still run but may mean something different now: `dtype == object` checks, `agg(np.sum)`,
  groupby on categoricals. Fix them, or add `# guard: ok` to a line where the new behaviour is what you want.
  That silences warnings, never errors.

## 3. Let pandas check it
```bash
uv run pytest -q -W error::pandas.errors.ChainedAssignmentError
```
- `ChainedAssignmentError` is a warning by default. Turning it into an error makes every lost write fail a test.
- `TypeError: ... got an unexpected keyword argument` and `AttributeError: 'DataFrame' object has no attribute`
  mean a removed API. The release notes name each replacement.
- `TypeError: Invalid value ... for dtype 'str'` means code put a non-string into a string column. Convert on purpose
  (`astype(object)`) only if mixed values are really wanted.

## 4. Report
Say how many removed calls and chained writes you replaced, which dtype checks you changed, and that the scanner and
the tests (with ChainedAssignmentError as an error) were clean.

#!/usr/bin/env python3
"""Find pandas 1.x/2.x code that pandas 3 removed, and code that copy-on-write or the str dtype quietly changed.

Usage: python3 pandas3_guard.py [PATH ...]        (default: the current folder)
       python3 pandas3_guard.py --list            (print every rule)

A "guard: ok" comment on a line silences its warnings (never its errors).
Prints one line per finding:  path:line: level [rule] what is wrong -> what to use
Exit 0 when nothing at "error" level is found, 1 when something is, 2 on bad usage.
Standard library only. Read-only: it never edits a file and makes no network calls.
"""
import fnmatch
import os
import re
import sys

# (rule id, level, regex, message, file patterns). Patterns run against each line with comments removed.
RULES = [
    ('applymap', 'error', '\\.applymap\\(',
     'DataFrame.applymap() was removed -> DataFrame.map()', []),
    ('append', 'error', '\\b(df\\w*|frame\\w*|data\\w*|result\\w*|out\\w*)\\.append\\(',
     'DataFrame.append() was removed (2.0) -> pd.concat([a, b], ignore_index=True)', []),
    ('iteritems', 'error', '\\.iteritems\\(',
     'iteritems() was removed (2.0) -> items()', []),
    ('fillna-method', 'error', '\\.fillna\\([^)]*\\bmethod\\s*=',
     "fillna(method='ffill') was removed -> .ffill() / .bfill()", []),
    ('replace-method', 'error', '\\.replace\\([^)]*\\b(method|limit)\\s*=',
     'replace(method=..., limit=...) was removed -> .ffill() / .bfill() after replacing with NA', []),
    ('interpolate-pad', 'error', '\\.interpolate\\([^)]*[\\"\'](pad|ffill|bfill|backfill)[\\"\']',
     "interpolate('ffill') was removed -> .ffill() / .bfill()", []),
    ('freq-alias', 'error', '\\b(freq|rule)\\s*=\\s*[\\"\'](\\d*)(H|BH|CBH|T|L|U|N|M|BM|SM|CBM|Q|BQ|Y|BY|A|BA|AS|BAS)(-[A-Z]{3})?[\\"\']',
     "old offset alias -> 'h', 'min', 'ms', 'us', 'ns', 'ME', 'QE', 'YE' (month/quarter/year END aliases)", []),
    ('resample-alias', 'error', '\\.(resample|asfreq)\\(\\s*[\\"\'](\\d*)(H|T|M|Q|Y|A)[\\"\']',
     "old offset alias in resample/asfreq -> 'h', 'min', 'ME', 'QE', 'YE'", []),
    ('delim-whitespace', 'error', '\\bdelim_whitespace\\s*=',
     'read_csv(delim_whitespace=True) was removed -> sep=r"\\s+"', []),
    ('date-parser', 'error', '\\b(date_parser|infer_datetime_format|keep_date_col)\\s*=',
     'read_csv date_parser / infer_datetime_format / keep_date_col were removed -> date_format=... or pd.to_datetime after reading', []),
    ('swapaxes', 'error', '\\.swapaxes\\(',
     'swapaxes() was removed -> .T or .transpose()', []),
    ('series-view', 'error', '\\]\\.view\\(|\\bseries\\w*\\.view\\(|\\bs\\.view\\(',
     'Series.view() was removed -> .astype(...)', []),
    ('groupby-axis', 'error', '\\.(groupby|rolling|resample)\\([^)]*\\baxis\\s*=',
     'the axis= argument was removed from groupby/rolling/resample -> transpose first', []),
    ('first-last', 'error', '\\.(first|last)\\(\\s*[\\"\']\\d',
     "DataFrame.first('3D') / last('3D') were removed -> df.loc[df.index < df.index[0] + pd.Timedelta('3D')]", []),
    ('errors-ignore', 'error', '\\bto_(numeric|datetime|timedelta)\\([^)]*errors\\s*=\\s*[\\"\']ignore[\\"\']',
     "errors='ignore' was removed -> errors='coerce' and handle NaN, or try/except", []),
    ('series-bool', 'error', '\\.bool\\(\\s*\\)',
     'Series.bool() / DataFrame.bool() were removed -> bool(s.item())', []),
    ('use-inf-as-na', 'error', 'use_inf_as_na',
     'mode.use_inf_as_na was removed -> df.replace([np.inf, -np.inf], np.nan)', []),
    ('copy-on-write-option', 'warn', 'mode\\.copy_on_write|copy_on_write\\s*=',
     'copy-on-write is always on in pandas 3; the option does nothing and will be removed', []),
    ('read-literal', 'error', 'pd\\.read_(json|html|xml|excel)\\(\\s*(f?[\\"\'][\\[{<]|response\\.text|r\\.text)',
     'read_json/read_html/read_xml no longer take literal text -> wrap it: pd.read_json(io.StringIO(text))', []),
    ('chained-assignment', 'error', '\\b(df\\w*|\\w+_df|frame\\w*|data)\\[[^\\]]+\\]\\[[^\\]]+\\]\\s*=[^=]',
     "chained assignment never writes back under copy-on-write -> df.loc[row, 'col'] = value", []),
    ('inplace-on-column', 'error', '\\b(df\\w*|\\w+_df|frame\\w*|data)\\[[^\\]]+\\]\\.\\w+\\([^)]*inplace\\s*=\\s*True',
     "df['col'].method(inplace=True) changes a copy, not df -> df['col'] = df['col'].method()", []),
    ('object-dtype-check', 'warn', 'dtype\\s*==\\s*(object|[\\"\']object[\\"\']|np\\.object_)|select_dtypes\\([^)]*[\\"\']object[\\"\']',
     "strings are now dtype 'str', not object -> pd.api.types.is_string_dtype(s) or select_dtypes(include=['str'])", []),
    ('setting-with-copy', 'warn', 'SettingWithCopyWarning|\\.copy\\(\\)\\s*#.*SettingWithCopy',
     'SettingWithCopyWarning is gone in pandas 3; the defensive .copy() is not needed for it', []),
    ('observed-default', 'warn', '\\.groupby\\((?![^)]*observed)[^)]*\\)(?=.*cat)',
     'groupby on categoricals now defaults to observed=True; pass observed=False to keep empty categories', []),
    ('future-stack', 'warn', '\\.stack\\([^)]*future_stack',
     'future_stack is the default in pandas 3; drop the argument', []),
    ('np-reduction-in-agg', 'warn', '\\.(agg|apply|transform)\\(\\s*(np\\.(sum|mean|max|min|std)|sum|max|min)\\s*[,)]',
     "agg(np.sum) is no longer swapped for pandas' own sum -> pass the string 'sum'", []),
]
FILES = ['*.py', '*.ipynb']
COMMENT = '#'
SKIP_DIRS = {".git", ".venv", "venv", "node_modules", "__pycache__", ".agents", ".claude", ".cursor", "dist", "build", "target", ".godot"}
COMPILED = [(rid, lvl, re.compile(rx), msg, globs) for rid, lvl, rx, msg, globs in RULES]


def strip_comment(line):
    """Drop a trailing comment, leaving string literals alone."""
    q = None
    i = 0
    while i < len(line):
        c = line[i]
        if q:
            if c == "\\":
                i += 2
                continue
            if c == q:
                q = None
        elif c in "\"'`":
            q = c
        elif line.startswith(COMMENT, i):
            return line[:i]
        i += 1
    return line


def wanted(name):
    return any(fnmatch.fnmatch(name, g) for g in FILES)


def walk(paths):
    for p in paths:
        if os.path.isfile(p):
            yield p
            continue
        for root, dirs, files in os.walk(p):
            dirs[:] = sorted(d for d in dirs if d not in SKIP_DIRS)
            for f in sorted(files):
                if wanted(f):
                    yield os.path.join(root, f)


def scan(path):
    out = []
    name = os.path.basename(path)
    try:
        with open(path, encoding="utf-8", errors="replace") as fh:
            lines = fh.read().splitlines()
    except OSError as e:
        return [(path, 0, "error", "unreadable", str(e))]
    for n, raw in enumerate(lines, 1):
        code = strip_comment(raw)
        if not code.strip():
            continue
        accepted = "guard: ok" in raw  # a person accepted the warnings on this line
        for rid, lvl, rx, msg, globs in COMPILED:
            if accepted and lvl == "warn":
                continue
            if globs and not any(fnmatch.fnmatch(name, g) for g in globs):
                continue
            if rx.search(code):
                out.append((path, n, lvl, rid, msg))
    return out


def main(argv):
    if "--list" in argv:
        for rid, lvl, _, msg, globs in RULES:
            print(f"{lvl:5} [{rid}] {msg}" + (f"  (only {', '.join(globs)})" if globs else ""))
        return 0
    if any(a.startswith("-") for a in argv):
        print(__doc__)
        return 2
    paths = argv or ["."]
    for p in paths:
        if not os.path.exists(p):
            print(f"no such path: {p}", file=sys.stderr)
            return 2
    files = list(walk(paths))
    findings = []
    for f in files:
        findings.extend(scan(f))
    for path, n, lvl, rid, msg in findings:
        print(f"{os.path.relpath(path)}:{n}: {lvl} [{rid}] {msg}")
    errors = sum(1 for x in findings if x[2] == "error")
    warns = sum(1 for x in findings if x[2] == "warn")
    print(f"pandas3_guard: {len(files)} file(s), {errors} error(s), {warns} warning(s)")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

# pandas 1/2 → pandas 3 table

From the pandas 3.0.0 release notes (https://pandas.pydata.org/docs/whatsnew/v3.0.0.html) and the copy-on-write
guide (https://pandas.pydata.org/docs/user_guide/copy_on_write.html). Checked against pandas 3.1.0.

## Removed: raises in pandas 3

| Old | pandas 3 |
|---|---|
| `df.applymap(f)` | `df.map(f)` |
| `df.append(other)` (removed in 2.0) | `pd.concat([df, other], ignore_index=True)` |
| `df.iteritems()` (removed in 2.0) | `df.items()` |
| `df.fillna(method="ffill")` | `df.ffill()` |
| `df.replace(x, method="pad")` | `df.replace(x, np.nan).ffill()` |
| `s.interpolate("pad")` | `s.ffill()` |
| `freq="H"`, `"BH"`, `"T"`, `"L"`, `"U"`, `"N"` | `"h"`, `"bh"`, `"min"`, `"ms"`, `"us"`, `"ns"` |
| `freq="M"`, `"Q"`, `"Y"`, `"A"` (ends) | `"ME"`, `"QE"`, `"YE"` |
| `read_csv(delim_whitespace=True)` | `read_csv(sep=r"\s+")` |
| `read_csv(date_parser=f)` | `read_csv(date_format="...")` or `pd.to_datetime` afterwards |
| `read_csv(infer_datetime_format=True)` | drop it; strict inference is the default |
| `df.swapaxes(0, 1)` | `df.T` |
| `s.view("int64")` | `s.astype("int64")` |
| `df.groupby("k", axis=1)` | `df.T.groupby("k")` |
| `df.first("3D")` / `df.last("3D")` | `df.loc[df.index < df.index[0] + pd.Timedelta("3D")]` |
| `pd.to_numeric(s, errors="ignore")` | `pd.to_numeric(s, errors="coerce")` |
| `s.bool()` | `bool(s.item())` |
| `pd.set_option("mode.use_inf_as_na", True)` | `df.replace([np.inf, -np.inf], np.nan)` |
| `pd.read_json('[{"a": 1}]')` | `pd.read_json(io.StringIO(text))` (same for read_html, read_xml) |

## Copy-on-write: writes that are silently lost

| Old | pandas 3 |
|---|---|
| `df["v"][0] = 1` | `df.loc[0, "v"] = 1` |
| `df[df.a > 0]["b"] = 0` | `df.loc[df.a > 0, "b"] = 0` |
| `df["v"].fillna(0, inplace=True)` | `df["v"] = df["v"].fillna(0)` |
| `sub = df[cols]; sub["x"] = 1` expecting `df` to change | change `df` itself |
| defensive `.copy()` to silence SettingWithCopyWarning | not needed; the warning is gone |

## Changed meaning: same code, different answer

| Change | What to do |
|---|---|
| String columns are dtype `str`, not `object` | `pd.api.types.is_string_dtype(s)`; `select_dtypes(include=["str"])` |
| A `str` column refuses non-strings | convert on purpose if mixed values are wanted |
| Missing strings are `NaN` | test with `isna()`, not `is None` |
| `groupby(...)` on categoricals defaults to `observed=True` | pass `observed=False` to keep empty categories |
| `agg(np.sum)` is no longer swapped for pandas' sum | `agg("sum")` |
| `Series[int]` always means a label, never a position | `s.iloc[i]` for positions |
| `mode.copy_on_write` option does nothing | remove it |

# A pandas 2 cleaning script, used to check the guard. Every marked line fails or silently changes in pandas 3.
import numpy as np
import pandas as pd

df = pd.read_csv("readings.txt", delim_whitespace=True)                       # delim-whitespace
df = pd.read_csv("readings.csv", parse_dates=["day"], date_parser=pd.to_datetime)  # date-parser
df = df.applymap(lambda v: v.strip() if isinstance(v, str) else v)            # applymap
df = df.fillna(method="ffill")                                                # fillna-method
df["reading"][0] = 0.0                                                        # chained-assignment
df["site"].fillna("unknown", inplace=True)                                    # inplace-on-column
hourly = df.set_index("day").resample("H").mean()                             # resample-alias
months = pd.date_range("2026-01-01", periods=12, freq="M")                    # freq-alias
codes = pd.to_numeric(df["code"], errors="ignore")                            # errors-ignore
more = df.append(df.tail(1))                                                  # append
text_cols = [c for c in df.columns if df[c].dtype == object]                  # object-dtype-check
totals = df.groupby("site").agg(np.sum)                                       # np-reduction-in-agg