schmonz.com is a Fediverse instance that uses the ActivityPub protocol. In other words, users at this host can communicate with people that use software like Mastodon, Pleroma, Friendica, etc. all around the world.
This server runs the snac software and there is no automatic sign-up process.
Most refactors do not break because tests are missing. They break because the tests pin the wrong surface. Model-drafted tests make this worse, not better: they are fast, plausible, and confident about internals.
My rule for messy repos is simple. Draft many candidate tests. Admit only three-gate survivors. Then make the smallest safe change.
The three gates are determinism, assertion surface, and sentinel failure. A candidate test must pass all three before it can guard a refactor. A test that cannot fail is decoration.
Below is the workflow I use, with a harness sketch you can run before trusting any thresholds.
Before writing a single test, capture what the code does today. Run the entrypoint once and store the raw output.
python -m yourapp.report --account 42 > baseline/report.json
sha256sum baseline/report.json
That hash is your reference point for the whole session. If the refactor changes the bytes, you will know. Not "roughly the same JSON" — the same bytes, or an explained diff.
Keep this file out of version control churn. You want diff noise to mean something.
This is where free model access helps most. Drafting candidate characterization tests is volume work, and volume is cheap. Judgment is not.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Use a prompt contract, not a vague request. The contract below forces the model onto the observable surface and demands observed values.
You are drafting characterization tests for an unchanged module.
Rules:
1. Import only the public API listed below. Never touch names starting with _.
2. Each test performs one call and asserts one observable value or exact output string.
3. Expected values must come from the observed run I pasted. Do not guess.
4. No mocks of the module under test. Patch only network and clock.
5. Name each test after the behavior, not the function: test_rounds_discount_to_cents.
Public API: calculate_invoice, render_report
Observed run: <paste baseline output>
Ask for fifteen candidates, expect eight usable ones. Reject anything that asserts on a private attribute, a log line, or an exception message you did not paste.
Run the gate script on the candidates before you read them closely. Reading first biases you toward keeping the clever ones.
A characterization test that flips on its own is worse than no test. It teaches the team to ignore red.
Run every candidate twenty times with a different PYTHONHASHSEED. Require twenty passes.
# tools/gate_candidate_tests.py (sketch, tighten thresholds for your repo)
import os, subprocess, sys
from pathlib import Path
RUNS = 20
def run_pytest(target: Path, seed: str) -> int:
env = {**os.environ, "PYTHONHASHSEED": seed}
cmd = [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", str(target)]
return subprocess.run(cmd, env=env, capture_output=True).returncode
def determinism_gate(test_file: Path, runs: int = RUNS) -> bool:
failures = [n for n in range(runs) if run_pytest(test_file, str(n)) != 0]
print(f"determinism: {runs - len(failures)}/{runs} pass")
return not failures
Twenty runs is a floor, not a proof. It catches hash-order dependence, unseeded randomness, and real clock reads. It will not catch a branch that fires once a month.
Use a free server for this step if your laptop sleeps. A gate that dies at run nine and reports "unknown" is a gate you will skip next time. Any always-on box works; the point is an uninterrupted twenty-run window.
Now check what the assertions actually touch. Parse the test file and flag private attribute access.
import ast
def surface_violations(test_file: Path) -> list[str]:
tree = ast.parse(test_file.read_text())
bad =
[] for node in ast.walk(tree):
if isinstance(node, ast.Attribute) and node.attr.startswith("_"):
bad.append(f"line {node.lineno}: private attribute .{node.attr}")
return bad
This is syntactic, so it misses indirect access through helpers. Treat a clean report as permission to look, not proof of safety.
Two more manual checks matter. Does the test assert on a log message? Logs are not API. Does it assert on an exception type without a pinned message? That is fine for control flow, weak for refactors.
Prefer assertions on returned data and rendered output. Those survive most internal moves.
The most important check. A characterization test must fail when behavior changes.
Write a sentinel patch: one deliberate, tiny behavior change. Apply it in a throwaway clone. The test must go red.
--- a/pricing.py
+++ b/pricing.py
@@
- return round(subtotal * (1 - rate), 2)
+ return round(subtotal * (1 - rate), 1)
import shutil, tempfile
def sentinel_gate(repo: Path, test_file: Path, sentinel: Path) -> bool:
with tempfile.TemporaryDirectory() as tmp:
work = Path(tmp) / "repo"
shutil.copytree(repo, work, ignore=shutil.ignore_patterns(".git"))
applied = subprocess.run(["git", "apply", "--unsafe-paths", str(sentinel)],
cwd=work, capture_output=True)
if applied.returncode != 0:
print("sentinel patch did not apply; fix the sentinel first")
return False
red = run_pytest(work / test_file.relative_to(repo), "0") != 0
print(f"sentinel: {'caught the change' if red else 'DECORATION'}")
return red
One sentinel per behavior you claim to protect. A test that stays green after a rounding change protects nothing about rounding.
Only survivors guard the edit. Then the refactor rule applies: one edit, one full suite run.
Green suite plus unchanged baseline hash means you moved code, not behavior. Red suite means you learned something cheap.
Keep the change smaller than your confidence. If a test goes red, do not edit the test to match the new code. That is how refactors turn into silent behavior changes nobody reviews.
Gate result
Action
Determinism fails
Quarantine; find the nondeterministic source first
Private attribute asserted
Rewrite against returned data
Passes all but sentinel stays green
Delete the test; it protects nothing
Passes all three
Keep; this one guards the edit
Red after a legitimate fix
Remove the test; characterization is not a spec
That last row matters. Characterization tests freeze current behavior, bugs included. If the frozen behavior is a real defect, file it and delete the test.
If you want to try the drafting step without spending anything, run the prompt contract against a free model, then let the three gates decide what survives. The gates are the part worth keeping.#testing #refactoring #python #ai #software #coding #development #engineering #inclusive #community
A Test That Cannot Fail Is Decoration: Gating Model-Drafted Characterization Tests
A green characterization suite is not evidence of safety. It is a claim. Verify the claim before you cut anything.
Most refactors fail in the same way. The pins pass before the change. The pins pass after the change. Then a downstream job breaks, because the pins never actually watched the behavior you moved. You need a score for the net itself.
This post covers one measurable gate: mutation-check the pins first, then make the smallest safe change. Earlier posts in this series built the pins. This one asks whether they can fail.
The blind pin. It asserts something stable and irrelevant. assert result is not None passes forever, including after you delete half the report.
The tautological pin. It copies the implementation into the test. It imports private helpers and asserts the same call order the source already has. It fails when you rename, not when you break.
Both produce green CI. Neither produces a safety net. Coverage numbers will not tell you which one you have.
This harness applies small text-level mutations to the source, then runs the pin suite against each mutant. A good net catches most mutants. A decorative net catches none.
Save as pin_check.py. Exit code 0 means the score cleared your threshold.
#!/usr/bin/env python3
"""Measure characterization-pin sensitivity with text-level mutations.
Usage:
python pin_check.py --repo . --min-score 0.6 \
--pin-cmd "python -m pytest -q tests/pins"
Exit codes: 0 = score >= min-score, 1 = below threshold, 2 = harness error.
"""
from __future__ import annotations
import argparse
import json
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass, asdict
from pathlib import Path
@dataclass(frozen=True)
class Mutation:
name: str
rel_path: str
old: str
new: str
# One row per behavior you believe the pins protect.
# Text-level on purpose: cheap, reviewable, no AST tooling required.
MUTATIONS: tuple[Mutation, ...] = (
Mutation("currency_default", "billing/invoice.py",
'currency: str = "USD"', 'currency: str = "EUR"'),
Mutation("round_precision", "billing/invoice.py",
"round(total, 2)", "round(total, 1)"),
Mutation("skip_none_rows", "billing/report.py",
"if row is None:", "if not row:"),
Mutation("sort_key", "billing/report.py",
"key=lambda r: r.date", "key=lambda r: r.amount"),
Mutation("header_order", "billing/report.py",
'["id", "date", "amount"]', '["date", "id", "amount"]'),
)
IGNORE = shutil.ignore_patterns(".git", "__pycache__", "*.pyc", ".venv")
def run(cmd: list[str], cwd: Path) -> int:
proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
return proc.returncode
def apply_mutation(root: Path, mut: Mutation) -> bool:
target = root / mut.rel_path
if not target.exists():
return False
src = target.read_text()
if mut.old not in src:
return False # stale pattern: refresh this row
target.write_text(src.replace(mut.old, mut.new, 1))
return True
def check(repo: Path, pin_cmd: list[str]) -> dict:
baseline_exit = run(pin_cmd, repo) # pins must be green on pristine tree
caught = applied = 0
detail: list[dict] =
[] for mut in MUTATIONS:
with tempfile.TemporaryDirectory() as tmp:
work = Path(tmp) / "repo"
shutil.copytree(repo, work, ignore=IGNORE)
if not apply_mutation(work, mut):
detail.append({"mutation": mut.name, "status": "stale-pattern"})
continue
applied += 1
hit = run(pin_cmd, work) != 0
caught += int(hit)
detail.append({"mutation": mut.name,
"status": "caught" if hit else "missed"})
score = caught / applied if applied else 0.0
return {"baseline_exit": baseline_exit, "applied": applied, "caught": caught,
"score": round(score, 2), "detail": detail}
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--repo", default=".")
ap.add_argument("--pin-cmd", required=True)
ap.add_argument("--min-score", type=float, default=0.6)
args = ap.parse_args()
report = check(Path(args.repo).resolve(), args.pin_cmd.split())
print(json.dumps(report, indent=2))
if report["baseline_exit"] != 0:
print("harness error: pins already fail on the pristine tree", file=sys.stderr)
return 2
return 0 if report["score"] >= args.min_score else 1
if __name__ == "__main__":
raise SystemExit(main())
The mutation list is the actual deliverable. Each row is a belief you are testing. Add a row every time a reviewer asks "but does this break X?"
Run the pins alone first. A failing baseline invalidates the whole measurement.
python -m pytest -q tests/pins # must exit 0
python pin_check.py --repo . --min-score 0.6 \
--pin-cmd "python -m pytest -q tests/pins"
Expect baseline_exit: 0. Expect a detail array. Expect missed rows. The misses are the point.
The number is a threshold check. Use it to decide whether to start, not to celebrate.
Score
Reading
Action
0.0 - 0.2
Pins are decorative
Do not refactor. Pin public outputs of the entry points first.
0.3 - 0.5
Blink spots cluster
Open the missed rows. Add one output-level pin per cluster.
0.6 - 0.8
Usable for a bounded change
Proceed with one behavior-preserving step only.
0.9 - 1.0
Possibly over-pinned
Grep for private imports. Trim pins that fail on rename.
A high score is not automatically good news. Suites that pin every internal call also score well. They just cost you every future rename.
Pins that import private names rarely specify behavior. They duplicate it.
# Suspects: pins reaching into underscore-prefixed helpers.
grep -rnE "^\s*from [A-Za-z_.]+ import _" tests/pins | tee tautology-suspects.txt
wc -l < tautology-suspects.txt
Review each hit. Ask one question: does this assertion describe output, or restate the source? If it restates, rewrite it against the public boundary.
One behavior-preserving step. One commit. No drive-by cleanups, no renames.
git switch -c refactor/invoice-totals
# single change here only
python -m pytest -q tests/pins # green after the change
git stash
python -m pytest -q tests/pins # green before it too
git stash pop
If the pins pass only on the changed tree, they were tailored to the change. That is a rewrite of the specification, not a verification of it.
The mutation list is the slow part. Someone has to guess which behaviors matter. I use MonkeyCode's free model access to draft candidate rows from a module's recent diff history, then delete most of them by hand. The candidates are proposals only, and every row still needs a human to confirm it describes real behavior. Its free server option is useful when the pin suite is too heavy for a laptop and CI minutes are budgeted elsewhere.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Treat availability, limits, and model lineups as operator-supplied and changeable. Verify current terms yourself. Do not put a free tier on a release-critical path.
Text-level mutations are crude. They catch only a pattern that still exists verbatim. Rename a variable and the row silently becomes stale-pattern. Refresh rows when you touch the file.
A high score does not prove correctness. It proves the net can fail on the mutations you thought of. Semantic drift outside your list stays invisible.
If you already run a real mutation framework, use it. Tools like mutmut or cosmic-ray generate far better mutants than a hand-written list. This harness is for repos where adding one more test dependency is politically harder than adding one script.
Skip this if the output has no stable entry point, or if nobody consumes the output. Skip it for throwaway scripts. The score only matters where a regression has a cost.
Run the harness on your messiest module and post the missed list. The score is a gate. The misses are the map.
One behavior-preserving change, one verified net. In that order.#testing #refactoring #python #cicd #software #coding #development #engineering #inclusive #community
Test the Safety Net First: Mutation-Check Characterization Pins Before a Refactor
Josh Marino 🦞 (@AIRoboticsInt)
Gemini 4 Argon의 최대 출력 길이가 기존 64K에서 100만 토큰으로 확대됐다는 내용이다. 멀티파일 애플리케이션 생성이나 대규모 리팩터링을 한 번의 응답으로 처리할 수 있는 규모이며, 얼리 액세스 사용자는 긴 작업에서 중도 포기가 적었다고 평가했다. 다만 3D 작업 성능 평가는 유보해야 한다고 언급했다.
Over time, some couples drift apart. They still share a sofa, but they’re really just cohabiting 💔.
What’s true for couples is also true for our shared abstractions.
Here’s how to spot the cohabiting trap, and escape it: https://williamboles.com/escaping-the-cohabiting-trap/
A late cleanup request arrived with a four-hundred-line diff that rewrote validation, deduplication, and persistence in one pass. The existing tests covered only the happy path, so a green CI run could not explain the legacy rows. The reviewer needed a smaller change that preserved every observed output before any structural boundary was allowed to move. This composite scenario shows how to freeze those outputs, then accept only the smallest safe edit.
The module under review is a batch upload normalizer that grew by successive tickets rather than by design. It reads a list of records, fills missing defaults, drops exact repeats, and writes survivors to storage. Those four jobs share one function, so a cleanup that touches all of them cannot be reviewed as one claim. The method below separates observation from edit, and it does not depend on any particular assistant.
Before changing code, write down the inputs that already matter and the outputs you are willing to treat as current truth. Include awkward rows from production samples, not only the tidy examples attached to the latest ticket. A characterization file is a pinned record of those pairs, not a statement that the behavior is desirable. If a row is known to be wrong, mark it as accepted debt instead of silently correcting it during the same change.
Three observations keep the freeze honest when the repository is already messy.
That list is the review contract for this module, and it should stay visible in the pull request description. It is narrower than a repository-wide characterization suite, and it is not a license to rewrite neighboring helpers. This pass does not hash a repository-wide side-effect ledger, and it does not ask you to freeze every neighboring seam. The next section turns the contract into a local command you can rerun after every proposed diff.
The following Python example is a proposed harness, not a log from a measured production run. It uses a memory store so the test process does not touch a real database or a shared queue. The cases cover a blank SKU, a negative quantity, an exact repeat, and a lowercase SKU that the current code uppercases. Run the capture against the untouched function, then commit the fixture before you accept any cleanup.
import json
from pathlib import Path
class MemoryStore:
def __init__(self):
self.batches =
[] def write(self, rows):
self.batches.append([dict(row) for row in rows])
def normalize_upload_batch(records, store):
seen = set()
kept =
[] for record in records:
sku = (record.get('sku') or '').strip().upper()
qty = record.get('qty', 0) or 0
if not sku or qty < 0:
continue
key = (sku, int(qty))
if key in seen:
continue
seen.add(key)
kept.append({'sku': sku, 'qty': int(qty)})
store.write(kept)
return kept
CASES = [
{
'name': 'blank-sku',
'records': [{'sku': ' ', 'qty': 2}],
'debt': False,
},
{
'name': 'negative-qty',
'records': [{'sku': 'ab1', 'qty': -1}],
'debt': False,
},
{
'name': 'case-and-repeat',
'records': [
{'sku': 'ab1', 'qty': 2},
{'sku': ' AB1 ', 'qty': 2},
{'sku': 'ab1', 'qty': 3},
],
'debt': False,
},
]
def capture(cases):
report =
[] for case in cases:
store = MemoryStore()
returned = normalize_upload_batch(case['records'], store)
report.append({
'name': case['name'],
'debt': case['debt'],
'returned': returned,
'written': store.batches,
})
return report
def main():
path = Path('batch_characterization.json')
payload = json.dumps(capture(CASES), indent=2) + '\n'
path.write_text(payload, encoding='utf-8')
print('wrote', path)
if __name__ == '__main__':
main()
A second, ordinary command compares a fresh capture with the fixture you already committed to the branch. The comparison is a textual check of the JSON, not a visual skim of a large assistant diff. If you later correct a debt row, change the fixture in a separate commit and state the product reason in that commit message. Until you trust the harness, treat a non-empty fixture diff as a failed review rather than as a suggestion.
Save the harness as capture_batch.py beside the fixture so the commands below match the file you run.
python capture_batch.py
git diff -- batch_characterization.json
The sample function above is embedded so a reader can follow the article without cloning a repository. In a real repository, import the existing function instead of pasting a simplified copy into the test file. A pasted copy can drift from the production function and then bless the wrong behavior during review. Keep the fixture next to the module so a reviewer can open both without searching the whole tree.
After the fixture is committed, the only edit this pass allows is an extraction of the pure selection loop. The wrapper may call the new function and then perform exactly one write to the existing store. It may not add a sort, a new skip rule, a retry, or a second store call. The table below is the acceptance gate for the diff, and a single failed row is enough to reject the change.
Check
Accept this pass
Send it back
Files touched
The one module that owns the batch function
Any helper outside that file
Fixture JSON
Unchanged after a local rerun
Any added, dropped, or reordered field
Store writes
One write, matching the returned rows
An extra write, a cleared batch, or a write before selection
New branches
None
A fallback, a retry, or a temporary compatibility flag
Debt rows
Still marked debt in the fixture
Quietly corrected inside the cleanup
The extracted shape that satisfies every row of the table looks like the unexecuted sketch below. It remains a proposal until you rerun the capture and the fixture diff is completely empty. Notice that the skip rules are copied, not improved, because an improvement is a behavior change and belongs in a later commit. The wrapper stays responsible for the write, so persistence bugs are not mixed into the pure move.
def select_upload_rows(records):
seen = set()
kept =
[] for record in records:
sku = (record.get('sku') or '').strip().upper()
qty = record.get('qty', 0) or 0
if not sku or qty < 0:
continue
key = (sku, int(qty))
if key in seen:
continue
seen.add(key)
kept.append({'sku': sku, 'qty': int(qty)})
return kept
def normalize_upload_batch(records, store):
kept = select_upload_rows(records)
store.write(kept)
return kept
That order matters more than the size or brand of the assistant that happened to propose the cleanup. A small extraction can still be wrong if you regenerate the fixture after the edit and then treat the new file as the baseline. Commit the baseline first, or the acceptance gate has nothing stable left to protect later. If step three fails, restore the extraction and inspect the case, rather than widening the diff to force a match.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A coding assistant can draft fixture cases or the extraction sketch after you have already named the rows you care about. For this draft, the operator supplied two availability claims: free model access and a free server option. This article does not state model names, token quotas, hardware size, duration, or benchmark scores, because those details were not provided as primary sources. Check the project documentation on the day you start, because availability terms can change and should not be copied from an older post.
The useful split should stay small even when the assistant is allowed to see the whole repository. Ask it to propose cases that match the contract, then paste those cases into the harness and run the capture yourself. Ask for the extraction only after the fixture is committed, and reject any diff that fails a row in the table. A free server can be a convenient place to run that capture when your laptop cannot match the language version. The committed fixture still belongs in the repository you review, not only on the temporary machine.
If that gate matches how you already review cleanups, read the current access notes before you try it. Then run the same fixture workflow on one module you can execute on a machine you control. Keep the product optional: the harness, the table, and the empty fixture diff are the parts a teammate can reuse without an account. A reviewer should never treat a model-written explanation as proof that the fixture file remained stable.
This approach freezes current behavior, including bugs that nobody has yet scheduled for a product fix. It is a poor fit when the ticket requires a visible behavior change inside the same pull request. It is also a poor fit when you cannot obtain representative rows, because an empty fixture will bless whatever the first draft happens to emit. A memory store does not prove transaction order, locking, or partial failure, so do not use it to clear a data-migration change.
Skip the gate as your only review when the diff touches authentication, billing totals, or deletion. Those areas need explicit expected outcomes, not a snapshot of whatever the old code already did. Do not plan a team rollout around an unverified free quota or an assumed server lifetime from an old post. If today's documentation does not confirm the access terms you need, treat the assistant as optional. Keep the harness on a machine you control rather than on a shared environment you have not reviewed.
The closing claim stays modest, and a reviewer can audit it from the fixture diff alone. A four-hundred-line cleanup becomes reviewable when one characterized seam moves and the fixture file stays still. Everything else can wait for a later commit that states the behavior it intends to change. That later commit should carry its own cases, rather than borrowing the freeze you used to keep this extraction honest.#testing #refactoring #python #programming #software #coding #development #engineering #inclusive #community
Require an Empty Fixture Diff Before Extracting a Batch Loop
Agents Refactor 300K Lines in Three Weeks, and Practitioners Ask What It Proves
CodeScene has published a case study in which coding agents refactored a 300,000-line C codebase over three weeks,…
#NewsBeep #News #Artsanddesign #agenticrefactoringcasestudy #AI #AIdevelopment #Architecture&Design #Arts #ArtsAndDesign #CodeQuality #Design #Development #Entertainment #ML&DataEngineering #Refactoring #softwareengineering #TechnicalDebt #UK #UnitedKingdom
https://www.newsbeep.com/uk/787296/
BOOTOSHI 👑 (@KingBootoshi)
에이전트 기반 코드 생성에서는 초기 구현의 완성도보다 먼저 동작하는 기능을 빠르게 만든 뒤 리팩터링하는 2단계 워크플로가 고품질 코드로 이어진다는 주장이다. 모델이나 복잡한 워크플로보다 실행 가능한 초안을 우선 확보하는 방식에 초점을 둔다.
Nous Research (@NousResearch)
Nous Research는 Hermes Agent에 100만 줄 규모의 Python 코드베이스 정리를 맡긴 결과, 1,393개 서브에이전트가 19시간 동안 작업해 코드 규모를 34.4% 줄였다고 공개했다. 대규모 멀티에이전트 기반 리팩터링의 운영 사례로, 약 200만 달러 상당의 엔지니어링 시간을 절감했다는 회사 측 추정도 포함됐다.
AI (@DeepLearn007)
Teknium이 AI 서브에이전트 1,393개를 19시간 운용해 코드베이스 크기를 34.4% 줄이고, 엔지니어링 작업 시간 약 200만 달러 상당을 절감했다고 공유했다. 대규모 병렬 에이전트 활용이 리팩터링·레거시 코드 정리 비용을 줄일 수 있음을 보여주는 사례다.