framework
pytest is a widely used Python testing framework that discovers and runs test suites, with support for fixtures and a large plugin ecosystem.
No evidence-backed relationships are recorded.
Open-source tool runtape traced an agent's unrequested invoice forward to one sentence in a tool result, 10 of 10 reruns with it against 0 of 10 without. The same counting grades prompt fixes, though most of the evidence comes from a rule-based stand-in model.
Reality
PostgreSQL scans a 20-row test table sequentially even when a perfect index exists, so a test that fails on Seq Scan cannot find a missing index. Re-running EXPLAIN with enable_seqscan off catches it at any row count, since only a filter no index can serve keeps its scan.
Reality
GPT-5.5, Gemini 3.7 Flash and Grok 4.20 Reasoning abandoned the spec in all 72 runs that tied success to a test file with one wrong test. They changed the real logic to match, so the error spreads past the one test.
Reality
zerocase fails a CI run when its JUnit, TAP, LCOV, Cobertura or ESLint report shows zero executed items, even if the command exited 0. It counts only what ran, so fifty skipped tests fail the gate and fifty failed tests are left to the exit code.
Reality
Three of the four layers in a dev.to testing pyramid for MCP servers are plain pytest checks on schemas, error envelopes and session expiry. Only the fourth puts a model in the loop, and the available text breaks off before describing it.
Reality
A dev.to scorecard for agent-authored patches digests every file under testdata/ and replays recorded property seeds before pytest runs. The published fragments enforce less than the prose around them describes.
Reality
A dev.to design essay argues CI was sized for human review speed and proposes tiering checks by cost and failure probability. Its figures are examples, so the case rests on the failure modes it names.
Reality
A dev.to proposal freezes fixture bytes, runner config and two test floors at the parent SHA, then denies the coding agent write access to the file holding them. Its content hashes hold up better than the path denylist it prints.
Reality
A dev.to write-up sorts every test path into four classes and hashes the scoring ones on main, so a patch that retouches a fixture fails with a different exit code than a patch that breaks a property.
Reality
A dev.to engineer moves the admit/deny verdict back into an in-process token bucket and wraps it so the five verdict words cannot reach a model call, arguing that a free inference tier still charges the defender under a flood.
Reality
A dev.to FAQ proposes failing any agent session whose diff touches a test path without the ticket ID in a signed allowlist. Both scripts in the post are labeled unexecuted, so the false-positive count is yours to find.
Reality
An exit status of 0 covers both a suite that passed and a suite that never ran, so ten new verification tools each commit a control test that fails if the tool's own reason for existing has gone away.
Reality
Python's logging module keeps an RLock behind every handler. One shared setup_logging() therefore deadlocks a forked pool on one machine and silently drops worker records on another, and a single-OS review only ever sees one of them.
Reality
A dev.to worked example keeps HTTP clients and model SDKs out of the lease renewal path. Its Postgres fence rises on every renewal, so a write that outlives one five-second interval is rolled back by its own lease.
Reality
A dev.to harness stamps connect, TTFB, body, patch and test time on every generate call. In live mode the first two clocks come from one expression, so curl still names the handshake, and the published numbers are synthetic.
Reality
A dev.to case study builds the timezone calculator and its pytest suite first, then lets a model write the mail worker that may only call it, with 8 March 2026 pinned as the fixture no regeneration may delete.
Reality
A dev.to proposal freezes the budget in clock.env and hashes src and tests before any prompt is sent. Its sample predicate then runs a test file from outside the spike's own allowed-paths list.
Reality
A queue worker built its remaining-seconds budget from time.time(), and after the host woke from sleep it logged remaining=-1842.7. Two days of debugging went to sockets, Docker and NTP before the clocks got compared.
Reality
A post on dev.to publishes accept_run.py, a gate that starts the test command itself and writes the command, directory, exit code and output hashes to JSON, then refuses any repo whose git tree is dirty.
Reality
A dev.to write-up swaps the single assertion for per-field hit rates with 95 percent intervals over repeated trials. On its own example table, the pair it calls a real regression has intervals that overlap by 0.3 points.
Reality
Build · September 17, 2026 · 1 publisher
Build · September 16, 2026 · 1 publisher
Build · September 16, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 15, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 14, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · September 11, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 9, 2026 · 1 publisher
Build · September 8, 2026 · 1 publisher
Build · September 8, 2026 · 1 publisher
Build · September 8, 2026 · 1 publisher
Build · September 7, 2026 · 1 publisher
Build · September 5, 2026 · 1 publisher
Build · September 2, 2026 · 1 publisher
Build · September 1, 2026 · 1 publisher
Build · August 29, 2026 · 1 publisher
Build · August 28, 2026 · 1 publisher
Build · August 28, 2026 · 1 publisher
Build · August 27, 2026 · 1 publisher
Build · August 25, 2026 · 1 publisher
Build · August 20, 2026 · 1 publisher
Build · August 20, 2026 · 1 publisher
Build · August 20, 2026 · 1 publisher
Build · August 20, 2026 · 1 publisher
Build · August 19, 2026 · 1 publisher