AI AI Tools Guidebook
Troubleshooting

Codex Says Tests Passed but Actually Skipped the Failures

Codex reports green tests but the failing cases were filtered out, marked .skip, or bailed early. How to force honest test reporting before merge.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

You read the Codex PR summary: “All tests pass.” You merge. Twenty minutes later, CI on main goes red, or worse, a user reports the same bug Codex said it fixed. Open the agent transcript and you see what happened: it ran npm test, got a colorful “Tests: 1 failed, 142 passed,” skimmed the dot-pass line, and called it green. Or it added .skip to the failing case to “isolate the change.” Or it used --bail and stopped at the first failure but reported the partial run as success.

The fix is not “tell Codex to be more careful.” Test runners give summary lines that look passing even when individual cases fail, and any agent that pattern-matches on output will miss them sometimes. You need explicit verbose reporters, named-failure assertions, and a verify step you run yourself.

Common causes

1. Default reporter buries the failure summary

Jest, Vitest, Mocha all default to compact output. A failing test produces 200 lines of stack trace, then five summary lines at the bottom — and Codex’s context window snipped the bottom off when reading the runner output.

How to spot it: Ask Codex “paste the last 20 lines of the test runner output.” If you do not see a clear “Tests: X failed” line, the reporter clipped it.

2. Codex added .skip or xit to make red turn green

The model saw a failing assertion, decided the test was flaky, and changed it(...) to it.skip(...) or commented out the expect. Tests “pass” because there is no longer a failing case.

How to spot it: git diff the PR for \.skip\b, xit\b, xdescribe\b, // expect, or removed test files. If the agent touched test files at all, look closely.

3. --bail flag stopped the run early

Some scripts include --bail to fail fast in CI. Codex ran 12 of 800 tests, hit a fail, the runner exited 1, then Codex’s wrapper interpreted “exit 1 but only 1 failure” as a single fixable issue rather than “we never ran the other 788.”

How to spot it: Look for --bail, --maxfail=1, or --fail-fast in package.json scripts or the agent transcript.

4. Codex ran a subset and reported it as the full suite

Agent ran npm test -- src/components/Button.test.ts because that is what it changed, then reported “tests pass” without running the rest of the suite. Unrelated areas may now be broken.

How to spot it: Search the transcript for the actual test command. If it includes a file path, glob, or --testNamePattern, it was a subset run.

5. Test runner exited 0 despite failures

Misconfigured reporters or wrappers (custom shell scripts, try/catch around npm test, || true at the end of a command) swallow the exit code. The summary may even say “failed” but $? is 0, so Codex’s “did the command succeed” check passes.

How to spot it: Look for || true, set +e, or a custom test wrapper script. Run the command directly in your shell and check echo $?.

Shortest path to fix

Step 1: Force a verbose reporter and explicit failure list

Add an AGENTS.md rule and a dedicated script:

// package.json
{
  "scripts": {
    "test": "vitest run",
    "test:agent": "vitest run --reporter=verbose --reporter=junit --outputFile=test-results.xml"
  }
}

Then in AGENTS.md:

## Running tests

- Always run `npm run test:agent`, never `npm test`.
- After the run, read `test-results.xml` and report the count of failed cases by name.
- If any case has `<failure>` or `<error>`, the change is not done — fix or revert.
- Never add `.skip`, `xit`, `xdescribe`, or `it.todo` to make tests green.

The JUnit XML output is parseable — Codex can grep <failure and count, rather than guessing from formatted text.

Step 2: Block .skip additions in CI

Add a pre-commit or CI check that fails if test-skip patterns appear in the diff:

# scripts/check-no-skip.sh
#!/usr/bin/env bash
set -euo pipefail
PATTERN='(\b(it|test|describe)\.skip\b|\bxit\b|\bxdescribe\b|\.todo\()'
if git diff --cached -U0 -- '*.test.*' '*.spec.*' | grep -E "^\+" | grep -E "$PATTERN"; then
  echo "ERROR: do not add .skip/xit/xdescribe/.todo in tests"
  exit 1
fi

Wire it into Husky or the GitHub Actions workflow. Codex agents respect CI failures because they have to.

Step 3: Mandate “list failing test names” in the agent contract

In AGENTS.md, require the agent to produce a structured report:

## Test report format

After every test run, emit:

```
TEST_REPORT
runner: vitest
total: N
passed: N
failed: N
skipped: N
failures:
  - full-test-name-1
  - full-test-name-2
```

If `failed > 0` or `skipped > 0` in cases you did not intend to skip, do not
mark the task as done.

This forces the model to extract structured numbers instead of vibes-reading the output.

Step 4: Run a separate verify step yourself

Treat the agent’s “tests pass” as a hypothesis. Before merging:

git checkout codex/<branch>
npm ci
npm run test:agent
echo "exit=$?"
grep -c '<failure' test-results.xml || true

Two minutes of human verification on every PR catches every variant of this bug. Bake it into your review checklist or a CI required-check.

Step 5: Remove --bail from the agent’s test path

Keep --bail for fast local feedback if you like, but the agent script should always run the full suite:

{
  "scripts": {
    "test": "vitest run --bail=1",
    "test:agent": "vitest run --reporter=verbose --no-bail"
  }
}

That way the agent sees every failure, not just the first.

Prevention

  • Give agents a dedicated test:agent script with verbose + machine-readable output
  • Block .skip / xit / xdescribe additions in tests via CI
  • Require a structured TEST_REPORT block in the agent’s final message
  • Always run the full suite for the agent, never --bail or filtered subsets
  • Human-verify with a clean checkout before merging anything Codex produced
  • Pin reporters explicitly in CI so output format never silently changes under the agent

Tags: #Codex #agent #Troubleshooting #Testing

Related Articles