Your tests pass. Would they catch a real bug?
Every month we plant 16 realistic bugs in your code, one at a time, and run your own tests against each. You get a score, every bug that slipped through, and the test that would have caught it. For teams whose code and tests are now written with Cursor, Claude Code or Copilot.
Three steps. One number that means something.
- 01
Give us read access
One repo. Nothing is ever pushed to it: your tests run in a private copy.
- 02
We plant 16 realistic bugs
Chosen from failures we have seen in real AI-written code, frozen before anything runs, and planted one at a time against your own test suite.
- 03
You get the score and the fixes
Every bug that slipped through, with the file, the line and the test that would have caught it. Next month, the same bugs again to prove your fixes hold, plus new ones in your new code.
Not a list of warnings. A list of tests to write.
Each miss says whether the code is never run by your tests or run without being checked, so you know which gaps are cheapest to close.
Your tests caught 4 of 16 realistic bugs.
Bug list frozen before the run
An error inside an event handler was swallowed; the webhook answered 200, so Stripe would never retry.
Test to add: make a handler throw; expect a 500.Updating only a subscription's metadata wiped its org and user link.
Test to add: update metadata alone; assert both keep their values.The bugs that get past good test suites
We planted small, realistic bugs, written blind, in 525 public projects and ran each project's own tests. Of the 7,319 bugs in code the tests ran, these kinds got through most often.
- A prefix is matched without its separator, so main2/ counts as main53% got through
- A lookup key loses one of its fields, so two records collide42% got through
- A more specific setting, such as a per-call argument, is ignored and the general default is used39% got through
- An upstream status maps to the wrong local status39% got through
Share of counted bugs the suite did not catch, for kinds counted 40 or more times. Bugs in lines our coverage showed were never run are not counted.
Free during early access.
The first report is free for public or private repos. If paid plans ever come, you hear from us first, and nothing is charged without your agreement.
- 16 realistic bugs planted every month
- Your score, and its trend month over month
- Every miss with the file, the line and the test to add
- A test pull request for each miss you want closed, only with your OK
- Last month's bugs planted again, to prove your new tests hold
- A public receipt and README badge, when you want to show your score
- Read access only; nothing is pushed to your repo
Repos up to about 100,000 lines. Full terms
Do you run our code?
Yes: your own test suite, in a private copy of your repository. Nothing touches your repo or your systems, and the copy is deleted within 30 days after you stop.
Do you need write access?
No. Read access to one repo. Public repos need nothing at all.
What if our tests catch everything?
Then you have a score worth showing: ask, and we switch on a public receipt and README badge. Next month shows whether it holds: all of this month's bugs are planted again, and new ones go into the code you changed.
Why every month?
Because the suite you measured in March is not the suite you have in June. Code changes, tests get deleted or loosened to make a build pass, and an assistant rewriting a module rarely rewrites its tests. Each month all of last month's bugs are planted again, so the tests you added are proven and a weakened test shows up as a miss, and up to 16 new bugs go into whatever changed since the last run. One run tells you where you stand. The series tells you whether it's holding.
How do you choose the bugs?
From a catalogue of about 600 kinds of mistake we've seen in more than 500 real codebases, and we track which ones test suites keep missing. For your repo we pick the kinds that fit your code (money handling gets unit mix-ups, logins get flipped checks) and write at least three that only make sense in your product. Every bug is written from your code and docs without opening your tests, and the list is frozen before anything runs, so it can't be aimed at your gaps.
How is this different from mutation testing or a PR bot?
Mutation tools such as Stryker flip operators by the thousand and leave you to sort the noise. A PR bot, or a Claude skill, can plant bugs against your suite too, but then the assistant that wrote your tests is grading them. We plant 16 bugs a developer or an assistant would actually write, taken from what broke in more than a hundred real AI-built codebases, hash the list before the first run, and count a catch only when a test fails twice. The number is one nobody on your team had a hand in, and every miss comes with the test that closes it. More on AI mutation testing.
Start with the repo you trust least.
Trenyx is free during early access. The first month is a full run and report. If the repo is public, we can run a free preview first.