Two days, built for engineers.

Same Bug Twice: What Happens When AI Writes Your Code And Your Tests

Mourjo Sen

AI can generate code faster than humans can write tests. More than speed though, when AI writes both the source and the test, it introduces a dangerous blind spot. Bugs that occur in the source code could also affect the test suite. The risk of bias and hallucinations in AI-generated source and tests mean that there is no longer a last line of defense that prevents the end user from a sub-par experience Hand-written example-based tests cannot keep up with AI generated code. Property-based testing (PBT) offers a different solution that relies on deterministic verification of a problem space. Instead of enumerating individual test cases, PBT explores the invariants in a system. It tests the code by systematically trying to find inputs that could break the invariant. When it finds a violation, it shrinks the failing input to the smallest possible case, surfacing bug reports that AI-generated code could likely have missed. The key idea here is that the exploration is deterministic, backed by mathematical probing of the problem space. We will walk through how PBT fights the logical holes of probabilistic source code and how it fits directly with modern trends in software engineering like spec-driven development.