Reviewing AI-written code: what AI gets wrong and a checklist before merge

AI writes code many times faster than a person, but that speed moves the work onto the reviewer. Code that looks tidy and runs can call a package that does not exist, swallow an error, or open a hole through an unsafe default. This article lists what AI gets wrong most often and gives a checklist you can use before you press merge.

Facts checked against the USENIX Security 2025 and ACM CCS 2023 papers, Veracode’s 2025 report and Google’s code review guide. Commands tested with npm 10.9, pip 25.0 and Python 3.12.

Why AI code needs a different review

When people write bad code it usually shows: odd names, half-finished logic. AI code is wrong while looking right: clean formatting, comments everywhere, sensible function names. That is what lulls a reviewer.

Numbers you can check:

• Veracode 2025 tested more than 100 models on 80 tasks and found vulnerabilities in 45% of test cases. Java was worst at over 70%; Python, C# and JavaScript sat between 38% and 45%. Larger models did not do significantly better than smaller ones.
• Perry et al. (CCS 2023): participants with an AI assistant wrote less secure code than those without, and were more likely to believe their code was secure. Those who trusted the AI less and reworked their prompts produced fewer vulnerabilities.

So the problem is not which model you pick. It is what happens after the AI hands the work over.

What AI gets wrong most often

1. Invented APIs and packages

The paper “We Have a Package for You!” (USENIX Security 2025) had 16 models write 576,000 Python and JavaScript samples. Of 2.23 million package references, 440,445 (19.7%) named packages that do not exist, 205,474 unique names. Commercial models averaged at least 5.2%, open-source models 21.7%.

The dangerous part: the fake names repeat. Asked the same thing 10 times, 43% of hallucinated names came back every time, and 58% came back more than once. So an attacker can register that name on npm or PyPI, put malware in it, and wait for someone to npm install what the AI suggested. This is slopsquatting, a term coined by Seth Larson of the Python Software Foundation as a play on typosquatting.

Invented APIs work the same way: a method that does not exist in the version you use, an option the library never had. It fails at runtime, not when you read it.

2. Missing edge cases

AI is good at the happy path. It forgets the empty list, zero, negatives, time zones, non-ASCII input, the request that arrives twice. Ask every time: what happens if the input is empty, huge, or malformed?

3. Silently swallowed errors

try:
    save_order(order)
except Exception:
    pass  # the save failed and nobody will ever know

Tests pass, the screen shows no error, and data is gone. Every except should do one of three things: actually handle it, log it and re-raise, or return something the caller knows means failure.

4. Tests that test nothing

def apply_discount(price, pct):
    return price * (1 - pct / 100)

def test_apply_discount():
    result = apply_discount(100, 10)
    assert result is not None   # passes even if the formula is wrong

You want assertions on the correct values, plus the edges:

import pytest

def test_apply_discount():
    assert apply_discount(100, 10) == 90
    assert apply_discount(100, 0) == 100
    assert apply_discount(100, 100) == 0

def test_discount_over_100_is_rejected():
    with pytest.raises(ValueError):
        apply_discount(100, 150)   # fails until the code validates pct

Other common forms: mocking the very function under test, or changing the expected value to match what the broken code returns. When an AI “fixes” a failing test, check whether it changed the code or the test.

5. Unsafe defaults

Veracode found models failed to defend against cross-site scripting (CWE-80) in 86% of cases and log injection (CWE-117) in 88%. What we see in real work:

• verify=False on HTTPS calls to “make it work”
• CORS * or debug=True that ships to production
• SQL built by string concatenation instead of parameterized queries
• user input dropped straight into HTML or logs
• a key or password hard-coded “as an example” and never removed

Worked example: the odds of at least one fake package

Say the AI suggests 10 dependencies for a project, and each has a 5.2% chance of being fake (the commercial-model floor from the paper). Assume each is independent, which is a simplification to make the point.

1. Chance one package is real = 1 − 0.052 = 0.948
2. Chance all 10 are real = 0.948¹⁰ ≈ 0.586
3. Chance at least one is fake = 1 − 0.586 ≈ 41%

Dependencies suggested	at 5.2%	at 21.7%
10	≈ 41%	≈ 91%
20	≈ 66%	≈ 99%

Real rates depend on the model and the task, but the lesson holds: check every one, not a sample.

Check a package before installing

A name that does not exist returns 404 straight away. For one that exists, look at when it was created; a package a few days old with no users deserves suspicion.

# npm: does it exist, when was it created
npm view <package-name> time.created

# PyPI: 404 means no such package
curl -s -o /dev/null -w "%{http_code}\n" https://pypi.org/pypi/<package-name>/json

Then open the package page and look at the source repo, the maintainers and the download count. Version pinning and lockfiles are covered in supply-chain safety for npm and pip.

The checklist before merge

#	Check	Fastest way
1	The diff is small enough to read	git diff --stat; if it is several hundred lines, have the AI split it
2	New dependencies exist and are trustworthy	npm view / PyPI JSON for each one
3	It actually runs	tests + linter + type check, before reading
4	Tests assert correct values	read tests before code; look for is not None, self-mocks
5	Errors are not swallowed	search for except: pass, empty catch {}
6	Edge cases	empty / zero / huge / duplicate input
7	Security settings	search for verify=False, CORS *, debug, string-built SQL, secrets in code
8	Called APIs exist in your version	open the docs for the version you pinned
9	You understand every line you merge	explain it to yourself; if you cannot, ask or rewrite

Small diffs, and run before you read

Google’s code review guide says about 100 lines is usually reasonable and 1,000 lines is usually too large. Small changes get a more thorough review, make bugs easier to spot and are easier to roll back. With AI this matters more, because it will happily hand you 1,000 lines in a minute. Ask for one step at a time, one step per commit.

The order we use:

1. Run it first. Tests, linter, type check, build. If it fails, do not read it; send it back.
2. Read the tests. Do they prove what you need?
3. Read the risky code closely. New dependencies, input handling, auth, money, anything that deletes data.
4. Try it for real. Open the screen, try the wrong inputs, not only the right ones.

If AI code runs on your laptop but breaks on the server, works on my machine covers how to prevent that.

AI can write the code, but it cannot sign for it. Whoever presses merge owns it.

Common mistakes

• Judging by how it looks. AI code always looks good. Run it and check the assertions.
• Installing whatever the AI names. Check the registry first, every time.
• Letting the AI fix tests until they pass without checking whether it changed the code or the expected value.
• Accepting a huge diff because you are in a hurry. Huge diff = skim = bugs slip through.
• AI reviewing AI as the final gate. Fine as an extra pass; a person is the last one.
• Forgetting the AI reads outside content. If an agent reads a web page or an issue and then writes code, that content can steer it. See defending against prompt injection.

Sources

• We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs — arXiv (USENIX Security 2025)
• Do Users Write More Insecure Code with AI Assistants? — arXiv (ACM CCS 2023)
• AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks — Veracode
• Small CLs — Google Engineering Practices
• AI hallucinations lead to a new cyber threat: Slopsquatting — CSO Online