Proving Lab → Notes
Most writing about AI-assisted development falls into two camps: it wrote my whole app, or it can't be trusted with anything. Both are useless if you actually have to ship something. So here is a log of one working day — six failures, what each one would have cost, and which check caught it. Two were caught by luck.
The work was ordinary: port a browser extension from Manifest V2 to V3, change its notification behaviour on Android, write three articles with measured data, and move the site to a new domain. Nothing exotic. That is the point — the failures below are the boring kind that happen on a normal day, not the dramatic kind that make headlines.
Case 1 · wrong by default
The extension already had a Chrome MV3 branch. Porting it had required rewriting the
whole image pipeline, because a Chrome service worker has no DOM: no
document.createElement("canvas"), no Image, no blob URLs.
Everything moved to OffscreenCanvas and createImageBitmap.
The obvious inference for the Firefox port was: same again. That inference is wrong. Firefox implements MV3 with an event page, not a service worker — the DOM stays available. The entire rewrite was unnecessary:
Firefox MV3 background: {'page': 'background.html'}
Chrome MV3 background: {'service_worker': 'background.js'}
What remained were four renames: browser_action → action,
tabs.executeScript → scripting.executeScript, the menu
context, and a dropped persistent key. Half a day of expected work
collapsed into twenty minutes.
Case 2 · nearly published
One article compared PDF extensions by what they declare in their manifests. A draft paragraph read, in essence: with a server-side converter, the contents of a logged-in banking view leave your machine. Plausible. Consistent with the permissions. Entirely wrong.
A check against the provider's own listing turned up this, in their words:
“Please note that for security reasons, it is not possible to create PDFs from pages that require a login, such as your webmail inbox, online banking, shopping cart contents […]”
A server-side converter fetches the URL itself, as an anonymous visitor. It never has your session. The claim was not a small exaggeration — it described something the product cannot do.
The corrected version turned out to be the better argument anyway: server-side converters cannot reach pages behind a login, and those are exactly the documents people want to archive. The honest version was stronger than the wrong one.
Case 3 · one commit from public
A background hook aggregates file activity into CHANGELOG.md. Useful in a
private repository. This one is public — it is the support URL on a store listing.
Staged for commit, the changelog contained 22 absolute paths of the form
/home/<user>/repos/…, plus an issue file with complete local shell
commands including Windows profile paths. Two build scripts had the Windows username
hardcoded; one of them had already been public for days.
Nothing here was catastrophic — usernames and directory layouts, not credentials. But it is the kind of leak that accumulates silently, because the automation refills the file after every run. Fixing it once is not enough; it needed a rule.
Case 4 · true when written
The README promised: “the finished PDF opens in your device's default PDF app.” Accurate when written. Two versions later the behaviour had deliberately changed — nothing opens by itself any more; a notification appears and tapping it shows the file in the browser's own viewer.
Worse, the same stale sentence sat in the German store listing, alongside a keyboard shortcut that belonged to a different browser. Anyone who tried it would conclude the extension was broken.
This is the failure mode nobody credits to AI assistance, and it is the most common one: code changes, prose does not. An assistant will happily write a correct changelog entry for a behaviour change and leave three other files describing the old behaviour, because nobody asked about those files.
Case 5 · the word meant two things
An API token had to be stored. It went into a locally encrypted bundle, and the result was reported as saved to the vault. There are two stores in this setup: a local encrypted file for infrastructure lookups, and a self-hosted password manager that is the actual vault. The written rule says the local one must never hold credentials.
The token was in the wrong one. It surfaced only because the operator asked a direct question: can you confirm the data is in the vault on the server? The honest answer was no.
The cost of not asking would have been real: the provider displays such a token exactly once. On a rebuilt machine, the local bundle gone, it would have been unrecoverable.
Case 6 · green means compiled
The MV3 port passed the linter with zero errors and zero notices. Syntax checks passed on every file in both branches. No V2 identifiers remained. It shipped.
It had never been executed. Not once. Static analysis confirms that the code is well-formed and uses the right API names — it says nothing about whether a capture completes, whether the event page survives a long scroll, whether the notification behaves as intended on a phone.
The project's own notes record two earlier bugs that survived exactly this kind of
clean check: an ImageBitmap lacking naturalWidth, and a
screenshot API silently rate-limited in one browser. Both are behaviour, not syntax.
Both were found by running the thing.
Sorting the six by what found them is more instructive than the failures themselves.
None of these were hallucinations in the popular sense — no invented API, no fabricated library, no citation to a paper that does not exist. Those are easy: they fail loudly the first time you run the code.
Every failure here was plausible. A reasonable generalisation from Chrome to Firefox. A claim consistent with the permissions. Documentation that used to be true. A word that means two things. Checks that were genuinely green. Plausible failures do not announce themselves, which is why review-by-reading finds so few of them — reading confirms internal consistency, and internal consistency is exactly what they have.
The practical conclusion is unglamorous. The checks that earn their keep are the ones that touch something outside the work: the vendor's own documentation, a grep across staged files, a read-back from the server, an actual run on real input. Everything else is a second opinion from the same source.