Running an AI Vulnerability Scanner Over Your Own Codebase Without the Bill Getting Silly

Why AI scanning is different from the linters you already run

Every engineering team already has static analysis. Linters, dependency audits, secret scanners, the security tab on your repository host. They are fast, cheap, and they catch a specific class of problem: known-bad patterns and known-vulnerable versions.

What they do not catch is the logic. An endpoint that checks authentication but not authorisation. A query builder that is parameterised everywhere except the one branch that takes a column name from user input. A webhook handler that trusts a payload because the signature check happens in a wrapper that this particular route does not use. None of those are a pattern match. Each one requires reading the code and understanding what it is trying to do.

That is the gap AI-assisted scanning fills. It reads the file, forms a hypothesis about what could go wrong, and checks whether the guard it expects is actually there. It is slower and more expensive than a linter by orders of magnitude, and it finds a category of bug your existing tooling structurally cannot.

The economics of that trade are the whole story, which is why the model you point it at matters more than anything else in the setup.

deepsec, and the flag that makes it useful

vercel-labs/deepsec is an open-source AI vulnerability scanner. The pipeline is four stages: scan, process, revalidate, report.

The first stage is regex only. It sweeps the repository and builds a candidate set of files worth looking at, and it costs nothing. Only process, revalidate and triage actually call a model. That split matters, because it means you can refresh the candidate set on every run for free and only pay for the expensive part on the files that warrant it.

The important design decision is that deepsec takes provider configuration on the command line rather than hard-coding a vendor:

Because it speaks the OpenAI-compatible API shape, anything exposing that interface is a valid backend. We point it straight at Moonshot's endpoint and run Kimi models, with no gateway in between. You could point it at anything else that speaks the same protocol without changing a line of the tool.

That flexibility is not a nice-to-have. It is what makes full-repo scanning affordable enough to do regularly rather than once, as a project, before an audit.

What a full sweep actually costs in time

Numbers from our own codebase, a Next.js application of around 1,200 source files.

Processing runs at roughly three minutes per file. That is not a typo and it is not a problem to be optimised away: the model is reading the file, reasoning about it, and forming and testing hypotheses. Multiply it out and a full-repository sweep is an overnight job measured in hours, not a step in your pull request pipeline.

This is the single most important planning fact about AI scanning, and it dictates how you scope every run.

Full repository: overnight, occasionally. Once a quarter, before a release that matters, or after a large refactor.

Scoped by path: the best risk per token by a wide margin. Point it at your API routes and you cover the surface where authentication, authorisation and input handling actually live, at a fraction of the cost of the whole tree.

Diff against your integration branch: fast enough to run frequently, and the natural fit for a scheduled job. Only files changed since the branch point get processed.

A sample of ten files: the smoke test. Run this first whenever you change a model or a setting, to confirm the pipeline works and to sanity-check your cost per file before committing to a long job.

The mistake to avoid is treating this like a linter and wiring it into every commit. You will spend a great deal of money to re-analyse code that has not changed.

Model selection, and two traps that cost us an evening

We use different models for different stages, because the stages want different things. Bulk scanning uses a code-specialised model, since throughput and code comprehension matter most across hundreds of files. Revalidation and triage use the strongest reasoning model available with the largest context, because that is where judgement quality decides whether you are reading real findings or noise.

Trap one: the model ID in the catalogue may not exist on the API you are calling. We started with a model ID taken from a gateway catalogue, pointed at the provider's direct API, and got an unhelpful failure that said nothing about the model being wrong. Gateway catalogues and direct provider APIs do not necessarily expose the same identifiers. Query the provider for its live model list before you trust any documentation:

Thirty seconds of checking against an evening of debugging an error message that pointed nowhere near the actual cause.

Trap two: reasoning models will eat your token budget before they say anything. These models split their output into reasoning and content. Set the maximum output tokens too low and the reasoning consumes the entire allowance, leaving the content field empty. The call succeeds, you are billed, and you get nothing back. If you write your own probe against one of these APIs, give it real headroom.

Both traps share a shape worth internalising: with reasoning models, a failure often looks like an empty success rather than an error.

Two rules that keep the setup safe

The scanner never fixes anything. This is the discipline that matters most and the one teams skip. The agent that finds a vulnerability does not get write access to the repository, not even for a one-line change that is obviously correct. It writes findings and a remediation note, and a separate reviewed path makes the change.

The reason is that a scanner operating on a hypothesis is exactly the wrong thing to let edit code. It has just convinced itself something is dangerous. It is primed to act. A confident wrong fix to a security-sensitive path is worse than the finding it was chasing, because it arrives wearing the authority of a security tool.

Keep the scanning workspace outside the application repository. Ours is a sibling directory rather than a subdirectory. We arrived at this for a mundane reason, a package manager conflict between the two projects, and kept it for a better one: scan output, findings, and intermediate state never land inside the repository being scanned. Nothing to gitignore, nothing to accidentally commit, and no chance of a findings file describing your vulnerabilities ending up in a deployment artefact.

Where this sits next to everything else

Static analysis reads source and makes no network requests. It cannot tell you whether your production headers are right, whether an object store is publicly readable, whether your origin is reachable around your CDN, or whether an authentication boundary that looks correct in code is actually enforced when deployed. Those are dynamic questions and they need a different exercise against the running system.

Run both, keep them clearly separated, and be precise about which one produced a given finding. A vulnerability that exists in source but is unreachable in production is a different priority from one you can demonstrate against the live service, and conflating the two is how security backlogs become noise that nobody triages.

The point of all this is not to replace your existing tooling. It is that logic bugs, the ones that need someone to actually read the code and think about it, used to be found only by expensive human review that most teams could afford once a year at best. Now they can be found on a schedule, overnight, for a cost that scales with how carefully you scope the run. That is a genuine change in what a small engineering team can defend.