The Model Proposes, the Engine Decides

Hi o/,

Today I want to share where I am with sek. I already wrote about the project in another post if you missed it. That one covers how the idea started and how I ended up building the tool. This post is about what happened when I started adding an agent to it.

A technical look at SEK’s deterministic-first agentic scan

There is a point in every security tool project where somebody says: “we should add an agent.”

It is a tempting idea. Give a model the repository, let it search around, ask it to find vulnerabilities, and wait for the report.

I have tried versions of that idea. The first version is always very attractive: give the model the repository, give it a few tools, tell it to act like a senior auditor, and wait for the report.

Then you run it a few times.

One run spends most of its turns understanding the directory layout. Another starts in an interesting-looking controller and never gets to the library it calls. A third finds a real-looking flow, but the quoted line is not actually in the file anymore. You fix that, and then discover that the model has decided to stop after finding one issue because it believes it has “covered” the project.

The problem is not that the model cannot find interesting things. The problem is that a security scanner has to do more than find interesting things. It has to be repeatable enough to debug, bounded enough to operate, and explicit enough that somebody can defend the result later.

That is the reason SEK’s agentic scan is not one autonomous loop. It is a workflow made of deterministic sensors and non-deterministic judgements.

This was not the conclusion I started with. I wanted the useful part of an agent, the ability to reason about code that does not fit a preset, without giving up the useful parts of a scanner, the ability to explain what ran, what was skipped, and why a result made it into the report.

The boring problem behind the exciting one

SEK already had a Code Property Graph. The CPG makes it possible to ask useful questions about calls, control flow, and data flow. A taint preset can look for paths such as:

attacker-controlled input -> application code -> SQL query

That is a very good sensor. It is also not an audit report.

A broad preset can return multiple paths for the same root cause. Some paths are sanitized. Some are only reachable under an unusual configuration. Some are technically reachable but not exploitable in the application’s deployment. The results still need to be read, compared, grouped, ranked, and explained.

That manual triage is where a lot of the audit time goes. You launch a scan, open the first flow, follow it through a few wrappers, decide it is a duplicate, open the next one, notice that it is the same sink with a different route, and then start wondering whether the one that looked harmless was actually the important one.

The computer has already done the expensive graph traversal. The human is still doing the repetitive comparison work.

This agentic layer started from this specific problem. The goal was not to replace the CPG analysis with an LLM. The goal was to put an LLM after the part the CPG is good at, and before the part a human normally has to repeat hundreds of times.

In other words, I was not trying to automate the entire audit. I was trying to automate the boring middle.

The obvious architecture

The obvious architecture is one big agent. It gets the repository and a toolbox. It can read files, grep, run queries, take notes, and emit findings. The bigger the context window and the longer the turn budget, the more autonomous it feels.

It is also a surprisingly difficult thing to trust.

There is no stable definition of “done”. There is no clean answer to whether a missing finding means the code is safe or the agent ran out of turns. A prompt change can alter not only the explanation, but the set of files the agent visits. And if the agent can execute commands, the repository being reviewed becomes part of the prompt-injection surface.

The solution is not to pretend the model is deterministic. The solution is to decide where non-determinism is useful, then put a hard interface around it.

Deterministic versus non-deterministic

The distinction sounds abstract, so here is how I use it in the implementation.

Deterministic work has an engine-owned answer. Given the same project, query, rules, and configuration, it should produce the same result, or at least make a different result explainable. This includes:

• running CPGQL queries;
• matching the source tree with ast-grep rules;
• extracting source snippets around a flow;
• detecting whether a sanitizer appears in the path;
• validating a query, pattern, path, or code quote;
• capping flows, candidates, turns, tokens, and batch sizes;
• storing workflow state and resuming a crashed run;
• assembling and scoring a CVSS vector from qualitative metrics;
• deciding whether a candidate may be promoted into a finding.

Non-deterministic work is where the answer depends on interpretation. This includes:

• deciding whether a bounded flow is realistically exploitable;
• explaining a precondition in plain language;
• deciding whether two different paths have the same root cause;
• proposing a query for a project-specific pattern;
• ranking several surviving issues against each other;
• composing multiple weaknesses into an exploit chain;
• exploring code that does not fit a known source-to-sink shape.

The model is useful for the second list. It is a poor replacement for the first.

That gives us a fairly simple rule: if an answer can be computed from facts, the engine should compute it. If an answer requires interpretation, the model can help, but its answer must remain a proposal until the engine validates it.

flowchart LR
    S[Deterministic sensors<br/>CPG, ast-grep, saved findings] --> E[Evidence pack<br/>engine-built facts]
    E --> J[LLM judgement<br/>verdict, dedup, rank]
    J --> V[Deterministic validation<br/>schema, quotes, caps, persistence]
    V --> H[Human review<br/>or explicit auto-promote]

This is not just a philosophical preference. It gives every stage a job that can be tested independently. It also gives me somewhere to look when a result is wrong. If the flow is wrong, inspect the query or the CPG. If the source quote is wrong, inspect evidence construction. If the flow is right but the verdict is wrong, inspect the prompt, provider response, and schema-validated output.

Without those boundaries, every bug becomes “the agent did something weird.”

Why not let the model search everything?

An LLM can search a repository. The newer, uncommitted version of SEK has an agent lane that does exactly that: the model can list directories, read files, grep for patterns, run restricted CPGQL queries, keep notes, and emit candidates.

That does not mean the rest of the scanner should become an open-ended agent loop. I still want the agent lane, just not as the thing that defines coverage.

Exhaustive discovery is a bad fit for a language model. The model can stop early, choose a different route through the tree, overlook a matching path, or spend most of its budget building a map of the repository instead of reviewing it. A second run can take a different path and produce a different set of candidates.

That variability can be useful for finding things outside the known rule set. It is not a good foundation for coverage claims.

There is also a cost side to this. You can run a fully non-deterministic scan: let the agent do the discovery, the reading, and the judging. It works. It is also the most expensive way to use a model. Every directory listing, every wrong turn, every re-read file is billed in tokens, and the bill grows with the size of the repository, not with the number of real problems in it.

And even after paying that bill, a single run says nothing about what was missed. The only way to turn a stochastic search into a coverage statement is to run it repeatedly and count how often it rediscovers the same issues. New findings taper off roughly logarithmically with the number of runs, so each extra unit of confidence costs more tokens than the previous one.

The hybrid inverts that trade. The deterministic lanes buy coverage at CPU prices, once. The model only spends tokens where tokens buy something rules cannot: judgement over evidence that already exists.

So the scan starts with deterministic lanes:

• CPG presets find known source-to-sink shapes through CPGQL.
• ast-grep packs find structural, single-location patterns that taint analysis does not express well, such as weak crypto, permissive file modes, or unsafe TLS configuration.
• Saved findings allow an existing audit result to be re-triaged.
• Hypothesis probes let a model propose a project-specific CPGQL query or ast-grep pattern, while the engine validates and executes it.
• The source agent explores the repository for logic bugs and other findings with no convenient taint shape.

The important detail is that these lanes converge. They do not each invent their own report format or promotion rules.

This is the part that took me a while to appreciate. “Agentic” does not have to mean that every stage is agentic. It can mean that the workflow has a place where the model is allowed to explore, while the rest of the pipeline stays boring and predictable.

flowchart TD
    P[CPG preset bundle] --> M[Merge and candidate cap]
    A[ast-grep rule pack] --> M
    Q[LLM hypothesis proposals] --> X[Validate and execute]
    X --> M
    G[Sandboxed source agent] --> Y[Host-side emit verification]
    Y --> M
    M --> EP[Deterministic evidence pack]
    EP --> T[Shared triage and post-processing]

This is why I think of the agent as another sensor, not as the architecture.

The workflow itself

The seeded scan is an ordered workflow. Each step receives typed JSON and emits typed JSON. The result of one step becomes the input to the next.

For a fresh project, the run looks roughly like this. The important part is that the model calls are surrounded by engine-owned steps, and every boundary is persisted as typed JSON.

sequenceDiagram
    participant U as User / UI
    participant E as Workflow engine
    participant C as CPG and source sensors
    participant M as LLM provider
    participant D as SQLite run state

    U->>E: Launch scan or dry run
    E->>D: Create queued run and workflow steps
    E->>C: Run CPG presets, ast-grep, probes, and agent lane
    C-->>E: Candidate stream
    E->>E: Structural dedup, caps, and merge
    E->>C: Build bounded evidence pack
    C-->>E: Evidence with source snippets and flow facts
    E->>D: Persist deterministic step outputs

    par One bounded call per candidate
        E->>M: Normalize CWE and CVSS metrics
        M-->>E: Schema-validated qualitative metrics
        E->>M: Judge exploitability and verdict
        M-->>E: confirmed, likely, uncertain, or false_positive
    end

    E->>D: Persist prompts, responses, outputs, and token usage
    E->>M: Cluster candidates by root cause
    M-->>E: Anchored deduplication proposal
    E->>M: Re-attack confirmed and likely candidates
    M-->>E: Skeptic verdicts
    E->>M: Rank survivors and synthesize exploit chains
    M-->>E: Comparative ranks and chain proposals
    E->>D: Persist candidates and provenance
    E-->>U: Timeline, verdicts, rank, and review queue
    U->>E: Promote candidate or reject it
    E->>D: Create finding only after explicit policy or review

The re-triage workflow is the same idea, but starts from saved findings instead of running the fresh candidate lanes.

1. Generate candidates

The preset lane resolves the project’s language, selects the matching built-in presets, runs their CPGQL queries, and turns each surviving flow into a candidate. Results are cached by project and query. A broad preset cannot flood the rest of the system: flows are capped per preset, flow steps are capped, and the merged candidate stream has a global cap.

The ast-grep lane does something different. It scans the source tree for curated structural patterns without requiring a CPG. This catches bugs that are easier to describe as a local shape than as a dataflow path.

The newer hypothesis lane is deliberately asymmetric. The model can suggest a query, but it does not execute the query. The engine checks dangerous tokens, size limits, language-specific shape constraints, and candidate limits before calling the CPG query engine or ast-grep.

The source agent is more open-ended, but its output still enters the same stream. It can read files and investigate, but every emitted flow must point to real files and lines. The host re-reads each quoted line and rejects a candidate when the model’s tracked code does not match the source.

That last check sounds almost annoyingly obvious. It is also exactly the kind of check that prevents a plausible-looking hallucination from becoming evidence.

The same pattern appears in the hypothesis lane. The model can say, “I think this project has a custom parser flow worth checking.” It can propose the query. It cannot quietly turn that idea into an arbitrary query against the service. The engine validates it, records the proposal, executes it under the same limits as a normal query, and records whether it produced anything.

2. Build the evidence pack

The model should not receive a vague instruction to “review this repository.” That is how a useful security question turns into a general-purpose writing prompt.

For a normal verdict, it receives a bounded evidence pack built by the engine:

• the vulnerability class;
• the source-to-sink flow, when there is one;
• source snippets around the relevant lines;
• the sanitizer-in-path result;
• the candidate metadata and previous deterministic checks.

The snippets are wrapped as untrusted code. Code from the repository is data, not instructions. This matters because the repository itself may contain comments, strings, or files that try to influence the model.

The tool-free verdict call is intentionally narrow. A prompt injection can still cause a bad judgement, but it cannot make that particular step execute a command or query the host.

3. Ask for bounded judgement

The verdict step fans out over candidates with bounded concurrency. Each call must return a schema-validated result such as:

{
  "verdict": "likely",
  "severity": "high",
  "reasoning": "The request parameter reaches the SQL construction without a sanitizer."
}

If the response does not validate, the provider receives one correction retry with the validation error and the invalid answer. This is not an infinite repair loop. There is no reason for a malformed JSON response to turn the scanner into an unbounded conversation.

This sounds less impressive than a ReAct loop. That is part of the point. A single-shot call with a small schema is easier to retry, easier to account for, and easier to replay six weeks later when somebody asks why a candidate was marked uncertain.

The normalize step follows the same principle. The model judges a CWE and the qualitative CVSS metrics. The engine assembles the CVSS vector and calculates the numeric score. The model does not get to invent arithmetic.

That split comes up repeatedly in this system: let the model provide the part that requires interpretation, then let ordinary code handle the part that has a well-defined answer.

4. Deduplicate, challenge, and rank

Deduplication happens in two stages.

First, deterministic structural keys collapse obvious duplicates. The shortest flow is kept as the representative, while a small number of alternate paths are retained for context. This is cheap and reduces the number of expensive model calls.

Then the model performs anchored clustering over genuinely distinct candidates. It is asked whether candidates share a root cause, not whether their text merely looks similar. If the model makes a mistake, the fallback keeps candidates canonical rather than silently deleting them.

The adversarial pass then gives confirmed and likely candidates to a second skeptical judgement. A refutation blocks automatic promotion. This is important because LLM verdicts are not stable enough to be treated as ground truth. The same candidate can be confirmed in one run and rejected in another.

Finally, ranking is comparative. Instead of asking the model to assign an absolute score to every issue, the ranker inserts candidates relative to anchors and normalizes the result to a clean 1..N ordering. Comparative questions are usually easier to answer consistently than pretending that a model’s severity number has scientific precision.

Why merge the two worlds?

The deterministic and non-deterministic approaches compensate for each other’s failures.

Using only deterministic analysis gives you a reliable but incomplete system. It will miss fail-open authorization, validation asymmetries, and project-specific logic that does not match a preset.

Using only an agent gives you flexibility, but makes coverage, cost, and debugging unclear. You are not sure whether the model missed a bug because it was not there, because it stopped, or because it spent the context window reading the wrong directory.

Merging them means the model gets the freedom it actually needs, without making the whole scan depend on freedom everywhere.

The engine is part of the security boundary

Once a workflow can call providers and run source-review actions, orchestration is not just plumbing. It is part of the security model.

Every workflow step is persisted in SQLite. The run stores its status, provider, model, token usage, prompt, response, and output. That gives the implementation three practical properties:

• Crash resume: a restarted API service resets interrupted steps and continues from the first incomplete boundary.
• Replay: prompts and responses are available when a verdict needs to be inspected later.
• Dry run: deterministic steps execute for real, while model steps are skipped and their call count is estimated before any provider spend occurs.

The worker also bounds concurrency, pauses and resumes after rate limits, supports cancellation during fan-out, and surfaces authentication or context errors rather than hiding everything behind “scan failed.”

For the source agent, the boundary is stricter. File access is jailed to the repository. CPGQL is read-only and uses the same dangerous-token denylist as the deterministic query step. Shell commands, when the Docker provisioner is enabled, run in a throwaway container with no network, a read-only source mount, dropped capabilities, resource limits, and forced cleanup.

The agent is allowed to be wrong. It is not allowed to turn a wrong answer into unbounded host access.

Why candidates do not immediately become findings

This was one of the most important design decisions. It is also where the word “agentic” could have caused a very bad shortcut.

The existing findings table is a durable audit record. It should not be mutated because a provider returned a confident paragraph.

Agentic candidates therefore live in separate tables. They carry the verdict, reasoning, cluster, rank, provider information, and provenance. A reviewer can promote a candidate manually. An explicit auto-promote threshold can also promote it, but only after the relevant checks, including the adversarial pass.

stateDiagram-v2
    [*] --> candidate
    candidate --> reviewed: human review
    candidate --> promoted: explicit threshold
    reviewed --> promoted: accept
    reviewed --> rejected: reject
    promoted --> finding: provenance snapshot

This separation makes uncertainty visible instead of pretending it does not exist. It also means a prompt change or provider change does not rewrite history.

What works, and what does not

The approach works because it does not ask the model to do everything.

It works when the model receives a small enough evidence pack to reason about. It works when deterministic preprocessing removes obvious duplicates. It works when the output schema is narrow and the host validates the dangerous parts. It works when the cost is visible and a dry run can answer “how much will this take?”

It does not make the model deterministic. It does not guarantee that a confirmed finding is real. It does not solve every coverage problem. A model can still miss a mitigation in an unfamiliar wrapper, overestimate exploitability, or make a different call on a later run.

That is why the workflow keeps the uncertainty in the candidate layer and why the engine remains responsible for execution and promotion.

The newer lanes are useful precisely because they are allowed to be different. The preset lane optimizes for reproducible coverage. The ast-grep lane covers local structural mistakes. Hypothesis probes let the model adapt to a project, but keep query execution deterministic. The source agent explores logic that no rule knows how to express, but has bounded turns and host-side verification.

They are not competing claims that one technique is the future of scanning. They are sensors with different blind spots, feeding one review workflow.

That is also why I do not want to report a single magical “AI confidence” number. The result is a collection of facts and judgements: which lane found it, which source lines support it, what the model thought, whether the skeptic disagreed, which provider produced the answer, and whether a human promoted it. The more interesting the automation becomes, the more important that boring provenance is.

The model proposes, the engine decides

I started this work because I wanted less manual triage, not because I wanted to put a chatbot in front of a codebase.

The useful architecture turned out to be less magical than that. It is a pipeline with clear hand-offs:

1. Deterministic code finds and prepares evidence.
2. The model judges bounded evidence or proposes a bounded next action.
3. The engine validates the result and executes any approved action.
4. Every boundary is persisted, capped, and replayable.
5. A human, or an explicit policy, decides what becomes a finding.

That is the compromise between a rigid scanner and an uncontrolled agent. The deterministic part gives us coverage, safety, and operational control. The non-deterministic part gives us reasoning where rules are brittle and manual triage is expensive.

The model proposes an idea, the engine validates it, persists it, and the workflow reaches a reviewable result.

So yes, SEK has an agentic scan now. But the agent is not the system. It is one component inside a workflow that is still accountable to the engine.

The model proposes. The engine decides.