# Security scanning my own code with Scrutineer and local coding models *2026-08-31 — note* After the recent surge of [security reports](https://anil.recoil.org/notes/rumour-is-the-exploit) in my projects, I wanted to help us open source maintainers use scanning tools to find bugs before the bad guys do. Luckily, [Andrew Nesbitt](https://nesbitt.io) leapt in [enthusiastically](https://amok.recoil.org/@andrewnez@mastodon.social/117150259002007792) with [Scrutineer](https://github.com/alpha-omega-security/scrutineer), a new project he's working on to help maintainers proactively manage the flood. I decided to give it a quick spin. By default, Scrutineer supports a number of different coding models, but I remain blocked on access to [frontier models](https://www.anthropic.com/glasswing) without security guardrails for our own code. Therefore, I decided to get a local coding model powered by [GLM 5.3](https://huggingface.co/zai-org/GLM-5.3-Flash) running on my own Mac Studio. Read on if you want to set up your own instance; Scrutineer is worryingly good at finding lots of bugs\! ## What Scrutineer does [Scrutineer](https://github.com/alpha-omega-security/scrutineer) is a scanner built under the [Alpha-Omega](https://alpha-omega.dev) project. It lets me add a Git repository or a whole org of them, and then schedules a big pipeline of agent scans via a swarm of ephemeral Docker containers. These scans include a first triage pass, then metadata analysis, dependency and SBOM extraction, static analysis with tools like [semgrep](https://semgrep.dev/) and [zizmor](https://docs.zizmor.sh/), defining a threat model, and then a security deep-dive that does hours of analysis. A web UI organises the (often raw) results for human triage, with a workflow designed to avoid [flooding maintainers with low-quality reports](https://alpha-omega.dev/blog/scrutineer-scanning-open-source-without-flooding-maintainers/). The agent CLI of choice is quite pluggable with Claude Code by default, with Codex, OpenCode and Copilot as alternatives (only one backend can be active at a time). Andrew kindly added an [OCaml runner profile](https://github.com/alpha-omega-security/scrutineer/pull/901) days after my [earlier post](https://anil.recoil.org/notes/2026w34), so OCaml repositories now also have a specialised scanning container with our toolchains like opam installed.
The skill pipeline fanning out across parallel scans after the initial triage of cohttp.
The skill pipeline fanning out across parallel scans after the initial triage of cohttp.
I deployed it on an [exe.dev](https://exe.dev) VM, a service I'm fast becoming a [fan of](https://anil.recoil.org/notes/2026w30) due to its nice CLI and convenient HTTPS proxies. Scrutineer runs as a dedicated host process there, as every scan it does spawns a fresh Docker container so there's no point running the service itself in a container. The UI doesn't have any auth of its own, so for now it's using exe's [authenticated HTTPS tunnel](https://exe.dev/docs/proxy). I'm not sure how to set up a 'production' multiuser version of this yet, but it's early days and I return to this [below](#figuring-out-usable-workflows-here). Deploying it was extremely simple by following the docs, despite it being under rapid development. I submitted a minor fix to the Dockerfile ([\#949](https://github.com/alpha-omega-security/scrutineer/pull/949)) and upgraded [`git-pkgs`](https://github.com/git-pkgs/git-pkgs) to the latest one with opam support ([\#950](https://github.com/alpha-omega-security/scrutineer/pull/950)). The latter meant that Scrutineer could break down the individual opam dependencies that a repo depends on. For example, [cohttp](https://github.com/mirage/ocaml-cohttp) went from 13 dependencies to 207, of which 194 are its opam packages (with version constraints). Not bad\! ## Driving Scrutineer with a local model I bootstrapped the instance with the default Claude Code backend just to get going. Its initial triage (of cohttp) took Claude Opus about 13 minutes/18 agent turns, with the lighter skills completing on Sonnet in seconds.
The initial triage scan under the Claude backend.
The initial triage scan under the Claude backend.
I then switched the instance to an OpenCode backend, pointed at a GLM 5.3 Flash (Q4) model served from my Mac Studio by my [ocaml-deepseek](https://tangled.org/anil.recoil.org/ocaml-deepseek) inference engine (the `glm-5.3` branch), which I [previously described](https://anil.recoil.org/notes/language-integrated-llms). This exposes an OpenAI-compatible endpoint, and Scrutineer's per-scan proxy routes each container's traffic to it. I was a little sad that I couldn't keep both Claude and OpenCode simultaneously active, though. Using the local model is _much_ slower than Claude as I've done absolutely no performance tuning yet (and for some reason, Spotlight is chewing up 100% CPU on my Mac Studio in the Golden Gate beta). I got decode speeds of around ten tokens per second, so I raised the scan timeout to eight hours to let the security deep-dives complete. While this seems slow, it's totally fine for security scans as I just left them running overnight and there's no marginal cost per scan with local models given cheap nighttime renewable electricity. [Sadiq Jaffer](https://toao.com) is going to install GLM 5.3 on our beefy 4-GPU machine in a few days to give us a speedup, but this is fine for now. ## My first findings The first repo was cohttp as that was the last one I issued an [advisory](https://osv.dev/vulnerability/OSEC-2026-16) for. The generated scan configuration first uses an agent to map its attack surface file by file, and so in the case of an HTTP parser it figured out that it needed to cover the request-line parser, the chunked transfer decoder, cookie and authorisation header parsing, and the lex/menhir grammar behind content negotiation. This is actually quite a good summary that would double as an onboarding doc for a new contributor\!
The generated scan configuration maps cohttp's attack surface file by file.
The generated scan configuration maps cohttp's attack surface file by file.
I picked a real 'harmless' finding for the purposes of exposition here, since I obviously don't want to talk about potential real bugs. In this case, the agent decided that the `COHTTP_DEBUG` environment variable is tested for presence rather than its value, so setting it to `0` or the empty string still enables full wire logging. I don't really care enough to fix this; the agent got alarmed that it logs authorisation headers to stderr, but this is strictly a debugging flag (as the name implies) so my judgement is that it's fine as is. Note that this might not always be true: if an attacker could somehow inject environment variables as part of a chain attack, they might suddenly be able to use this to retrieve some valuable auth tokens.
A sample trivial finding where any value of COHTTP_DEBUG logs credentials to stderr.
A sample trivial finding where any value of COHTTP_DEBUG logs credentials to stderr.
Scrutineer then hands this over to a 'disclosure' workflow where it uses another agent to independently verify the bug and/or draft a vulnerability disclosure. ## Figuring out usable workflows here I'm pretty convinced I want to put more time into using Scrutineer now. Here are some notes (in no particular order) about what's needed to do a bigger deployment. As I type this, I left it running over the weekend on 10 more repositories and it found over 100 potential issues that need triaging\! ### Supporting multiple backend code models The coding model backend is currently instance-wide as far as I can tell, so once I switched to OpenCode I could no longer mix in Claude or Codex. It would be quite nice for routine security deep-dives to run overnight on the local model, but then to push promising fixes to a frontier model (since those fixes usually aren't blocked by the cybersecurity guards, just the act of finding them!). ### Quick'n'dirty multiuser support via exe.dev Since the Scrutineer UI doesn't have a login of its own, the exe.dev proxy is the only access control on a deployment like mine. I do this by running `ssh exe.dev share add scrutineer ` which grants that person [web access](https://exe.dev/docs/sharing) to the VM via the private HTTPS proxy once they've logged into exe.dev. It's therefore now possible to distinguish user requests via the `X-ExeDev-UserID` and `X-ExeDev-Email` headers that the proxy [adds to authenticated requests](https://exe.dev/docs/login-with-exe). That solves the immediate problem for a handful of co-maintainers, but until Scrutineer itself has user logins, triage decisions all look like they came from the same person. ### A git notes CLI workflow would be amazing Scrutineer already has some [fledgling federation support](https://github.com/alpha-omega-security/scrutineer/blob/main/docs/interchange.md), to let a peer check if another instance holds the same finding before disclosing it. I haven't looked closely to verify this protocol, but I'm paranoid here as even the disclosure of the existence of bugs in a repo might be enough to trigger exploit generation. Reading this did make me wonder if a more CLI-driven workflow would make it easier to cherry-pick patches and pull/push across maintainers, since we already have a very git-centric workflow for normal maintenance. When [Tangled](https://tangled.org) gains support for [private repositories](https://atproto.com/blog/atproto-spaces-alpha), it would be very cool for Scrutineer to expose '[magic refs](https://stackoverflow.com/questions/72022248/git-refs-pull-vs-pull)' like GitHub does to deal with triaging workflows. E.g. something like: ```bash # this is just a sketch of a possible CLI and not actually implemented yet git remote add scrutineer https://scrutineer.example.com/git/mirage/ocaml-cohttp git fetch scrutineer '+refs/findings/*:refs/remotes/scrutineer/findings/*' git log --oneline main..scrutineer/findings/17 git diff main...scrutineer/findings/17 git cherry-pick scrutineer/findings/17 && dune runtest git notes --ref=triage/avsm add -m 'wontfix: debug-only flag, not a real issue' scrutineer/findings/17 git push scrutineer refs/notes/triage/avsm ``` Each maintainer's findings could then live in their own git notes, allowing maintainers to disagree. The overall decision of a finding would then be the merge of those refs. This is the sort of thing that [Irmin](https://github.com/mirage/irmin) was [built for](https://anil.recoil.org/papers/2014-oud-irminsule)! It provides Git-compatible storage with [mergeable branches](https://anil.recoil.org/papers/2015-jfla-irmin) and a full history of user commits, although I've never tried git notes with it. The more I think about it, the accept/reject step that maintainers do seems to be a really valuable part of the process here. Keeping the full set of triage decisions maintainers are making _across_ repositories would feed back to subsequent scans and perhaps make a broader organisation-wide "living" threat model. It would also remove some of the mind-numbing tedium of triage a bit. This is also a reflection of my longstanding desire to have an open-source version of [Iron](https://www.youtube.com/watch?v=MUqvXHEjmus) coming to the fore again\! Synopsis: Deploying Alpha-Omega's Scrutineer on a VM, driving its scans with a local GLM 5.3 on my Mac Studio, and thinking through practical workflows Words: 1541 ## Related - [.plan-26-35: All I want for Tesseramas is (GP)U](https://anil.recoil.org/notes/2026w35) (note, 2026-08-30) - [.plan-26-34: Am I human or am I antibotty](https://anil.recoil.org/notes/2026w34) (note, 2026-08-23) - [Just a rumour of a bug is enough to find a security exploit these days](https://anil.recoil.org/notes/rumour-is-the-exploit) (note, 2026-08-22) - [.plan-26-30: Bananas ripen, models open, and OxCaml releases](https://anil.recoil.org/notes/2026w30) (note, 2026-07-26) - [Language integrated LLMs as an OCaml function](https://anil.recoil.org/notes/language-integrated-llms) (note, 2026-06-14) - [Mergeable persistent data structures](https://anil.recoil.org/papers/2015-jfla-irmin) (paper, 2015-01-01) - [Irminsule: a branch-consistent distributed library database](https://anil.recoil.org/papers/2014-oud-irminsule) (paper, 2014-09-01) --- Canonical: https://anil.recoil.org/notes/scrutineer-local-llm Type: note License: CC BY 4.0 Tags: security, ai, ocaml, llms