Fable cut out on me at 1am on Saturday while I was sweeping over the OCaml runtime looking for concurrency bugs. There have been excellent takes on the sovereignity implications of this, and I figured I'd roll my sleeves up and get serious about using the open weights models. DeepSeek's models have been getting more capable since their first release, and v4 Flash is small enough to run on my Mac (admittedly, very high-end Macs with 128GB/512GB of RAM respectively for my laptop and desktop).
The question is whether the agentic CLIs I've been using can be easily replaced. The best way to learn how a system works is to build it unikernel style, and so I aimed to expose the LLM as a normal OCaml library. This avoids routing via bloated CLIs, and lets the linking application drive the agentic loop according to its specific needs.
What makes this practical is Antirez' Dwarfstar, a self-contained native inference engine that supports Apple Metal and portable(ish) C. I bound this directly to OCaml 5 and Eio as ocaml-deepseek, and now a plain function call on my laptop gets me an LLM in my application.
For example, I can now embed Deepseek inference directly into the OCaml webserver that drives this very site in order to look for suspicious bot activity, and because it's open weights and running locally, there's no dependency on external services!
(* A traffic-triage agent in-process in OCaml. The agent is handed two
OCaml function tools and works out for itself how to combine them. *)
let agent =
Agent.create engine ~system:"You are a web-traffic analyst."
~tools:[
Toolbox.read ~dir:logs; (* read-only sandboxed handle to the logs dir *)
query_db ~conn; (* a SELECT-only tool over the local database *)
]
in
Agent.send agent ~on_event
"Cross-reference today's 404 spikes in the access log against the \
client IPs in the requests table. Anything coordinated indicating a bad bot?"
The log reader and the database query are just two OCaml functions the model is allowed to call, each scoped and sandboxed (using Eio) to exactly what it needs. The model decides when and how to combine them.
1 Trying out Humpty the OCaml agent
I've submitted the package to opam, so opam install deepseek or opam pin add deepseek https://tangled.org/anil.recoil.org/ocaml-deepseek.git should work.
The package also ships a binary called humpty[1] with two variants: humpty-metal for Apple Silicon and a portable humpty-cpu that should run anywhere (slowly).
There are four subcommands that we'll use to explain how to build an agent up in OCaml:
first list the available models and download one, then chat with it statelessly, and then wrap that into an agent.
1.1 Choose the right Deepseek model
Before we can get started you'll first need the open model weights downloaded.
humpty list
prints a the catalogue of available weights:
$ humpty-metal list
Models (download dir: /Users/avsm/.local/share/ds4)
TARGET ALIASES DESCRIPTION
[ ] q2-imatrix q2 2-bit Flash routed experts (~81 GB); for 96-128 GB RAM.
[*] q2-q4-imatrix q2q4 Mixed Flash quant (~98 GB); higher quality for 128 GB.
[ ] q4-imatrix q4 4-bit Flash routed experts (~153 GB); for 256 GB+ RAM.
[ ] pro-q2-imatrix pro-q2 PRO q2 single file (~430 GB); for 512 GB RAM.
[*] = present, [ ] = not downloaded
Pick one based on how much RAM you have; I use q2q4 on my laptop (with 128GB RAM),
and the extremely beefy pro-q2 on my Mac Studio (with 512GB RAM).
There are also split files for running the model distributed across several machines, which I'll
skip here for now.
1.2 Grab the Deepseek model weights
Once you've chosen, humpty download q4
(or pro-q2, or whichever) shells out to the Hugging Face CLI to fetch the GGUF.
You'll either need the Huggingface CLI installed or have uvx in your path.
Once this gets doing go have a cup of tea while the gigabytes of LLM weights download, and then we'll start to build an agent from the camel up!
2 Building an agent from the ground up
I first want to pin down what an "agent" actually means, as the term seems to have accreted much mystique this year. The whole OCaml Deepseek stack is a small library you can read through quickly, so let me build an agent up from scratch. The code below links to the Tangled source.
2.1 An LLM is a stateless request-reply function
A basic LLM takes in a text prompt, performs inference on some weights, and generates a text reply back.
To illustrate this in our OCaml code, we need to load the model weights and spin up an
engine with a cache directory for the compiled Metal kernels (if using the
Apple GPU version):
let engine = Deepseek.V4.create ~cache ~model ~domain_mgr ~sw () in
V4.generate engine "Explain monads in one sentence." ~on_token:print_string;
- : unit
=> Monads are a design pattern that allows you to chain operations together
while automatically handling extra behavior like error handling, state, or
side effects, by wrapping values in a context and providing a way to
transform and combine them.
Deepseek.V4.create
opens the GGUF model file and, the first
time it runs, materialises the embedded Metal shaders in the cache. Generating a
reply is then a single call to V4.generate that
encodes the supplied prompt, runs a prefill, and samples one token at a time into the on_token callback until the
end-of-sequence marker.
All the inference is done in the Metal library in a separate OCaml domain,
so we can continue to use other Eio fibres in our main application.
You can try this single-shot request/response using humpty chat, which keeps no memory between runs and can't take any action beyond showing the reply.
$ humpty-metal chat 'Explain algebraic effects in OCaml 5 in one sentence'
Algebraic effects in OCaml 5 allow functions to suspend execution and invoke
user-defined handlers for operations (like state, exceptions, or generators)
via a lightweight, type-safe mechanism that integrates with the language's
existing type system and is used primarily for effectful programming, such as
with the new `Effect` library for handling delimited continuations.
A "conversation" is therefore just a list of role-tagged messages (e.g. system, user, assistant, tool) that we concatenate in our library into a prompt string for the LLM.
3 How the stateless LLM asks for effects to the external world
The single-step text-to-text function from earlier emits text in an agreed "shape" so we can figure out what to do next based on its output. DeepSeek has trained their model to understand a little markup language called DSML, which looks something like this:
<｜DSML｜tool_calls>
<｜DSML｜invoke name="edit">
<｜DSML｜parameter name="path" string="true">/tmp/x.c</｜DSML｜parameter>
<｜DSML｜parameter name="line" string="false">42</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>
DSML is a pseudo-XML language that's interspersed in the text responses from the agent.
Those bars are actually the full-width vertical line ｜ (U+FF5C) and not an ASCII |. DeepSeek reserves the rarer codepoint for DSML's control tokens, so they can't be produced by ordinary text or code the model emits.
We've got a DSML implementation in OCaml, which parses the XML into an OCaml record type:
type thinking_mode = Chat | Thinking
type reasoning_effort = High | Max
type task = Action | Query | Authority | Domain | Title | Read_url
type tool_call = { id : string option; name : string; arguments : string }
type parsed_message = {
content : string;
reasoning_content : string;
tool_calls : tool_call list;
}
Parsing a raw text reply splits it into the visible content the user will see,
the model's hidden reasoning content, and any tool calls it
emitted along the way. A tool_call is just a string name plus its arguments
as a JSON string, with an optional identifier to pair the result back up.
The other three types are knobs on how the model replies rather than what it returns:
- a
thinking_modeofChatanswers directly whileThinkingreasons in a<think>block first reasoning_effortturns that reasoning up to maximum at the cost of slower inference timetaskis a quick-instruction hint into DeepSeek's internal pipeline as to whether this turn is anAction, aQuery, etc.
3.1 Making custom tool functions in OCaml
We now need to define specific tools that the model knows about. We do this by defining a bidirectional codec that decodes the model's JSON/DSML arguments into a typed OCaml value, and renders that value back. Here's a simple touch tool:
let touch =
let open Dsml.Codec in
Invoke.map "touch" (fun path -> path)
|> Invoke.param ~enc:Fun.id "path" string ~description:"file to create"
|> Invoke.seal
in
Tool.v ~description:"Create an empty file." touch
(fun path ->
Out_channel.with_open_text path ignore;
"created " ^ path)
We just wrap the effect we are doing (in this case, just writing an empty file) with the JSON metadata to let the model know when and how to invoke the tool as it works its way through the prompt. Unlike most policy languages, we describe these in plain text since we're dealing with an LLM, and it decides when the description of the tool should be applied in the conversation.
There's still no state here, but we can now use the DSML library to build up a full prompt string by keeping track of all the messages:
val encode_messages :
?context:message list ->
?drop_thinking:bool ->
?add_default_bos_token:bool ->
?reasoning_effort:reasoning_effort ->
thinking_mode ->
message list ->
string
(** [encode_messages mode messages] renders the conversation to the prompt
string. [context] prepends an already-encoded prefix and suppresses the
leading token; [drop_thinking] (default true) drops reasoning from turns
before the last user message; [reasoning_effort] [Max] maximises reasoning.
*)
4 Adding state to make an agentic OCaml library
We're now ready to add state to this! An "agent" is just a wrapper around the LLM that does three things:
- remember the conversation so far for
encode_messages - keep the engine's KV-cache warm via a persistent session
- run the tool callbacks as they arrive from the LLM
The loop is expressed as a simple OCaml event datatype:
type event =
| Reasoning of string (* the model's <think> text *)
| Content of string (* a chunk of the reply *)
| Tool_call of Dsml.tool_call
| Tool_result of string * string
| Done
val send : t -> on_event:(event -> unit) -> string -> unit
When the LLM responds each turn, plain text means the turn is Done. Tool
calls get looked up by their name, then run, and the results folded back into
the conversation as tool messages. All the agent function does is just run
this to a fixed point until the model answers with text alone.
4.1 Writing custom tools using OS sandboxing and Eio capabilities
Here's where the unikernel-style magic shows up though. Given that a tool is just something we define ourselves, then we can start to take advantage of OCaml itself! And in particular, I want better security and more fine-grained tool calls that are tailored to my applications and not generic shell scripts that are difficult to sandbox.
Eio is a library built over OCaml 5's effects that follows an object-capability discipline to eliminate ambient authority where possible. In our Toolbox module, we define some example Eio tools:
val read : dir:_ Eio.Path.t -> Tool.t
val write : dir:_ Eio.Path.t -> Tool.t
val dns : net:_ Eio.Net.t -> Tool.t
val bash : proc:_ Eio.Process.mgr -> Tool.t
Notice that each OCaml signature here takes in a capability for that particular tool:
readandwritecan only access the directory you pass as~dirand nothing above it, since Eio uses openat(2) to sandbox the call.dnsreaches the network only because it holds a net capabilitybashspawns processes only because it holds a process manager.
When using these from applications, we can select the exact sandboxing we need, or just write our own tool functions with application-specific logic. This is exactly what humpty agent does, confining the file and shell tools to a workspace directory you pass with --dir:
(* Sandbox the filesystem tools to [workspace] so that tools only have access there *)
Eio.Path.with_open_dir Eio.Path.(fs / workspace) @@ fun ws ->
let agent =
Agent.create engine ~system ~thinking:think
~tools: [
Toolbox.read ~dir:ws;
Toolbox.write ~dir:ws;
Toolbox.dns ~net;
Toolbox.bash ~proc; ] in ...
The fs, net and proc values all come from Eio's stanard environment, and
its now up to the programmer to decide how to dole them out.
If you want a read-only agent, then just drop write and bash.
Since a tool is just a Tool.v
function over a codec and a handler, callers can add their own depending on the business
logic of the app. For example, I've now got some tools in this webserver that query the
size of the connection pool, another that can search the in-memory logs, and so on.
4.2 Managing deterministic and reproducibility
One of the advantages of running a local model is that we have better control over the determinism of the inference. Aside from obvious factors like the weights and the inference code being the same, we normally still get different results from GPU based inference due to the parallelism.
However, if we don't mind taking a slowdown, the CPU backend here supports saving and passing in the same seed:
$ humpty-cpu chat 'tell me a joke about camels in one sentence' -v
humpty-cpu: [INFO] model: DeepSeek V4 Flash (vocab 129280)
humpty-cpu: [INFO] seed: 1690400691090126 (random; pass --seed N to reproduce)
Why did the camel cross the desert? To get to the other hump-side!
$ humpty-cpu chat 'tell me a joke about camels in one sentence' -v
humpty-cpu: [INFO] model: DeepSeek V4 Flash (vocab 129280)
humpty-cpu: [INFO] seed: 1690392111533869 (random; pass --seed N to reproduce)
Why did the camel cross the desert? Because it was sick of the hump-drum of the same old sand dunes.
$ humpty-cpu chat 'tell me a joke about camels in one sentence' --seed 1690400691090126
Why did the camel cross the desert? To get to the other hump-side!
I was surprised, however, to find that the deterministic seed also worked in the Metal backend!
$ humpty-metal chat 'tell me a joke about camels in one sentence' -v
humpty-metal: [INFO] model: DeepSeek V4 Flash (vocab 129280)
humpty-metal: [INFO] seed: 2042575750328474 (random; pass --seed N to reproduce)
Why did the camel cross the desert? Because it was tired of the hump-drum of its daily routine.
$ humpty-metal chat 'tell me a joke about camels in one sentence' -v
humpty-metal: [INFO] model: DeepSeek V4 Flash (vocab 129280)
humpty-metal: [INFO] seed: 2043271538873965 (random; pass --seed N to reproduce)
Why do camels never get stuck in traffic? Because they have built-in "hump" day passes.
$ humpty-metal chat 'tell me a joke about camels in one sentence' -v --seed 2043271538873965
humpty-metal: [INFO] model: DeepSeek V4 Flash (vocab 129280)
humpty-metal: [INFO] seed: 2043271538873965
Why do camels never get stuck in traffic? Because they have built-in "hump" day passes.
5 Implications of using LLMs as a library
We've seen so far that the LLM model can be exposed as a mostly pure function; and that a tool is a function whose type says what it may touch; and that an agent is just a loop over them.
Crucially, there's no need for serialisation protocols, REST APIs, MCP authentication and many of the other layers built over them unless the application wants it. One of the big advantages of unikernel-style libraries is that necessary functionality can be specialised at compile-time much more easily.
5.1 Building a safe bastion using Dikjstra monads or refinements
This library approach is orthogonal to the idea of "language integrated" LLMs, which involve discharging verification conditions by having LLMs attempt to synthesise proofs of statements embedded within the source code. Ranjit Jhala observed that:
"Language integrated" is a drum I've been beating on for a while (e.g. with refinement types), but in the age of LLMs I wonder if it really matters, if the AIs are going to also be generating the proofs? -- Ranjit Jhala, June 2026
Yaron Minsky noted that "language-integrated assertions and modular specification look like a form of intelligence amplification for whatever is doing the proofs". I also found Shriram Krishnamurthi and Flatt's "Type-Error Ablation and AI Coding Agents" paper that found richer type errors help an agent fix code, and that a type system helps it more than test failures do.
All this means is that we needn't stop at just using types to encode our safety properties. Since each tool is an ordinary OCaml function, nothing stops them being synthesised or checked by a proof assistant like Fstar, so that a tool can ship with a machine-checked guarantee that it stays inside the policy its signature advertises.
We explored this last year in a HOPE abstract via Dijkstra monads, since they let you reason precisely about the effects a computation is allowed to have. Patrick Ferris Cyrus Omar and I sketched how to modularise some more dependently typed policy reasoning in Bastion. Eio's capabilities are really just a lightweight version of the sorts of things you can express in a full proof framework, and as Ranjit notes above, LLMs make it easier than ever to just pick the right specification language for the problem at hand.
Another obvious direction to take my library-agentic-harness is to link it in with OCaml's compiler-libs to build a much more tightly integrated agent debugger that doesn't need to go via CLI tooling!
5.2 Letting a service watch...and mutate...itself
What else could we do with an LLM we can call as cheaply as any function, on hardware you own? (I'm assuming we have spare CPU/GPU here!)
The experiment I'm mid-way through is wiring this into my zero-allocation OxCaml webserver, which already emits custom runtime events alongside OCaml's native GC and scheduler ones. Normally they pile up in a ring buffer nobody reads until something's already broken. The idea is to spend idle CPU on them, so when the server isn't busy, I hand a window of recent events to my agent function in a separate domain and check to see if the latency is drifting, or if that event is firing far more than yesterday (based on summary stats), and so on. Whether a local model is any good at this is something I don't know yet, but I'm enjoying experimenting.
As we discussed in the rewilding the web workshop and our Internet ecology paper, self-hosted services unfortunately don't just run themselves and need tending to. I could therefore imagine building enough local tooling so that the agent harness would recompile and re-exec the binary autonomously in response to external stimuli...
The code's here if you'd like to pull it apart. I'd love to hear more about any improbable stunts or agentic harnesses you build yourself! In the future, I'll investigate expanding this out to CUDA and also beyond just Deepseek.