# Was software just always too complex for humans?

*2026-11-22 — note*


I was [glued to the Lean chat](https://leanprover.zulipchat.com/#narrow/channel/113488-general/topic/Discussion.3A.20Alp.C3.B6ge.20Proves.20the.20Riemann.20Hypothesis) as the legendary [Riemann hypothesis was cracked](https://en.wikipedia.org/wiki/Riemann_hypothesis#Alpöge_alleged_proof) via 17 million (!) lines of AI-assisted code last week. This has since been [certified correct](https://leanprover-community.github.io/blog/posts/riemann-hypothesis/) despite [no human (yet) understanding](https://www.science.org/content/article/ai-may-have-solved-math-s-greatest-mystery-can-humans-understand-answer) how it works.
The future of maths, and much science, is clearly going to involve human-AI boosting.

However, it also struck me deeply just how broken software engineering has been for my entire life\!
There's always been an imbalance between our understanding of the theory of computation and our relentless drive to deploy software throughout society. We're paying the piper now as all those decades of human-written code turn out to be full of bugs. Since software essentially drives society now, weird things happen like [supermarket fridges needing James Bond to secure them](https://www.bbc.com/news/articles/c6eq49zyx7p4o). This has led to people like [Tim Gowers joining](https://x.com/wtgowers/status/2124287061620613160) the [Pause AI march](https://www.bbc.co.uk/news/articles/cmvg23q5xq31o).

How the hell did we get here? Have we just completely overestimated the human ability to write software? Or will software engineering 'catch up' once LLM harnesses get better?

Software was reasonably under control in pre-AI times when attackers were roughly symmetric with defenders, but all the recent [mysteriously correlated outages](https://www.bbc.co.uk/news/articles/c6z0kly2lll7o) and disclosed issues ([Java](https://cyberscoop.com/java4shell-critical-java-flaw-openjdk/), [PHP](https://en.wikipedia.org/wiki/PHP2Shell), [Linux](https://openai.com/index/a-thousand-fewer-paths-to-root/)) all point to an [acceleration](https://anil.recoil.org/notes/rumour-is-the-exploit) of security pain in 2027.
I only see two ways this can go...
	
<a href="https://x.com/headinthebox/status/2124248546157432852"> <figure class="image-center"><img src="/images/maths-vs-cs-loc.webp" alt="The difference between mathematics and computer science, via Erik Meijer" title="The difference between mathematics and computer science, via Erik Meijer" loading="lazy" srcset="/images/maths-vs-cs-loc.768.webp 768w, /images/maths-vs-cs-loc.640.webp 640w, /images/maths-vs-cs-loc.480.webp 480w, /images/maths-vs-cs-loc.320.webp 320w, /images/maths-vs-cs-loc.1440.webp 1440w, /images/maths-vs-cs-loc.1280.webp 1280w, /images/maths-vs-cs-loc.1024.webp 1024w"><figcaption>The difference between mathematics and computer science, via Erik Meijer</figcaption></figure> </a>

## Derive everything from specifications

In April this year, I [couldn't believe](https://anil.recoil.org/notes/internet-immune-system) we'd ever formally specify every layer of hardware and software:

> I find it hard to believe that we will **formally specify all layers of hardware
> and software and roll them out into production**...ever. [A Fire Upon the Deep](https://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep)'s
> software archeologists are a profession that's [here today](https://www.indeed.com/jobs?q=Cobol&fromage=14&vjk=6773c80abba24067) and not a scifi trope.
> <cite>\-- [The Internet needs an antibotty immune system, stat](https://anil.recoil.org/notes/internet-immune-system), April 2026</cite>

How a few months makes a difference. With the pace of frontier model advances, translating the vast body of existing legacy code seems entirely plausible _if_ we first speedrun towards defining formal specifications for all of our services, and derive _all new_ software from those as well.

That means rebuilding operating system stacks from the ground up and even [reengineering instruction sets](https://architectlabs.com/blog/frostbite) to match. There are [terrible legacy interfaces](https://x.com/yminsky/status/2123949643121696822) throughout the stack, largely because our software layers grew so that human groups could scale working on different parts. Other approaches do make it easier though. [Unikernels](https://anil.recoil.org/projects/unikernels) with a [libOS](https://anil.recoil.org/papers/2013-asplos-mirage) linking discipline plus [shared memory](https://youtu.be/abDWZ9D8kEE?list=PLyrlk8Xaylp4ZOkYWR5676Qxd22Y5f9xv) interconnects are [remarkably composable](https://anil.recoil.org/notes/icfp25-post-posix) and allow agents to specialise the whole stack at link time, with fewer interfaces making the [specification task less daunting](https://anil.recoil.org/ideas/lean-io-uring-backend).

Having good specs also conveniently takes humans out of the security response patch loop. Attackers are currently bounded only by their desire to break in somewhere, while defenders have to wait on [maintainer triage](https://anil.recoil.org/notes/rumour-is-the-exploit). However, fixes re-derived from a formal spec and machine-checked could work more reliably in a timely fashion. The human job remains [writing the interfaces](https://anil.recoil.org/notes/ocaml-modules-agentic) iteratively with an AI.

## Engineer diversity in deliberately

Historically, software has tended towards a monoculture through reuse, so [one bug](https://therecord.media/cisa-issues-java4shell-alert) can screw over vast numbers of hosts. Agents could instead force diversity via novel ['genetic' mechanisms](https://anil.recoil.org/papers/2025-internet-ecology) so digital services aren't all simultaneously infected. This works well with formal specifications, since any decent spec allows [multiple valid implementations](https://anil.recoil.org/papers/2015-sosp-sibylfs).  The Internet's entire philosophy is based around [Postel's Law](https://en.wikipedia.org/wiki/Robustness_principle) and encouraging hosts to be liberal in what we receive and strict in what we transmit.

The catch, of course, is having just a handful of [gated frontier models](https://www.anthropic.com/glasswing) generating all our software takes us out of the frying pan into an AI controlled fire.  The diversity has to extend _beyond_ the code to the models and harnesses too. We seem to have done ok with [Internet RFCs](https://www.rfc-editor.org/) not splintering too much, so perhaps formal specs can remain singular. Once that's done, it's time to go let hosts [mutate and inoculate](https://anil.recoil.org/notes/rewilding-the-web-report) each other and generally maximise unpredictability at scale\!

## Digital society is going to have to change

To answer my opening question, I don't think better harnesses and models _alone_ will let software engineering catch up, as they just produce mountains of slop code. We need _both_ of the above; formal specifications that let hosts finally be strict about what they receive instead of liberally accepting whatever shows up, while diversity ensures that things that the specs (inevitably) don't capture doesn't break everywhere, everywhere all at once.

The real upheaval here will be to social coding though. Open source communities have run on social contracts of human review and trust built up over decades, and every single one I'm a member of is struggling under matters of [AI disclosure](https://anil.recoil.org/notes/opam-ai-disclosure-update) and [trust](https://blog.tangled.org/vouching/).

At least humans aren't out of a job yet though\!
Somebody still has to decide what our specs contain, as we have done for decades with RFCs. There's also plenty of new computer science research needed! We need to figure out how to [debug diversity](https://anil.recoil.org/ideas/antibotty-testbed) when no two nodes run the same code, and on how to force simplicity on systems that agents maintain, and ultimately how we settle the blame games when things do go wrong...

<small class="credits">(Thanks to [Dave Scott](https://dave.recoil.org), [KC Sivaramakrishnan](https://kcsrk.info), [Thomas Gazagnaire](https://github.com/samoht) and [Sadiq Jaffer](https://toao.com) for helpful conversations about this post, and to [Yaron Minsky](https://github.com/yminsky) for triggering this line of thought with his eBPF observations)</small>

## References

1. Madhavapeddy et al (2025). Steps towards an Ecology for the Internet. Association for Computing Machinery. <https://doi.org/10.1145/3744169.3744180>
2. Madhavapeddy (2026). OCaml's module language is the perfect fit for agentic programming. <https://doi.org/10.59350/g6hbj-qmc76>
3. Madhavapeddy (2026). Just a rumour of a bug is enough to find a security exploit these days. <https://doi.org/10.59350/tngsm-6rx23>
4. Madhavapeddy (2025). It's time to go post-POSIX at ICFP/SPLASH 2025. <https://doi.org/10.59350/mch1m-8a030>
5. Madhavapeddy (2026). Rewilding the Web: my workshop report from Edinburgh. <https://doi.org/10.59350/g40yy-ks003>
6. Madhavapeddy (2026). The Internet needs an antibotty immune system, stat. <https://doi.org/10.59350/snnnf-asc02>
7. Ridge et al (2015). SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems. ACM. <https://doi.org/10.1145/2815400.2815411>
8. Madhavapeddy et al (2013). Unikernels: library operating systems for the cloud. ACM. <https://doi.org/10.1145/2451116.2451167>

Synopsis: Musings on whether we (as a species) royally screwed up by trying to run before we could walk with software in society
Words: 887

## Related

- [OCaml's module language is the perfect fit for agentic programming](https://anil.recoil.org/notes/ocaml-modules-agentic) (note, 2026-10-09)
  Write interfaces first, statically type check the design, then hand each module to an agent one at a time. OCaml's module language makes this a natural workflow for future formal methods too.
- [Just a rumour of a bug is enough to find a security exploit these days](https://anil.recoil.org/notes/rumour-is-the-exploit) (note, 2026-08-22)
  Thinking through how the conventional OSS security embargoes no longer buy us time, and what open source maintainers might do instead to respond
- [An antibotty defensive testbed to protect network services](https://anil.recoil.org/ideas/antibotty-testbed) (idea, 2026-08-01)
  Available · MPhil
- [Rewilding the Web: my workshop report from Edinburgh](https://anil.recoil.org/notes/rewilding-the-web-report) (note, 2026-05-30)
  Notes from a wonderfully interdisciplinary Edinburgh workshop on 'Rewilding the Web', ranging coopetition and biological variety through the philosophy of self-organisation, polycrisis governance, protopian science fiction, and moderation seen through the lens of artisanal cheese.
- [Voluntary AI disclosure proposal for OCaml: update 1](https://anil.recoil.org/notes/opam-ai-disclosure-update) (note, 2026-05-14)
  An update on the voluntary AI disclosure proposal, digesting the security, quality and legal feedback, and some concrete next steps around maintenance intent, multi-repository tooling, and reputation.
- [The Internet needs an antibotty immune system, stat](https://anil.recoil.org/notes/internet-immune-system) (note, 2026-04-08)
  Anthropic's Mythos makes autonomous vulnerability chaining across devices a sudden reality, so I've been thinking about how digital 'antibotty' inoculation networks may be needed far sooner than I expected.
- [An io_uring IO implementation for Lean](https://anil.recoil.org/ideas/lean-io-uring-backend) (idea, 2026-04-01)
  Ongoing · Part II
- [It's time to go post-POSIX at ICFP/SPLASH 2025](https://anil.recoil.org/notes/icfp25-post-posix) (note, 2025-10-08)
  VMIL keynote arguing for post-POSIX shared memory interfaces like io_uring in language runtimes for high-performance concurrent computing.
- [Steps towards an Ecology for the Internet](https://anil.recoil.org/papers/2025-internet-ecology) (paper, 2025-08-01)
  Anil Madhavapeddy, Sam Reynolds et al. — Proceedings of the sixth decennial Aarhus conference: Computing X Crisis
- [SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems](https://anil.recoil.org/papers/2015-sosp-sibylfs) (paper, 2015-10-01)
  Tom Ridge, David Sheets et al. — Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP)
- [Unikernels: library operating systems for the cloud](https://anil.recoil.org/papers/2013-asplos-mirage) (paper, 2013-03-01)
  Anil Madhavapeddy, Richard Mortier et al. — Proceedings of the 18th international conference on architectural support for programming languages and operating systems (ASPLOS)
- [Unikernels](https://anil.recoil.org/projects/unikernels) (project, 2010-01-01)

---
Canonical: https://anil.recoil.org/notes/software-too-complex-for-humans
Type: note
License: CC BY 4.0 <https://creativecommons.org/licenses/by/4.0/>
Tags: ai, security, verification, unikernels, llms
