← Back to writing
OS Kernel Museum 9 min Part 8 of 8

What Six Kernels Taught Me About Software

The meta-lessons from writing six operating system kernels that apply to all software engineering — scope, tooling, layers, rewrites, and the value of incomplete projects.

Six kernels. Thousands of files. Five years. Zero useful programs running on any of them.

Here’s what they taught me about building software — not OS-specific lessons, but the kind that change how you approach any engineering problem.

1. Build the Tools First

Every kernel project spent more time on the toolchain, build system, and libc than on actual kernel features. This felt wrong at the time — I wanted to write kernel code, not configure GCC. But the toolchain investment paid compounding returns.

When Voleta’s naked toolchain was solid, writing kernel code became fast. When the sysroot was correct, linking worked on the first try. When the libc had good string functions, debugging output was easy.

The general version: Before you build the thing, build the environment that builds the thing. CI/CD pipelines, development containers, linting configurations, test harnesses — these feel like distractions from “real work,” but they multiply the productivity of every hour spent afterward. The toolchain is not overhead. It’s leverage.

2. Scope to What You Can Debug

KairosOS aimed for a Lua desktop environment. I could barely debug memset. IsmenaOS aimed for a Swift runtime. The Swift runtime requires POSIX semantics I hadn’t implemented.

Both projects failed at the gap between ambition and infrastructure. The kernel code was fine. The scope was impossible given the foundation.

The general version: The right scope for a project isn’t “what would be cool to build.” It’s “what can I test and fix when it breaks.” If you can’t debug the failure mode, the feature is out of scope regardless of how simple it seems. A web app with “just add authentication” is KairosOS with “just add Lua” — the complexity is in the infrastructure the feature requires, not the feature itself.

3. Understand Every Layer You Depend On

The assembly experiments were the foundation for everything. Writing ELF headers by hand made the ELF loader possible. Understanding the boot sequence in raw assembly made C kernels debuggable. Knowing what syscall actually does at the CPU level made designing a syscall ABI natural.

The general version: If there’s a layer in your stack you don’t understand, that’s where your bugs will be hardest to fix. You don’t need to implement every layer — but you need to understand the interface well enough to reason about behavior when things go wrong. “It works on my machine” is the developer equivalent of “the kernel boots in QEMU” — true but insufficient.

4. Rewrites Are Strategic, Not Wasteful

Each kernel wasn’t “starting over.” It was applying the previous kernel’s lessons with the previous kernel’s sunk costs removed. KairosOS v1 → v2 was a refinement. KairosOS → Voleta was a scope change (toolchain focus). Voleta → AstraeaOS was a complexity reduction. AstraeaOS → TerranoxOS was a synthesis.

If I’d stuck with KairosOS and tried to iterate it into TerranoxOS, I’d still be fighting a build system from 2020 and a libc architecture that was designed before I understood libc architecture. The rewrites let each kernel have the right foundation for its goals.

The general version: A rewrite isn’t “throwing away work.” It’s “starting with better decisions.” The code is gone, but the knowledge isn’t. The second system doesn’t have the same problems as the first because you designed around them. The cost of a rewrite is re-implementation. The cost of not rewriting is carrying every bad decision forever.

But: rewrite strategically. KairosOS v1 → v2 (same project, refined) was valuable. A sixth version of KairosOS with the same architecture would have been waste. The signal to rewrite is “I’d make a different architecture decision” not “I’d write the same code cleaner.”

5. The Incomplete Project Is Still Valuable

None of these kernels run anything useful. None of them will. TerranoxOS might eventually have a complete syscall implementation, but it will never compete with Linux, and it shouldn’t try to.

The value was never the kernel. The value was what building the kernel taught me:

  • Docker containers are just namespaces + cgroups. I’ve built the things they abstract over. When a container networking issue arises, I debug it at the syscall level, not the Docker CLI level.

  • Syscalls are an API I’ve designed 91 of. When I use strace to debug a production issue, I’m reading a protocol I’ve implemented.

  • Memory management is something I’ve built from physical page allocators to virtual memory mapping. When a production app has a memory leak, I understand what “leak” means at every layer from the heap to the page tables.

  • Security at the lowest level — capabilities, privilege rings, access control — informed how I think about application security. NIST 800-53 controls make more sense when you’ve implemented the mechanisms they control.

The general version: A project doesn’t need to ship to be valuable. The side project that teaches you distributed systems, even if nobody uses it, makes you better at every production distributed system you touch. The cost is your time. The return is depth that production work alone can’t provide.

How OS Development Changed My Day Job

At GDIT, I worked on Win32 C++ supply chain software. The kernel projects didn’t directly apply — I wasn’t writing an OS there. But the systems thinking does:

When a Win32 service monitor needs to watch SCM events, I think about it as a syscall interface. When a static analysis pipeline needs to parse PE binaries, I think about it as an ELF loader for a different format. When supply chain security needs capability-based access control, I think about TerranoxOS’s token model.

The kernels gave me a mental model of how computers work from the bottom up. Every abstraction I work with in production — processes, memory, files, network sockets — maps to something I’ve implemented or designed from scratch. The abstraction isn’t a black box. It’s a familiar mechanism with known tradeoffs.

The Honest Ending

I’ll probably write a seventh kernel. Not because TerranoxOS is finished — it isn’t — but because each kernel answers the question I had when I started it, and each answer generates new questions.

The assembly experiments asked “what does a computer actually execute?” KairosOS asked “can I build a real kernel?” Voleta asked “can I build the tools properly?” IsmenaOS asked “can I host a modern language?” AstraeaOS asked “what’s the minimum?” TerranoxOS asks “can I design a coherent system?”

The seventh kernel will ask something I can’t predict yet. That’s the point.

This is Part 8 of 8 in the OS Kernel Museum series. The beginning is Part 1 — Why I Keep Writing Kernels. The museum repository is at os-kernel-museum.