Lalit Maganti (Articles only)

The git history command deserves more attention

Mon, 13 Jul 2026 19:40:00 +0100

Working with lots of changes in parallel on git can be painful. You end up juggling branches and commits, and running scary rebase -i commands that can leave your tree in a half-broken state if you so much as sneeze.

jj, an alternative to git, gets discussed a lot these days (1, 2, 3, 4) and is often pitched as a solution. While I’m very sold on the problems jj is trying to solve, the way it solves them hasn’t quite hit home with me. Every 3 months, for the last 1.5 years, I try it out for a few days, really trying to make it part of my workflow but eventually I give up and go back to git.¹

That’s where git history comes in. It’s an experimental command that arrived across two releases, 2.54 (April, reword and split subcommands) and 2.55 (June, fixup subcommand). It got a flurry of attention on each release day, and then, as far as I can tell, not much community discussion since. Which is a shame, because IMO it already delivers several of the benefits people tout for jj without needing to switch your whole workflow. And the cool thing is that it’s part of the core git distribution, so you can try it without installing anything.

There are three subcommands: fixup, reword and split.

fixup

git history fixup fixes an old commit that has something wrong in it, then autorebases all your branches to match.

You stage the fix as usual with git add, then run git history fixup <commit> to fold those staged changes into the target commit. It’s like a git commit --fixup plus an autosquash rebase but with the extra magic that it also updates any other branch which contained that commit.

That last part goes further than git rebase --update-refs, which only moves refs sitting inside the range you’re actively rebasing. git history instead finds and rewrites every local branch descended from the commit (while also having an option to limit it to only the current branch). On the other hand it does not work in the presence of merge commits which, for some usages of git, is going to be a dealbreaker.

Here’s how it works in practice:

Before, with a fix staged for B:

After git history fixup B:

B* is B with the fix folded in. Rewriting a commit gives it a new hash, so C and D are automatically re-created on top as C* and D*, and the feat-1 and feat-2 branch tips move with them.

The most important property, common to all three commands, is that it’s atomic: it never leaves your tree in a half-broken state. It manages this by refusing any operation that could produce a conflict.

To be clear, this is strictly less powerful than jj. jj treats conflicts as first class so it can carry a conflicted state through a rebase and let you sort it out later. git history doesn’t do this yet but the docs leave the door open:

“This limitation is by design as history rewrites are not intended to be stateful operations. The limitation can be lifted once (if) Git learns about first-class conflicts.”

So basically, this limitation may change in the future; excited to see if it does!

reword

git history reword updates the commit message on an old commit and automatically rebases everything on top. This is very useful for going back and fixing commit messages when the design shifts as you iterate.

git history reword <commit> opens your editor with that commit’s existing message. You edit it, save, and the rest of the stack is rebuilt on top with the branches following along. It’s exactly like fixup but for commit messages instead of the tree contents.

Because it only changes a message, reword (like split later) never touches your index or working tree at all; it works purely on the commit graph. So both let you rewrite a commit on a branch you don’t have checked out without disturbing whatever you’re in the middle of.

Before:

After git history reword B:

Only B’s message changes, but that still gives it a new hash, so C is rebuilt on top as C* and feat-1 follows along.

split

git history split takes one commit and splits it into two, interactively picking what you care about from each. It’s the equivalent of git add -p, but without needing gymnastics with git rebase. I’ve found this to be the most specialized of the three, but invaluable when I need it.

Specifically, git history split <commit> drops you into a hunk-by-hunk prompt over that commit’s diff. The hunks you keep make up the first commit, the rest fall into the second.

Before, with B bundling two unrelated changes:

After git history split B:

B becomes B1 and B2, and C is rebuilt on top of the pair as C*.

Conclusion

Judging by how many people are using jj, I do think there’s still some key mental shift which I’m not yet making. And to be clear, git history doesn’t close the full gap: jj still gives you an operation log with easy undo, models your working copy as a commit, and can carry conflicts through a rebase, none of which this is trying to do.

But for now, git history is a big step forward in adopting many of the pieces that attract people to jj, and it’s already in the tool I use every day. And the way the documentation is written makes me hopeful that more improvements will be coming in upcoming releases!

If there’s interest, happy to write up a post on my experience on this. ↩︎

Security analysis is finally reaching software's long tail

Sun, 07 Jun 2026 16:26:00 +0100

This post was formerly titled “17 bugs in 10 weeks from AI security scanning”. It was changed upon feedback from trusted readers who suggested, while accurate, it did not give a good sense of what I’m actually arguing in the post.

Over the last several weeks, I’ve been receiving more security bug reports for Perfetto’s trace processor than I ever have before, all of them found by AI. And I’m very happy about it! These are bugs that would almost certainly not have been found a year ago and it feels good to close these loopholes even though trace processor is by no means security critical.

For years, security researchers concentrated their time on the highest-stakes targets: kernels, cryptography libraries, password managers. But there’s a lot of code out there which is security-relevant but not truly security-critical. In my experience, these sorts of projects didn’t draw much attention. Now systems in the long tail can get that attention which they wouldn’t have before.

Why is this happening

Trace processor is a project which sits squarely in that long tail. It’s a C++ library (yes, Rust would be the obvious choice today but it’s not practical to rewrite, see footnote ¹) for processing recorded traces of various formats. These are typically traces you collected yourself or in your test infra and process offline so “untrusted input” isn’t much of a concern.

However, some people do process traces they didn’t collect themselves (e.g. user bug reports, automated collection from dogfood users). For those cases we’ve strongly recommended sandboxing trace processor (e.g. gvisor, sandbox2, or minijail) or, for even more sensitive use cases, a VM.

Beyond sandboxing, for catching issues proactively, we mainly relied on fuzzing running internally in Google. These fuzzers occasionally surfaced real, actionable bugs: we set them up to pass in arbitrary trace bytes (as this is the main “attack surface”) but over time these became quite rare as they discovered much of the low hanging fruit, which we quickly fixed. The bugs that remained tend to live deep in the internals, reachable only with a very precisely crafted sequence of bytes that a fuzzer is unlikely to hit by mutation alone.

Apart from this, there has rarely been any bandwidth or resources for a human, either a security expert or someone from my team, to spend lots of time finding security issues in trace processor. There were always other parts of Perfetto more worth spending security time on (e.g. the tracing service, on-device profilers) as they’re actively running in production systems.

All of this changed as of a couple of months ago. We started receiving bug reports filed by some central team which appears to be running AI-based security scanning against various projects throughout Google. Unfortunately, I have to be hand wavy about what exactly they’re doing as their work doesn’t appear to be public.

Starting in early April, we had a slow drip of 1 bug a week, but since the end of April this increased to a rate of several a week, with some days having 3 or 4 being opened in quick succession. This lasted until mid-May, at which point it started tapering back to 1-2 a week with some weeks having none.

I also want to say that the quality of the bugs is high. They’re well-described, often with the relevant attacker model already worked out and even minimal fixes proposed: basically everything I could ask for from a bug report. This matches what both curl and Linux kernel maintainers have noted about security bugs they’ve received, especially how sharply quality has improved in the last few months.

As I can only see the bugs that get filed against me, not the raw output of the AI scanner, I don’t know exactly how much triage happens upstream. My guess is there’s a human doing a light pass to drop obvious noise before reports reach client teams, but judging from the rate at which bugs are opened and the way they’re filed, I doubt anyone is deeply triaging each one.

In total, we’ve received 21 bugs (17 real issues and 4 not actionable), which can be broken down into the following categories:

10 bounds checking: arbitrary trace data flowing into fixed-size buffers or unchecked array indices, leading to out-of-bounds reads or writes.
5 use-after-free: back-pointers, pointer snapshots, or hashmap keys outliving the object they refer to.
1 stack overflow: unbounded recursion when input is deeply nested.
1 access control: not enforcing allowlists on some rare codepaths.
4 closed as not actionable: either where the chance of exploit was purely hypothetical or where fixing would have required fundamental design changes which were not worth the tiny security risk.

All 17 real issues have been fixed, almost all shipping in Perfetto v56.0 ².

How it feels to get a report

How does receiving one of these reports actually feel? Well not as bad as you’d think. Unlike a security-critical application like OpenSSL or curl, in trace processor, a security issue is very unlikely to be a P0 I have to drop everything to fix. Don’t get me wrong, it’s still a priority but one where I have the luxury of taking a few days to figure out the right answer and can release fixes according to our normal schedule, instead of trying to rush out a CVE and get everyone to patch immediately.

Also thankfully, because the majority of the issues are mechanical, the fixes are generally quite straightforward. Take this PR, for example:

We build a key string into a fixed-size stack buffer while parsing some metadata.
The bounds check only runs in debug builds, and the metadata name comes straight from the trace. Putting a long enough name means you would escape the buffer.
The fix is a simple matter of swapping the stack buffer for a std::string. The code path is very cold (only once or twice in a trace) so the extra heap allocation doesn’t matter.

In fact, these sorts of issues are so mechanical that I trust a coding agent to just fix them with minimal guidance: take the well-written report, feed it to the agent, and within ~10 minutes there’s a 10-20 line PR which fixes it. I review every line thoroughly and make sure I understand it, but these tasks are not difficult and firmly inside the “jagged frontier” of what AI can do.

I want to stress though that not every issue is mechanical or can be left to AI; a few reports actually point more to design problems than incorrect function implementations. This use after free is a good example:

The problem was a state object held a back-pointer that could end up pointing to freed memory given certain data appearing in the trace.
The immediate dangling case was easy to patch by just having a callback which invalidated the back-pointer on free. But this is a horrible hack which makes the lifetimes of the objects involved impossible to reason about.
The real problem here is that you had a child object whose parent could go away before it, which really shouldn’t happen if this code is properly architected.
Fixing it properly meant restructuring the ownership model so the lifetime was correct by construction.

The interesting thing was that this was a problem I was aware of and that I had been meaning to clean up for close to a year but never got round to: the security bug just gave me the push and justification to do it. This applied in a couple of other bugs as well and made me internalize that security issues can sometimes be correlated with deeper design flaws or hacky code so there are wider benefits to “security scanning” than just the direct bugs they find.

Will this last?

One thing I am wary of is how long this stream of bugs will keep up; I’m feeling good about it given it’s only been going on for a couple of months, but I can easily imagine that if this goes on for several more months, it might become mentally exhausting.

But my suspicion is that this will go to zero. Why? It’s to do with the pattern of how these bugs are being filed. Each part of the codebase seems like it’s getting a day or two of attention (and associated bugs) before moving on to a different part. Repeats are rare, and the pace of bugs has slowed especially in the last couple of weeks: we had a lot more in the start of May (several a week) but now we’re down to 1-2 a week. There are a finite number of files, so eventually my gut tells me they will run out.

An important consideration is whether we’ll add new bugs faster than the scanner can find old ones. My suspicion is no; the 17 real issues so far are from scanning across 9 years of development. Even if that number triples before things settle, the scanner is still working through years of accumulated code. And we wrote a lot more code, a lot faster, in the earlier years of the project, so the rate of new code being added now is lower than it once was.

The other question is whether new model releases will find more complex design issues rather than the simple issues we’re finding today. Those take significantly more time and effort to fix and so would be a lot more painful if we were to get many of those. I’m very unsure on this so we’ll just have to wait and see!

Where this leaves us

I feel Daniel Stenberg (curl maintainer) phrased it well in this post:

Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools.

This rings very true to me. More broadly, I think folks will have one of three experiences:

Untrusted input + security critical (e.g. curl, kernel, OpenSSL): many complex reports, with a higher false positive rate than the other categories, because there’s a lot of attention on the project and much of the low hanging fruit would already have been picked in the critical codepaths. Though codepaths for lesser-used functionality (e.g. legacy drivers) could end up in category 2 instead.
Untrusted input + not previously audited (e.g. trace processor): a wave of mechanical bugs at a manageable pace and low individual stress because the project is not on a security critical code path. This is where both Daniel and I expect AI security scanners to have the most impact.
No untrusted input (internal tools, math libs, anything operating only on trusted data): you probably won’t notice this shift at all.

My own case sits squarely in that second bucket. But I don’t want to over-generalize from my experience, because there are three things that make this manageable for me that wouldn’t be true for everyone: a) I’m paid to maintain trace processor as part of my full time job; b) someone else is taking the effort to run the AI scans and discover the bugs in the first place; c) the reports appear to be lightly filtered by an upstream human reviewer, enough to strip obvious noise but probably not a deep triage.

To me, this points to a gap in the ecosystem: most open-source projects cannot afford to have a dedicated team doing security scanning for them, and telling a maintainer to stand up their own pipeline when their security risk is marginal will restrict this to only the most motivated projects. I would guess we’re going to see a lot more innovation in this space, including from the big AI labs.

All in all, I’m cautiously positive about my own experience: most of the bugs are mechanical, a few have nudged long-overdue design cleanups, and the pace is manageable. There’s plenty I don’t know about how this evolves: whether the pace holds, whether future models start finding harder design issues. So this should very much be treated as a snapshot, not a forecast!

A common response I expect is: “if it’s parsing arbitrary binary data like traces, it should be written in Rust.” In a vacuum I agree and if I was writing trace processor from scratch today, I would definitely use Rust. But switching to Rust is unfortunately quite impractical; the library is a significant amount of code and is embedded in hundreds of downstream tools, many in environments that don’t have a Rust toolchain. Asking all our embedders to start using Rust would be a significant burden and one I don’t want to impose. Not to mention that our team doesn’t actually have any Rust expertise so reviewing this code at a standard I want from trace processor would be a significant productivity hit until folks got up to speed. And unlike others in the industry, I don’t feel comfortable just rewriting the whole project in one shot and calling it a day… ↩︎
The couple of remaining bugs were found after the v56.0 release was cut and are low-priority enough that it’s not worth rushing out a release for them. They have already been fixed on main and will be fixed in an upcoming point release. ↩︎

Don't answer the first question

Sat, 16 May 2026 14:19:00 +0100

In my work on Perfetto, a performance debugging tool, one question I get often is: “how do I split a Perfetto trace into multiple files?” Instead of answering directly, I say: “there isn’t an easy way to do that, but what’s leading you to collect traces large enough to want to split?”

This is one of my golden rules at work. When a user asks me something “weird”: don’t answer the first version of the question.

On the surface this might appear like I’m talking about the XY problem, but that stops one step short. It treats the user’s stated question as a puzzle to decode: figure out what they really meant, answer that, move on. I think we can go much further.

Instead, the confusion that produced the wrong question is itself an opening, and the conversation it sparks is valuable to both sides. The user walks away with a better mental model of the tool. I walk away with a clearer picture of where the product confuses people. And sometimes, between us, we figure out that the product itself needs to change.

I’ve written before about how I can still be a successful engineer while avoiding the spotlight. While that covered the general strategy, this is one of the concrete tactics that makes it work. I should also say this post is aimed at people who build things for other engineers. If you’re building a consumer product, or a B2B service, it will translate less directly, but the underlying instinct might still be useful.

Diagnosing the ask

Some questions are easy, routine, and purely a matter of pointing at documentation; those don’t merit much discussion here. The interesting cases are where something is out of the ordinary, and it’s rare that the user will have given me enough information in their first ask¹.

So I run a mental checklist to figure out where to go next:

Have I seen this before? If so, I might already have an answer to hand. If not, it’s uncommon enough that I want to slow down.
Does the question even sound reasonable compared to others I’ve seen? If not, why might they be asking it, and is there a more normal question underneath?
Does it fit the shape of the tool? Or is the user fighting the architecture without realizing it?

Once I’ve figured out what feels off, the next step is asking something that will surface the missing context. I might say something like “well the answer to your immediate question is X but that’s a pretty strange thing to ask for because of reason Y. Can you tell me more about the wider problem you’re trying to solve?”

This will probably be the start of a back and forth. How quickly it moves depends on how well the user can communicate their thoughts. But we’ll usually end up in one of a few places: they’re missing the philosophy of the tool, the product is hiding the right path or the product itself needs to change.

When they’re missing the philosophy

It’s quite common for users to come to us not knowing what they want, or not understanding the problem they’re trying to solve.

To be clear, I’m not criticizing them for this; teams are often trying to solve problems with limited time or resources, and they turn to new debugging tools when they’re struggling to make progress. As a result, they often find the tool, find it does most of what they want but doesn’t match their model of “how it should work”. So they file a feature request.

A common version of this: people come to Perfetto, see that a trace is a highly detailed recording of what a device did over a window of time, realize you can compute metrics from a Perfetto trace, and treat it as a holy grail solution to all their problems. Want a frame rate? Count frames in the trace. Memory used by an app? Look at the allocations and frees. In principle, any metric could be computed from a trace.

But this is a bad idea for a simple reason: traces are expensive to collect and process: you’re collecting all the data about the system rather than samply a single number. You’re going to waste a lot of resources when instead, a dedicated metric collection system would do the job much more efficiently².

My overarching point is that there’s a certain philosophy to how tools are designed, and users often miss it because they’re focused on their immediate problem.

A big part of my job is teaching the team how to approach performance engineering in the first place, not just explaining how to use Perfetto. It means making people aware of the tools they have available, how to think about things like startup, frame drops, memory, and power, and how to work with them both in normal situations and when something goes wrong.

When the right path is hidden

Other times the team understands the problem; they just don’t see how to put existing tools together. Our tools are powerful by design, and we have to be mindful that other teams might not understand the full range of what we’ve built. It’s my job to figure out what they actually want. Often, something we built for a different purpose can be repurposed to meet their needs.

A perfect example here is what I already discussed: trace splitting. The conversation goes something like, “…what’s leading you to collect traces large enough to want to split?” They say it’s because they have periods of interest in a long trace and want to slice it up. Partly for performance, partly to make visualizing easier.

But then I point out that Perfetto already supports periodic trace snapshots, short repeated recordings instead of one long one, which removes the need to collect a long trace at all. They’re trying to solve a problem they shouldn’t be having in the first place.

It’s always satisfying to see people say “that’s exactly what I needed!” even though it’s not what they asked for. It means I successfully figured out what they actually wanted rather than what they thought they did.

When the product needs to change

Occasionally, the response reveals something genuinely new, something that could set us on the path to building something big. These cases are tricky: even when the ask is novel, the asker often can’t tell you what they actually need.

The cost of getting things wrong in foundational software is high, so I err on the side of not building something until not having it hurts: multiple teams have come to me saying “we want this.” Ideally by then we’ve found the essence of what’s actually worth building. This is very unlikely to happen after just one ask, so waiting is really powerful.

We’ve made this mistake at Perfetto. Take ad-hoc UI customization. People wanted to hack on the UI to suit their workflows, and they kept complaining about how hard that was. So we let them, and immediately took on a huge amount of technical debt. Every new feature had to interact with every existing one, and the whole thing quickly became impossible to scale.

It took us roughly one year to dig ourselves out of this hole by designing a proper plugin API. The real need, which no one could articulate up front, was a way to personalize the UI to the needs of their team or use case without affecting every other user. But of course no one was able to name this until it was too late. So it was our responsibility to spot this early as the requests flowed in.

Allowing folks to “merge” Perfetto traces was one we got right. People asked about it constantly, but we held off. We pointed people at workarounds and said “we’ll see.” We knew doing it properly was a lot of work and easy to get wrong, so we waited. We finally built it last year in a maintainable way, but only because we understood the problem space so well in the first place.

The point

The first version of a question is rarely the real one. Ask why before you answer. Sometimes the answer teaches the user how the tool is meant to be used. Sometimes it tells you the product is hiding the right path. Sometimes it tells you there’s nothing worth building yet. And sometimes it sets you up to build the next big thing once, not twice.

The technique is simple, but easy to skip because the pull to be responsive is constant, and every quick answer feels productive. Take the small step back anyway. Both sides almost always walk away with more than they came in with.

It’s always a pleasant surprise when that does happen! ↩︎
Computing metrics from traces works fine for local development and most lab tests. It breaks down when you need fine-grained measurement with low noise, or when you’re collecting in the field on real users’ devices. ↩︎

Eight years of wanting, three months of building with AI

Sun, 05 Apr 2026 13:00:00 +0100

For eight years, I’ve wanted a high-quality set of devtools for working with SQLite. Given how important SQLite is to the industry¹, I’ve long been puzzled that no one has invested in building a really good developer experience for it².

A couple of weeks ago, after ~250 hours of effort over three months³ on evenings, weekends, and vacation days, I finally released syntaqlite (GitHub), fulfilling this long-held wish. And I believe the main reason this happened was because of AI coding agents⁴.

Of course, there’s no shortage of posts claiming that AI one-shot their project or pushing back and declaring that AI is all slop. I’m going to take a very different approach and, instead, systematically break down my experience building syntaqlite with AI, both where it helped and where it was detrimental.

I’ll do this while contextualizing the project and my background so you can independently assess how generalizable this experience was. And whenever I make a claim, I’ll try to back it up with evidence from my project journal, coding transcripts, or commit history⁵.

Why I wanted it

In my work on Perfetto, I maintain a SQLite-based language for querying performance traces called PerfettoSQL. It’s basically the same as SQLite but with a few extensions to make the trace querying experience better. There are ~100K lines of PerfettoSQL internally in Google and it’s used by a wide range of teams.

Having a language which gets traction means your users also start expecting things like formatters, linters, and editor extensions. I’d hoped that we could adapt some SQLite tools from open source but the more I looked into it, the more disappointed I was. What I found either wasn’t reliable enough, fast enough⁶, or flexible enough to adapt to PerfettoSQL. There was clearly an opportunity to build something from scratch, but it was never the “most important thing we could work on”. We’ve been reluctantly making do with the tools out there but always wishing for better.

On the other hand, there was the option to do something in my spare time. I had built lots of open source projects in my teens⁷ but this had faded away during university when I felt that I just didn’t have the motivation anymore. Being a maintainer is much more than just “throwing the code out there” and seeing what happens. It’s triaging bugs, investigating crashes, writing documentation, building a community, and, most importantly, having a direction for the project.

But the itch of open source (specifically freedom to work on what I wanted while helping others) had never gone away. The SQLite devtools project was eternally in my mind as “something I’d like to work on”. But there was another reason why I kept putting it off: it sits at the intersection of being both hard and tedious.

What makes it hard and tedious

If I was going to invest my personal time working on this project, I didn’t want to build something that only helped Perfetto: I wanted to make it work for any SQLite user out there⁸. And this means parsing SQL exactly like SQLite.

The heart of any language-oriented devtool is the parser. This is responsible for turning the source code into a “parse tree” which acts as the central data structure anything else is built on top of. If your parser isn’t accurate, then your formatters and linters will inevitably inherit those inaccuracies; many of the tools I found suffered from having parsers which approximated the SQLite language rather than representing it precisely.

Unfortunately, unlike many other languages, SQLite has no formal specification describing how it should be parsed. It doesn’t expose a stable API for its parser either. In fact, quite uniquely, in its implementation it doesn’t even build a parse tree at all⁹! The only reasonable approach left in my opinion is to carefully extract the relevant parts of SQLite’s source code and adapt it to build the parser I wanted¹⁰.

This means getting into the weeds of SQLite source code, a fiendishly difficult codebase to understand. The whole project is written in C in an incredibly dense style; I’ve spent days just understanding the virtual table API¹¹ and implementation. Trying to grasp the full parser stack was daunting.

There’s also the fact that there are >400 rules in SQLite which capture the full surface area of its language. I’d have to specify in each of these “grammar rules” how that part of the syntax maps to the matching node in the parse tree. It’s extremely repetitive work; each rule is similar to all the ones around it but also, by definition, different.

And it’s not just the rules but also coming up with and writing tests to make sure it’s correct, debugging if something is wrong, triaging and fixing the inevitable bugs people filed when I got something wrong…

For years, this was where the idea died. Too hard for a side project¹², too tedious to sustain motivation, too risky to invest months into something that might not work.

How it happened

I’ve been using coding agents since early 2025 (Aider, Roo Code, then Claude Code since July) and they’d definitely been useful but never something I felt I could trust a serious project to. But towards the end of 2025, the models seemed to make a significant step forward in quality¹³. At the same time, I kept hitting problems in Perfetto which would have been trivially solved by having a reliable parser. Each workaround left the same thought in the back of my mind: maybe it’s finally time to build it for real.

I got some space to think and reflect over Christmas and decided to really stress test the most maximalist version of AI: could I vibe-code the whole thing using just Claude Code on the Max plan (£200/month)?

Through most of January, I iterated, acting as semi-technical manager and delegating almost all the design and all the implementation to Claude. Functionally, I ended up in a reasonable place: a parser in C extracted from SQLite sources using a bunch of Python scripts, a formatter built on top, support for both the SQLite language and the PerfettoSQL extensions, all exposed in a web playground.

But when I reviewed the codebase in detail in late January, the downside was obvious: the codebase was complete spaghetti¹⁴. I didn’t understand large parts of the Python source extraction pipeline, functions were scattered in random files without a clear shape, and a few files had grown to several thousand lines. It was extremely fragile; it solved the immediate problem but it was never going to cope with my larger vision, never mind integrating it into the Perfetto tools. The saving grace was that it had proved the approach was viable and generated more than 500 tests, many of which I felt I could reuse.

I decided to throw away everything and start from scratch while also switching most of the codebase to Rust¹⁵. I could see that C was going to make it difficult to build the higher level components like the validator and the language server implementation. And as a bonus, it would also let me use the same language for both the extraction and runtime instead of splitting it across C and Python.

More importantly, I completely changed my role in the project. I took ownership of all decisions¹⁶ and used it more as “autocomplete on steroids” inside a much tighter process: opinionated design upfront, reviewing every change thoroughly, fixing problems eagerly as I spotted them, and investing in scaffolding (like linting, validation, and non-trivial testing¹⁷) to check AI output automatically.

The core features came together through February and the final stretch (upstream test validation, editor extensions, packaging, docs) led to a 0.1 launch in mid-March.

But in my opinion, this timeline is the least interesting part of this story. What I really want to talk about is what wouldn’t have happened without AI and also the toll it took on me as I used it.

AI is why this project exists, and why it’s as complete as it is

Overcoming inertia

I’ve written in the past about how one of my biggest weaknesses as a software engineer is my tendency to procrastinate when facing a big new project. Though I didn’t realize it at the time, it could not have applied more perfectly to building syntaqlite.

AI basically let me put aside all my doubts on technical calls, my uncertainty of building the right thing and my reluctance to get started by giving me very concrete problems to work on. Instead of “I need to understand how SQLite’s parsing works”, it was “I need to get AI to suggest an approach for me so I can tear it up and build something better"¹⁸. I work so much better with concrete prototypes to play with and code to look at than endlessly thinking about designs in my head, and AI lets me get to that point at a pace I could not have dreamed about before. Once I took the first step, every step after that was so much easier.

Faster at churning code

AI turned out to be better than me at the act of writing code itself, assuming that code is obvious. If I can break a problem down to “write a function with this behaviour and parameters” or “write a class matching this interface,” AI will build it faster than I would and, crucially, in a style that might well be more intuitive to a future reader. It documents things I’d skip, lays out code consistently with the rest of the project, and sticks to what you might call the “standard dialect” of whatever language you’re working in¹⁹.

That standardness is a double-edged sword. For the vast majority of code in any project, standard is exactly what you want: predictable, readable, unsurprising. But every project has pieces that are its edge, the parts where the value comes from doing something non-obvious. For syntaqlite, that was the extraction pipeline and the parser architecture. AI’s instinct to normalize was actively harmful there, and those were the parts I had to design in depth and often resorted to just writing myself.

But here’s the flip side: the same speed that makes AI great at obvious code also makes it great at refactoring. If you’re using AI to generate code at industrial scale, you have to refactor constantly and continuously²⁰. If you don’t, things immediately get out of hand. This was the central lesson of the vibe-coding month: I didn’t refactor enough, the codebase became something I couldn’t reason about, and I had to throw it all away. In the rewrite, refactoring became the core of my workflow. After every large batch of generated code, I’d step back and ask “is this ugly?” Sometimes AI could clean it up. Other times there was a large-scale abstraction that AI couldn’t see but I could; I’d give it the direction and let it execute²¹. If you have taste, the cost of a wrong approach drops dramatically because you can restructure quickly²².

Teaching assistant

Of all the ways I used AI, research had by far the highest ratio of value delivered to time spent.

I’ve worked with interpreters and parsers before but I had never heard of Wadler-Lindig pretty printing²³. When I needed to build the formatter, AI gave me a concrete and actionable lesson from a point of view I could understand and pointed me to the papers to learn more. I could have found this myself eventually, but AI compressed what might have been a day or two of reading into a focused conversation where I could ask “but why does this work?” until I actually got it.

This extended to entire domains I’d never worked in. I have deep C++ and Android performance expertise but had barely touched Rust tooling or editor extension APIs. With AI, it wasn’t a problem: the fundamentals are the same, the terminology is similar, and AI bridges the gap²⁴. The VS Code extension would have taken me a day or two of learning the API before I could even start. With AI, I had a working extension within an hour.

It was also invaluable for reacquainting myself with parts of the project I hadn’t looked at for a few days²⁵. I could control how deep to go: “tell me about this component” for a surface-level refresher, “give me a detailed linear walkthrough” for a deeper dive, “audit unsafe usages in this repo” to go hunting for problems. When you’re context switching a lot, you lose context fast. AI let me reacquire it on demand.

More than I’d have built alone

Beyond making the project exist at all, AI is also the reason it shipped as complete as it did. Every open source project has a long tail of features that are important but not critical: the things you know theoretically how to do but keep deprioritizing because the core work is more pressing. For syntaqlite, that list was long: editor extensions, Python bindings, a WASM playground, a docs site, packaging for multiple ecosystems²⁶. AI made these cheap enough that skipping them felt like the wrong trade-off.

It also freed up mental energy for UX²⁷. Instead of spending all my time on implementation, I could think about what a user’s first experience should feel like: what error messages would actually help them fix their SQL, how the formatter output should look by default, whether the CLI flags were intuitive. These are the things that separate a tool people try once from one they keep using, and AI gave me the headroom to care about them. Without AI, I would have built something much smaller, probably no editor extensions or docs site. AI didn’t just make the same project faster. It changed what the project was.

Where AI had its costs

The addiction

There’s an uncomfortable parallel between using AI coding tools and playing slot machines²⁸. You send a prompt, wait, and either get something great or something useless. I found myself up late at night wanting to do “just one more prompt,” constantly trying AI just to see what would happen even when I knew it probably wouldn’t work. The sunk cost fallacy kicked in too: I’d keep at it even in tasks it was clearly ill-suited for, telling myself “maybe if I phrase it differently this time.”

The tiredness feedback loop made it worse²⁹. When I had energy, I could write precise, well-scoped prompts and be genuinely productive. But when I was tired, my prompts became vague, the output got worse, and I’d try again, getting more tired in the process. In these cases, AI was probably slower than just implementing something myself, but it was too hard to break out of the loop³⁰.

Losing touch

Several times during the project, I lost my mental model of the codebase³¹. Not the overall architecture or how things fitted together. But the day-to-day details of what lived where, which functions called which, the small decisions that accumulate into a working system. When that happened, surprising issues would appear and I’d find myself at a total loss to understand what was going wrong. I hated that feeling.

The deeper problem was that losing touch created a communication breakdown³². When you don’t have the mental thread of what’s going on, it becomes impossible to communicate meaningfully with the agent. Every exchange gets longer and more verbose. Instead of “change FooClass to do X,” you end up saying “change the thing which does Bar to do X”. Then the agent has to figure out what Bar is, how that maps to FooClass, and sometimes it gets it wrong³³. It’s exactly the same complaint engineers have always had about managers who don’t understand the code asking for fanciful or impossible things. Except now you’ve become that manager.

The fix was deliberate: I made it a habit to read through the code immediately after it was implemented and actively engage to see “how would I have done this differently?”.

Of course, in some sense all of the above is also true of code I wrote a few months ago (hence the sentiment that AI code is legacy code), but AI makes the drift happen faster because you’re not building the same muscle memory that comes from originally typing it out.

The slow corrosion

There were some other problems I only discovered incrementally over the three months.

I found that AI made me procrastinate on key design decisions³⁴. Because refactoring was cheap, I could always say “I’ll deal with this later.” And because AI could refactor at the same industrial scale it generated code, the cost of deferring felt low. But it wasn’t: deferring decisions corroded my ability to think clearly because the codebase stayed confusing in the meantime. The vibe-coding month was the most extreme version of this. Yes, I understood the problem, but if I had been more disciplined about making hard design calls earlier, I could have converged on the right architecture much faster.

Tests created a similar false comfort³⁵. Having 500+ tests felt reassuring, and AI made it easy to generate more. But neither humans nor AI are creative enough to foresee every edge case you’ll hit in the future; there are several times in the vibe-coding phase where I’d come up with a test case and realise the design of some component was completely wrong and needed to be totally reworked. This was a significant contributor to my lack of trust and the decision to scrap everything and start from scratch.

Basically, I learned that the “normal rules” of software still apply in the AI age: if you don’t have a fundamental foundation (clear architecture, well-defined boundaries) you’ll be left eternally chasing bugs as they appear.

No sense of time

Something I kept coming back to was how little AI understood about the passage of time³⁶. It sees a codebase in a certain state but doesn’t feel time the way humans do. I can tell you what it feels like to use an API, how it evolved over months or years, why certain decisions were made and later reversed.

The natural problem from this lack of understanding is that you either make the same mistakes you made in the past and have to relearn the lessons or you fall into new traps which were successfully avoided the first time, slowing you down in the long run. In my opinion, this is a similar problem to why losing a high-quality senior engineer hurts a team so much: they carry history and context that doesn’t exist anywhere else and act as a guide for others around them.

In theory, you can try to preserve this context by keeping specs and docs up to date. But there’s a reason we didn’t do this before AI: capturing implicit design decisions exhaustively is incredibly expensive and time-consuming to write down. AI can help draft these docs, but because there’s no way to automatically verify that it accurately captured what matters, a human still has to manually audit the result. And that’s still time-consuming.

There’s also the context pollution problem. You never know when a design note about API A will echo in API B. Consistency is a huge part of what makes codebases work, and for that you don’t just need context about what you’re working on right now but also about other things which were designed in a similar way. Deciding what’s relevant requires exactly the kind of judgement that institutional knowledge provides in the first place.

Relativity

Reflecting on the above, the pattern of when AI helped and when it hurt was fairly consistent.

When I was working on something I already understood deeply, AI was excellent. I could review its output instantly, catch mistakes before they landed and move at a pace I’d never have managed alone. The parser rule generation is the clearest example³⁷: I knew exactly what each rule should produce, so I could review AI’s output within a minute or two and iterate fast.

When I was working on something I could describe but didn’t yet know, AI was good but required more care. Learning Wadler-Lindig for the formatter was like this: I could articulate what I wanted, evaluate whether the output was heading in the right direction, and learn from what AI explained. But I had to stay engaged and couldn’t just accept what it gave me.

When I was working on something where I didn’t even know what I wanted, AI was somewhere between unhelpful and harmful. The architecture of the project was the clearest case: I spent weeks in the early days following AI down dead ends, exploring designs that felt productive in the moment but collapsed under scrutiny. In hindsight, I have to wonder if it would have been faster just thinking it through without AI in the loop at all.

But expertise alone isn’t enough. Even when I understood a problem deeply, AI still struggled if the task had no objectively checkable answer³⁸. Implementation has a right answer, at least at a local level: the code compiles, the tests pass, the output matches what you asked for. Design doesn’t. We’re still arguing about OOP decades after it first took off.

Concretely, I found that designing the public API of syntaqlite was where this hit home the hardest. I spent several days in early March doing nothing but API refactoring, manually fixing things any experienced engineer would have instinctively avoided but AI made a total mess of. There’s no test or objective metric for “is this API pleasant to use” and “will this API help users solve the problems they have” and that’s exactly why the coding agents did so badly at it.

This takes me back to the days I was obsessed with physics and, specifically, relativity. The laws of physics look simple and Newtonian in any small local area, but zoom out and spacetime curves in ways you can’t predict from the local picture alone. Code is the same: at the level of a function or a class, there’s usually a clear right answer, and AI is excellent there. But architecture is what happens when all those local pieces interact, and you can’t get good global behaviour by stitching together locally correct components.

Knowing where you are on these axes at any given moment is, I think, the core skill of working with AI effectively.

Wrap-up

Eight years is a long time to carry a project in your head. Seeing these SQLite tools actually exist and function after only three months of work is a massive win, and I’m fully aware they wouldn’t be here without AI.

But the process wasn’t the clean, linear success story people usually post. I lost an entire month to vibe-coding. I fell into the trap of managing a codebase I didn’t actually understand, and I paid for that with a total rewrite.

The takeaway for me is simple: AI is an incredible force multiplier for implementation, but it’s a dangerous substitute for design. It’s brilliant at giving you the right answer to a specific technical question, but it has no sense of history, taste, or how a human will actually feel using your API. If you rely on it for the “soul” of your software, you’ll just end up hitting a wall faster than you ever have before.

What I’d like to see more of from others is exactly what I’ve tried to do here: honest, detailed accounts of building real software with these tools; not weekend toys or one-off scripts but the kind of software that has to survive contact with users, bug reports, and your own changing mind.

syntaqlite: high-fidelity devtools that SQLite deserves

Tue, 17 Mar 2026 05:35:45 +0000

Most SQL tools treat SQLite as a “flavor” of a generic SQL parser. They approximate the language, which means they break on SQLite-exclusive features like virtual tables, miss syntax like UPSERT, and ignore the 22 compile-time flags that change the syntax SQLite accepts.

So I built syntaqlite: an open-source parser, formatter, validator, and LSP built directly on SQLite’s own Lemon-generated grammar. It sees SQL exactly how SQLite sees it, no matter which version of SQLite you’re using or which feature flags you compiled with.

It ships as a CLI, VS Code extension, Claude Code LSP plugin, and C/Rust libraries.

There’s also a web playground which you can try now: paste any SQLite SQL and see parsing, formatting, and validation live in the browser, no install needed. Full documentation is available here.

Here’s syntaqlite in action:

Formatting with the CLI

> syntaqlite fmt -e "select u.name,u.email,count(e.id) as events from users u join events e on e.user_id=u.id where u.signed_up_at>=date('now','-30 days') group by u.name,u.email having count(e.id)>10 order by events desc"

SELECT u.name, u.email, count(e.id) AS events
FROM users AS u
JOIN events AS e ON e.user_id = u.id
WHERE
 u.signed_up_at >= date('now', '-30 days')
GROUP BY
 u.name,
 u.email
HAVING
 count(e.id) > 10
ORDER BY
 events DESC;

Validation with the CLI

> syntaqlite --sqlite-version 3.37.0 validate \
 -e "SELECT json_extract(data, '$.name') FROM events;"

error: unknown function 'json_extract'
 --> <expression>:1:8
 |
1 | SELECT json_extract(data, '$.name') FROM events;
 | ^~~~~~~~~~~~

LSP support in VSCode

LSP in Claude Code

Why it exists

I maintain PerfettoSQL, a dialect of SQLite SQL used for trace analysis in Perfetto. Across Google there are 200K+ lines of the language. At that scale, you hit every corner of it. I tried every SQLite tool I could find and time after time I was disappointed: false positives on valid syntax, misparsed statements, formatters that silently corrupted the SQL by changing semantics. It always came down to a hand-written or generic grammar approximating SQLite rather than matching it to the letter.

Moreover, SQLite’s SQL is not one fixed language. Apart from flags that change the syntax, there are another 12 flags that gate built-in functions, as well as APIs for programs to register custom functions and virtual tables. In Perfetto, we make extensive use of both virtual tables and functions and most tools just don’t give us the flexibility to treat these as first-class extensions of the language.

The SQLite language also constantly evolves: in the last several years, 3.25.0 added window functions, 3.35.0 added RETURNING, 3.38.0 added built-in JSON functions to the amalgamation. And because SQLite is embedded, you can’t assume everyone is on the latest version. This is certainly the case for Perfetto: people link against different versions of SQLite and we need to make sure our standard library is resilient to that.

Now with all of this said, I could have built all these tools just for PerfettoSQL and called it a day. But the more I thought about it, the more I felt that the difference between “good tooling for PerfettoSQL” and “good tooling for SQLite” is so small that it’s worth solving for the wider use case. SQLite is ubiquitous, yet it still lacks the quality tooling other languages have had for over a decade.

Android is one case I think could especially benefit. SQLite is integral to almost every Android app not to mention the platform itself. But the ecosystem is also famously fragmented: if you’re an Android app developer, you need to reason about older platform versions, which in turn means older versions of SQLite.

As an example, Android 13 ships with SQLite 3.32 (link). If you use the RETURNING keyword and your users are on Android 13, it will fail at runtime even though it passes fine on your test device. Wouldn’t it be nice to statically verify this on your host instead of discovering it on a real device?

No tool I found handled all of this: the precise SQLite grammar, version matching, compile flags, dialect extensions. And that’s exactly why syntaqlite exists.

Highlights

Validate semantics with great error messages

Parsing alone is table stakes. But I also wanted to catch common mistakes like reaching for user_name when the column is actually username. For this reason, syntaqlite also resolves table, column, and function references, catching errors like misspelled column names or references to tables that don’t exist.

I’ve also tried to make the error messages really good: at the quality level you would expect to see from modern tools in other languages like rustc or tsc. This means both accurate, line/column level indication with snippets of the error but also “did you mean” hints, pointing you in the right direction. The error messages should be significantly better than SQLite’s and it’s something I plan on making even better with time.

As for correctness, I’ve tested syntaqlite against ~396K statements from SQLite’s upstream test suite with ~99.7% agreement on parse acceptance ¹. Before we get to 1.0, I fully expect this to reach 100%.

There’s also a detailed comparison against other tools covering parser accuracy, formatter correctness, validator quality, and performance. Of course the benchmarks and stress tests were written by me, so take it with a grain of salt. But hopefully the results speak for themselves.

LSP for editors and coding agents

syntaqlite ships a full language server: completions, hover, go-to-definition, find references, rename, diagnostics, and formatting. Unlike most languages, SQL queries depend on an external schema that isn’t in the source file. The LSP needs to know about that schema to give you useful diagnostics.

The most common way to provide this is with a syntaqlite.toml at the root of your project ². Point it at your CREATE TABLE definitions (which you can dump using .schema on your SQLite database; see the project setup guide for details) and syntaqlite uses those for autocompletion, diagnostics, and find references.

[schemas]
"**/*.sql" = ["db/schema.sql"]

This tells syntaqlite: for every .sql file in the project, validate against the definitions in db/schema.sql. You can point at multiple schema files, and scope different schemas to different directories.

syntaqlite.toml can be used to configure other preferences like indentation for formatting and the SQLite version as well. The idea is that you check this into your repo and the whole team gets the same experience.

It validates SQL inside other languages (experimental)

SQL is unusual in that it often lives inside other languages: string literals in Python, template literals in TypeScript, raw strings in C++, and so on.

As an experimental feature (and with caveats!), syntaqlite can extract and validate SQL inside other languages, even if you have string interpolation. For example, in a Python f-string:

# app.py
def get_user_stats(user_id: int):
 return conn.execute(
 f"SELECT name, ROUDN(score, 2) FROM users WHERE id = {user_id}"
 )

syntaqlite validate --experimental-lang python app.py

warning: unknown function 'ROUDN'
 --> app.py:4:23
 |
4 | f"SELECT name, ROUDN(score, 2) FROM users WHERE id = {user_id}"
 | ^~~~~
 = help: did you mean 'round'?

It handles the {user_id} interpolation hole and still catches the function typo with a did-you-mean suggestion.

And more

SQLite version- and flag-aware. Tell syntaqlite which SQLite version and compile-time flags your target has; the formatter, validator, and LSP all respect them.
Custom dialect support. Define grammar, formatting, and validation rules, compile to a shared library, load at runtime. I built this for PerfettoSQL, but it works for any engine that extends SQLite’s syntax. For example, a mobile analytics engine with custom aggregation functions or a local-first app framework with its own replication primitives.
Fast. The formatter processes 3,500 lines of SQL in ~5ms. Full benchmarks on the comparison page.

How it works

I plan on writing a detailed followup on the technical side, but I did want to give a short summary here.

Fundamentally syntaqlite is a C/Rust/C sandwich:

+-------------+---------------+--------------+-------+--------------------+
| VS Code | Claude Code | Playground | CLI | C bindings |
| Extension | / Agents | (WASM) | | (Go, Kotlin...) |
+-------------+---------------+--------------+-------+--------------------+
| Language Server (LSP) |
+-------------------------------------------------------------------------+
| Formatter · Validator · Analyzer (Rust) |
+-------------------------------------------------------------------------+
| Parser · Tokenizer · AST Arena (C, from SQLite) |
+-------------------------------------------------------------------------+

The bottom layer is C: SQLite’s Lemon-generated grammar and tokenizer, plus an arena storing a super flat AST. This layer has no Rust dependency and can be used entirely on its own, which matters if you’re building a database engine or other tool in C/C++ that just needs to parse SQLite SQL ³.

The middle layer is Rust. It reads the C arena directly with no copying or deserialization. The formatter generates a Wadler-Lindig document using an AST-walking interpreter; the semantic analyzer walks the same AST, resolving references and producing diagnostics with byte-accurate source locations.

The top layer is C again, completing the sandwich: the Rust tools are exported back through FFI so C/C++ projects can link syntaqlite as a plain C library. This also makes bindings for other languages possible: Go, Kotlin/Java, and Python are planned before 1.0. Of course, if you’re using Rust directly you can just use the Rust library, skipping the top C layer.

The obvious question I’m expecting is “why not just write everything in Rust”? I wanted the parser to work exactly as SQLite does, which meant literally extracting SQLite’s own code and adapting it to my needs. And writing it in C with the same constraints as SQLite itself (no dependencies beyond the C standard library) means it can be linked into practically anything: database engines, C++ projects like Perfetto’s trace processor, or any environment where SQLite runs.

On the other hand, I really did not want to write a formatter, semantic analyzer, and LSP in C. While I do like the “simplicity” of C, writing a full JSON-RPC server and protocol sounds like a nightmare.

Where this is going

In my head, there’s a clear roadmap for the project side:

100% upstream parity. The ~0.3% gap against SQLite’s test suite is mostly from not fully tracking built-in tables and virtual tables. It just needs another round of “source code introspection” across all the SQLite versions and integrating those tables into syntaqlite’s builtin catalog.
LSP enhancements. Code actions, document symbols, inlay hints for inferred column types. All of these basically making the editor experience even better.
Robust embedded SQL. The Python/TypeScript support is really just hacked together with regexes as a proof of concept. My goal is to extend this to use proper parser libraries and add support for more languages.
Polish and performance. The focus for 0.1 was getting the architecture right. There’s room to improve performance, API surface, and documentation across the whole stack.

As for the blog, this project was the reason I’ve been completely silent over the last month, but now I’m excited to start writing a lot more again. For one, there’s the technical followup for the decisions I took and what I learned along the way.

For another, I used coding agents (mainly Claude Code) extensively while building syntaqlite. There were plenty of moments where following them blindly would have sunk the project, but I can also say with certainty that syntaqlite would not exist without them. I want to dive into that tension: what worked, what didn’t, and the complicated feelings I came away with.

So if you work with SQLite SQL at any scale, I’d appreciate you giving syntaqlite a try and telling me what breaks! SQLite is a project which famously plans to be supported until 2050. My hope is that syntaqlite will be there as a companion every step of the way, making the experience of writing SQLite SQL that much nicer.

syntaqlite on GitHub | Docs | Playground

Per-statement results are in tests/upstream_baselines/. The remaining 0.3% is largely down to not perfectly matching the builtin tables and then edge cases around where and when column aliases come into scope: it’s actually really complicated! ↩︎
You can also pass schema files directly via --schema on the command line without needing a config file. ↩︎
The plan is to replace the PerfettoSQL parser in Perfetto’s trace processor with this C layer. ↩︎

The surprising attention on sprites, exe.dev, and shellbox

Thu, 29 Jan 2026 00:34:22 +0000

Over the last few weeks, three new products have announced themselves on Hacker News to great success, each making the frontpage:

Sprites (fly.io) with 508 votes and hit #7
exe.dev with 457 votes and hit #2
shellbox.dev with 316 votes and hit #4

All three have a very simple pitch: they will give you full access to Linux virtual machines to act as a sandboxed developer environment in the cloud.

At first glance, the attention these have gotten is very head-scratching. The idea of a Linux VPS has been around for more than 20 years at this point and VPS providers like DigitalOcean and Hetzner are widely known and used in the industry. From a technological standpoint, there’s very little revolutionary here.

Is it price then? Well no: the hardware specs are pretty awful for what you pay. For example, exe.dev gives you 2 CPUs and 8GB RAM shared across your whole account for $20/month. For comparison, at Hetzner for roughly that price, you can get a single VPS with 16 CPUs and 32GB RAM…

And moving a local dev environment to the cloud has real downsides: added latency, needing a network connection, making graphical access painful. If all you wanted was the ability to partition hardware resources, we’ve had VMs and containers for more than a decade. You can use Docker on any decent Linux machine and spin up containers to your heart’s content. And with apple/container, it’s very easy to launch a semi-sandboxed Linux environment on macOS too.

So why is there so much excitement around fancy VPSes?

Why the criticisms miss the point

The real reasons are two-fold: we’re in the age of AI and the user experience of these products.

First, sandboxing. If you’re using LLM coding agents and you’re not looking to YOLO things, sandboxing matters. Most prominently, Simon Willison has been beating the drum on the security risks of agents, with sandboxing being a good way to cut off the “Access to Private Data” part of the Lethal Trifecta. Segregating to a fresh VM, while by no means perfect, significantly limits the blast radius compared to using your personal dev machine if something goes wrong.

Second, there’s been an explosion in prototyping projects due to tools like Claude Code and Codex. They make it so easy to spin things up that developers want more “fresh starts” instead of piling features onto existing projects. And if an experiment doesn’t work out, you just delete the machine and start over without figuring out how to clean up.

But why these new services over a traditional VPS? Speed. All three are optimized to get you from zero to “machine running Claude Code” in 1-2 minutes, versus 20-30 with traditional providers. They also handle the annoying bits of sharing web services: reverse proxy, TLS termination, DNS are all available instantaneously and with zero fuss; it’s night and day vs trying to roll your own.

In some sense, this is developer environments catching up to what has happened to production systems over the last decade. We’ve had infrastructure-as-code and “cattle not pets” thinking in deployment since containers and Kubernetes took the world by storm. GitHub Codespaces was a step in this direction for dev environments. Put another way, this new crop of services are an evolution: they let you go beyond just the “development” part and to the full end-to-end stack.

Speedrunning the services

But enough about the abstract philosophy about why these services are having a moment: I decided to try out all of these services to see for myself whether they were actually useful and which one I liked the most.

My motivating use case for this: I was looking to understand how COOP and COEP work in all modern browsers for building a new feature in Perfetto ¹. This was a perfect vibe-coding task: it was fully throw-away code for testing something that would never directly end up in any prod project. And it was a web based thing where I needed https and certs etc. Literally could not design a better use case for this.

Here’s a quick rundown of the services:

Service	Pricing	Specs	Interface
exe.dev	$20/mo for up-to 25 VMs	2 CPU, 8GB RAM, 25GB disk shared across all 25 VMs	SSH + web shell + built-in AI agent
sprites	Usage-based ($0.07/CPU-hr, need to pay for RAM + storage also)	Bursts to 8 CPU, 16GB RAM	CLI tool for setup + SSH afterwards
shellbox	Usage-based ($0.05/hr running, $0.005/hr stopped)	2 CPU, 4GB RAM, 50GB SSD	Pure SSH

Immediately, it’s clear that there are three different “dimensions” these services are competing on:

Pricing model: flat fee vs usage based
Maximum possible specs: laughably poor vs somewhat reasonable
Interface: ssh vs custom CLI

It’s amazing how three tools in such a short space ended up in such different places! But which one actually worked best for me?

My experiences

shellbox

The UX to interface with shellbox is very cool: everything (billing, creating VMs, deleting VMs) is all done via ssh! You ssh into a server and they use your public key to figure out your account. You then add “credits” to your account that you can then use. Having not seen any other service like this before, I was already very impressed by the “cool factor”.

Unfortunately, that quickly faded as I found I couldn’t even try it without paying first some amount of money. And the payment UX was rough: no indication of how long I’d have to wait, and for a while their SSH status command kept telling me I had no credits; they finally showed up 15 minutes later.

Once in, I found Ubuntu 18.04: not exactly a “modern” environment (it’s actually been EOL since 2023…). And nothing was pre-installed; I even had to install Python myself! Certainly not ideal if the goal is “get to a usable dev environment fast”.

But the real dealbreaker for me is that, in their world, they consider a VM “stopped” the moment you disconnect SSH. To my understanding, this means you can’t run a webserver in the background even for a day or two to test something. Instead you need to stay connected constantly, which for the sort of thing I was trying to build, was very non-ideal.

exe.dev

Firstly, I want to make clear that the homepage for exe.dev sucks, especially on mobile: it doesn’t tell you anything about what you’re getting and just tells you “ssh exe.dev”; I wasn’t about to SSH into a random server without understanding why. You only understand what you’re getting by clicking “About” at the bottom which takes you to their docs (which are actually decent!).

Once you SSH in, though? The UX is fantastic. It recognized my public key, looked it up against my GitHub account automatically ² and pulled my email and associated it to my account. On the VM, there was Ubuntu 24.04, Claude pre-installed, recent Python. They clearly understand their user base and what they’re trying to accomplish.

And beyond the command line, there’s a web shell for when SSH isn’t convenient. And they have a custom, web-based coding agent called Shelley which works surprisingly well (honestly, in many ways, nicer than Claude Code on web). I especially like their ability to look at the web service running and take screenshots to understand the state of the page; very cool!

It’s night and day between the homepage and the actual product. The only annoyance I ran into is that you can only expose a single public port, which I needed in my particular case (but I can understand this is a somewhat niche ask). Beyond that, I had a very nice experience.

sprites

Having tried shellbox and exe.dev, the need to generate a token and install their CLI for sprites made me feel like I’d left my magic SSH kingdom and was back in the real world.

Once installed, the process to create a VM and ssh in was very smooth. Claude comes pre-installed, and interestingly, they configure it with --dangerously-skip-permissions by default! You’re given Ubuntu 25.04, so relatively bleeding edge. Beyond that, it all felt very standard, you’re just getting a VPS fast and fuss-free but with no other “magic” like exe.dev had.

In general though, the vibe I got throughout my experience of interacting with the tools and their web dashboard was that it felt very corporate" and designed for companies with teams of developers.

I think this feeling is best summed up by how their home page looks when logged in:

Not exactly flattering to see myself labelled as an organization!

Overall assessment

exe.dev is my pick for solo developers. The flat $20/month cap feels psychologically nice—you know your max spend. The 2 vCPUs might feel limiting, but for Claude Code sessions where you’re not doing anything CPU-intensive, it’s probably fine. And frankly, the UX is so good it makes up for a lot.

shellbox has the simplest pricing model, but the user experience left me underwhelmed. The environment is ancient, the payment flow was confusing, and the SSH-disconnect behavior kills too many use cases. It feels a bit hacked together.

sprites is the most interesting technically. I love their design blog post: the ability to burst to 8 CPUs and 16GB RAM while only paying for what you use is very clever and I can imagine it would be great if you were only planning light usage. But the UX feels corporate, not magical. If I were running a company, I’m reasonably certain sprites would be the way to go. But for solo experimentation, exe.dev wins.

Conclusion

You might find these services incredible or baffling depending on your stance on AI and how much you’re willing to pay for convenience.

These services aren’t targeting customers who care about “value for money” or raw specs. They’re targeting developers who want frictionless access to machines, without setting up VMs, wrestling with containers, or configuring certificates to show off prototypes.

I’ve been won over by the neat UX and “joyous” feeling of exe.dev. I’ll probably pay for it in months where I expect to do a lot of prototyping. Having tried the alternatives (Docker, local VMs, traditional VPSes) it’s nice to have something that just works.

https://github.com/google/perfetto/discussions/3227 if you’re interested! ↩︎
How, you might ask? Well if you didn’t know GitHub by default makes the all SSH public keys publicly accessible both via their API but also just by going to github.com/username.keys. For example, for my account: https://github.com/lalitmaganti.keys. ↩︎

Why Senior Engineers Let Bad Projects Fail

Tue, 13 Jan 2026 10:48:00 +0000

When I was a junior engineer, my manager would occasionally confide his frustrations to me in our weekly 1:1s. He would point out a project another team was working on and say, “I don’t believe that project will go anywhere, they’re solving the wrong problem.” I used to wonder, “But you are very senior, why don’t you just go and speak to them about your concerns?” It felt like a waste of his influence to not say anything.

So it’s quite ironic that I found myself last week explaining to a mentee why I thought a sister team’s project would have to pivot because they’d made a poor early design choice. And he rightfully asked me the same question I had years ago: “why don’t you just tell them your opinion?” It’s been on my mind ever since because I realized I’d changed my stance on it a lot over the years.

The answer is that being right and being effective are different.

In large companies, speaking up about what you see as a “bad project” is a good thing. But only in moderation. Sometimes the mark of seniority is realizing that arguing with people who won’t listen isn’t worth it; it’s better to save your counsel.

Bad projects

What I mean by a “bad project” is many things:

UX: making product complicated, solving a problem which doesn’t exist, breaking existing workflows
Technical: overcomplicated design, wrong library, poor performing architecture
Political: chasing hype cycles, exists primarily to justify a promotion

It’s important to point out that for much of the lifecycle of a project, whether it’s “bad” is highly subjective. Software engineering is largely a game of tradeoffs and making decisions which are not perfect but the best possible with the information available. There often can be disagreements on whether correct choices are made and it only becomes obvious much later on, potentially years after a project has shipped.

But as you become more senior, you’ll start to have “taste” when it comes to software projects and that will cause you to look at some fraction of the software projects and feel “this doesn’t make sense”. And this gut feeling is the sign to me of a “bad project”, one which you can see in advance of when it’s obvious to everyone.

Drawing on my personal experience, the most memorable example was a few years ago at Google . There was a high-profile announcement internally of a “game changer” project that sat right at the intersection of two extremely large organizations. It was technically amazing and elegant, and full of clever ideas for really hard problems.

But I distinctly remember sitting in the room for the announcement, turning to my lead and whispering, “This project has no chance of succeeding, right?” He turned to me and just said, “Yup.” We both realized the problem immediately. The project was entirely based on a platform team asking a flagship product team to give up control of their core user flow: technically the right move, but no lead or PM would ever cede ownership of something that central to another team. Politically, this project was a total fantasy.

The project kept quietly chugging away in the background for almost two years. Every time it got close to launch, it would get pushed back as “not ready yet.” Over time, we heard less and less about it until, eventually, the inevitable “strategic pivot” email appeared in my inbox. Resources were reallocated and the code was deleted. We were told the company “learned a lot from the effort,” but to me it felt like it was doomed from the beginning. Politics and solving the correct problem matter just as much as technical beauty.

Why you cannot stop them all

When I started noticing “bad projects” and I felt that I had some expertise to share, the temptation for me was to start calling them out. Reach out to the team doing it, tell them “this doesn’t make sense” and explain to them why. Use facts and logic to persuade.

And I did do this. But only for a very short time before I realized that there are a lot of costs to doing this that I just wasn’t thinking about.

Firstly, software companies have an inherent bias for action. They value speed and shipping highly. Concerns, by definition, slow things down and mean people have to look at things which they hadn’t budgeted for. And so unless your concern is big enough to overcome the “push for landing”, there’s little chance for any meaningful change to come from you saying something. In fact, it’s very likely that you’ll be largely ignored.

Related to this, even if the team does take your concern seriously, you have to be careful not to do it too often. Once or twice, you might be seen as someone who is upholding “quality”. But do it too often and you quickly move to being seen as a “negative person”, someone who is constantly a problem maker, not a problem “fixer”. You rarely get credit for the disasters you prevented. Because nothing happened, people forget about it quickly.

There’s also the problem that every time you push back, you are potentially harming someone’s promotion packet or a VP’s “pet project.” You are at risk of burning bridges and creating “enemies”, at least of a sort. Having a few people who disagree in a big company with you is the cost of doing business, but if you have too many, it starts affecting your main work too.

Finally, there is also the psychological impact. There is one of you and hundreds of engineers working in spaces that your expertise might help with. Your attention is finite, but the capacity for a large company to generate bad ideas is infinite. Speaking from experience, getting too involved in stopping these quickly can make you very cynical about the state of the world. And this is really not a good place to be.

Manage influence like a bank account

So if you cannot stop all the bad projects, what do you do? You get strategic. Instead of trying to fix everything, view your influence as a bank account. You have a certain amount of “influence” coming in every month as you do your job, help people, ship successful projects, and generally remain low friction.

Then, when it matters, you should be ready to make “withdrawals.” Every time you block something or raise concerns, no matter how small, you are writing a check against your balance. But not all checks are the same size:

The $5 Check: A nitpick on a code review. Cheap, daily expense.
The $500 Check: Challenging an architectural decision or pushing back on a timeline. Requires some savings.
The $50,000 Check: Trying to kill a VP’s pet project. This is a massive spend. You might only afford this once every few years.

The problem comes if you spend $5 on every minor inefficiency you see. If you are constantly saying “no” to small things, your account will be empty when you need to write the big check to stop a true disaster.

If you “go overdrawn,” you enter political bankruptcy. People stop inviting you to meetings, they stop asking for your opinion, they essentially start working around you. Once you are bankrupt, your influence drops to zero and you not only harm your ability to influence things but also start hurting your own ability to get things done.

When to spend influence

Given that we’ve now accepted that we cannot weigh in on everything, we need to figure out when it does make sense to do so.

The most important thing to do first is to be humble and evaluate whether you actually have the expertise to make a judgment. Seniority often brings opinions, but those are not always informed opinions. For example, while I have some frontend experience, I do not feel qualified to give deep advice on it because my knowledge is “enough to get by” rather than deep expertise that comes from long term ownership. It is easy to lose sight of the fact that high-quality judgments require informed opinions. If you find yourself in this position, see yourself as an opinionated observer and stop there.

You must also internalize the fact that just because you say something does not make it the truth. You are raising awareness of a point of view, not issuing a decree. So if some team doesn’t listen to your concerns and decides to go ahead with what they were doing anyway, then you have to accept that and move on: at the end of the day, you’re an engineer, not a CEO with authority over them!

Given these points, I use three main factors to decide when to speak up:

How close is the project to my team?
If it goes wrong, how much impact will it have on my team?
If it goes wrong, how big will the problem be for the company?

Proximity. If a project is close to you, the “price tag” of saying something is lower. If it is within your own team, the cost is near zero because you have high trust and a quick conversation often solves it. If it is in your broader organization, the price goes up; you have to spend social capital and potentially stake your reputation. If it is outside your org? The cost is often prohibitive. You have zero leverage, different reporting chains, and stopping it would require a massive withdrawal.

Team Impact. Sometimes another org does something that deeply affects your work. For example, because Perfetto (the performance tool I work on) has users throughout Google, sometimes a team will ask us to sign off on a very complex integration. This is a classic risk: if things go right, they get the credit, but if things go wrong, your leadership might expect you to help solve a problem you didn’t create. In these cases, the payoff of speaking up is high because you are protecting your team.

Company Scale. Finally, consider the blast radius. Some projects are self-contained; if they fail, they only take themselves down. Others are so intertwined with core systems that their failure causes widespread damage or creates technical debt that persists for years. These can be deadly to the long-term health of a project.

How to act with bad projects

It’s also not just about when you put your opinions forward but how you do it. There’s a very wide range of actions you can take depending on what you’re facing.

When you intervene

The nuclear option is to directly say “we should not do this” and try to shut the project down. This almost always requires escalation to your leads and the leads of the owning team, requiring great conviction in both the fact that you’re right and that this project will be actively harmful. But on some occasions, this is the right thing to do, especially if the cost of not saying something can be existential to your project or team.

A slightly softer but still quite risky variant of this is, instead of doing a direct escalation, you raise concerns in directly with the team. Usually this is done with a meeting with the team or a strongly worded “concern” or “rebuttal” doc. The goal is to speak in strong enough terms that the team themselves conclude that this the project might not be a good idea.

Then there are the smaller interventions, nudging things in the right direction. These are perfect for when a team is about to do something that makes sense from a high level but they are going about this the wrong way. I see this often with Perfetto: a team sends a design doc proposing a complex use of Perfetto that I know will cause them pain later. I sit down with them, understand their actual problem, and guide them to a better solution. It costs an hour but saves them months. If you do it right, you can even be seen as a helper rather than a hindrance, even if you do slow down the team.

When you don’t

Sometimes you conclude that the ROI just isn’t there to do anything direct: the political momentum is too strong, or the issue is too small to justify spending any influence. At this point, what you do depends on how much your team is involved.

If it overlaps with your team’s work heavily then it might be best to make some subtle contingency plans: reducing your dependency on it or building abstractions to cope if it goes away. There is also a long game trick here. Even a bad project usually has an “essence” of a good idea, a specific problem it was trying to solve or an insight it was based on. If it fits with your job, it’s often a good idea to take that essence and see if you can naturally incorporate a better version of that specific solution into your own project. That way, if the bad project stalls or gets canceled, you can be proactive instead of reactive to the fallout.

Alternatively, if you’re not involved, it’s easy: just stay out of the picture. Vent to friendly colleagues in private, commiserate, but in public, live with the reality.

Managing your team through it

Finally, you must manage your own team through the process. If you can see the flaws in a project, other senior engineers probably see them too. Don’t try to gaslight them or “walk the company line” by pretending a bad project is actually good. It destroys trust.

Instead, be honest about the facts on the ground without going into unnecessary political details. Tell them that you will do the best you can under these constraints.

Conclusion

So what did I tell my mentee? “I’ve learned that being right and being effective are different things. I could go tell them my concerns. They probably wouldn’t listen. I’d burn some goodwill. And in six months, nobody will remember that I called it, they’ll just remember I was the guy who tried to block their work”.

When you’re earlier in your career, you want to believe that good ideas win on merit, that if you just explain clearly enough, people will see reason. It took me quite some time to accept that big companies don’t work that way.

But this doesn’t mean you stop caring. It means you get strategic about when to spend your credibility. Pick the battles where you can actually change the outcome, where your team will be hurt if you stay silent, where the cost of being wrong is low but the cost of the project failing is high.

And for everything else? You vent to colleagues, you make quiet contingency plans, and you watch. Sometimes you learn something. Sometimes you’re wrong and the project actually works. And sometimes you get to feel that grim satisfaction of predicting exactly how things would fall apart.

None of this is as satisfying as fixing everything. But it works and keeps me sane.

One Number I Trust: Plain-Text Accounting for a Multi-Currency Household

Thu, 01 Jan 2026 10:09:00 +0100

Two people. Eighteen accounts spanning checking, savings, credit cards, investments. Three currencies. Twenty minutes of work every week.

One net worth number I actually trust.

The payoff: A single, trustworthy net worth number growing over time.

No app did exactly what I needed, so I built my own personal finance system using plain-text accounting principles and a powerful Python library called Beancount. This post shows you how I handle imports, investments, multi-currency, and a two-person view.

How I got here

It all started during the 2021 tax season. I had blocked out an entire weekend and was juggling statements, trying to compute capital gains, stressing about getting the numbers mixed up. “This is chaos”, I thought. “There must be a way to simplify this with automation”. Being a software engineer, I did what felt natural and hacked together a bunch of scripts on top of a database.

Though it worked and I kept using it day-to-day, by the next tax season the cracks became obvious. The code was hard to debug, random transactions went missing, and worst of all, the balances the scripts computed didn’t match the balances on my statements. I tried to fix it but the more I tried, the more I felt lost about what the system was really doing. Eventually I just gave up.

Why did I fail so spectacularly? My entire approach was flawed from the start! I’d ignored centuries of accounting wisdom and repeated fundamental mistakes humanity solved long ago. So I learned from my mistakes and did the research. And over time I incrementally discovered double-entry bookkeeping, plain-text accounting and Beancount.

Fast forward to today, and I have a flexible, powerful, and private system, fully customized to how my brain works. Most transactions import automatically from PDF statements (counterintuitively, it’s often more reliable than CSV!). Tax time is a simple matter of checking always-fresh reports and copying numbers over. The weekly ritual is simple: download statements, categorize transactions in a web UI, run a bunch of scripts to regenerate, commit (I walk through this in more detail later).

However, I want to be realistic: building a system like this takes time and effort.¹ You will need to learn some basic accounting concepts, be comfortable with Python, and consistently spend time every week keeping things up-to-date. If your finances are simple or you just want day-to-day budgeting, this is almost certainly overkill. Apps like YNAB or even the humble spreadsheet work great.

But if you want uncompromising control over how you look at your finances, read on.

Chapter 1: The Concepts

Double-entry bookkeeping

Suppose on a Saturday, I transfer money from my checking account to a savings account. The money leaves on the same day but doesn’t show up on the other side until Monday. So where was it for those two days?

In a “normal”² personal finance system, the answer would be that it was just gone. That is, for those two days, there would be a drop in the total money in two accounts. But this is weird because in reality my “net worth” did not change, yet there’s no good way to represent this.

Or suppose I pay $90 for a dinner for me and two friends. They pay me back a week later. Again, in this case the money is “gone” for that week. And even worse, the full $90 would be categorized as a “restaurant expense” while each $30 my friends paid would be “income”. But this is wrong. My expense is just $30 and the money they give me should be matched against the $60 they owe me.

Both of these are fundamental problems with how so-called “single-entry bookkeeping” works: each account’s transactions and balance are tracked individually but without the context of the “whole”. In the case of the transfer, because we’re looking at each account in isolation, we lose the fact that even though the money has left one account, it’s really still part of the “pool of money” that belongs to us. Similarly, when our friends pay us back, we’re not tracking the fact that our friends owe us money when the original transaction happened and their payment later neutralizes the debt.

Double-entry bookkeeping is the solution to both these problems. Businesses have been using it for hundreds of years³ to run their accounts, and it has powerful yet elegant ways to solve these problems and many others too.

Let’s consider again the transfer. In double-entry bookkeeping, we would represent the initial move of money as:

Bank-Checking -1000
Transfer-In-Flight +1000

And when it arrives:

Transfer-In-Flight -1000
Savings-Account +1000

In both cases, we see the “golden rules” of double-entry bookkeeping:

Every transaction has at least two sides and the sum of all the sides is zero. -1000 + 1000 = 0. That is, transactions always “balance”.
Every side of a transaction is an account, whether it exists in the real world or not.

It should be clear that “Bank-Checking” and “Savings-Account” are labels for your checking and savings accounts, respectively. But what is “Transfer-In-Flight”?

Well, it’s also an account! It’s not an account you’ll find on your bank’s website, but within the double-entry system, it’s just as real. Concretely, accounts in double-entry are just labels for a “bucket of money”. So there’s no “category to put this transaction under”, no “expense tracking”, no special “transfer tag”. Everything is an account.

In this specific case, Bank-Checking, Savings-Account, and Transfer-In-Flight are all a specific type of account: they are Asset accounts. Assets are stuff you own; these can be real accounts (bank account, savings, stocks, bonds) or conceptual accounts (money in transit between accounts).

Now let’s consider the dinner example. There are 4 sides to the transaction:

Credit-Card -90
Restaurant-Expense +30
Owes-Me:Alice +30
Owes-Me:Bob +30

Again, -90 + 30 + 30 + 30 = 0. It balances. All of Credit-Card, Restaurant-Expense, Owes-Me:Alice, and Owes-Me:Bob are just accounts.

Credit-Card is a different type of account though: it’s a Liability. Liabilities are the opposite of assets: instead of stuff you own, they’re stuff you owe to someone else. So for example, loans, credit cards, and mortgages are all liabilities.

Why negative 90? The rule is always: negative means money flowed from this account and positive means it flowed into this account. The credit card company fronted you $90, so that money flowed from your credit card to fund the purchase.

Restaurant-Expense is yet another type of account, an Expense account. Expense accounts are money “leaving your world”. So any time you spend some money and you no longer have access to it, that’s an expense.

Finally, Owes-Me:Alice and Owes-Me:Bob are also Assets. Alice and Bob have promised to pay you back, and that promise has value, $30 each. It’s not cash in your pocket, but it’s money you have a claim on. In double-entry, anything with economic value you control is an asset, whether it’s a bank balance or an IOU.

Later, when Alice pays you back:

Bank-Checking +30
Owes-Me:Alice -30

This is just money moving from the “virtual” Owes-Me:Alice to the “real” Bank-Checking account. Both of these are still assets; it’s just the type of asset that’s changing. So no money has “entered the system” at this point. You’re just settling the debt Alice owed you.

Let’s take one last example: a paycheck.

Bank-Checking +3000
Salary -3000

Bank-Checking is an Asset as we’ve learned. But Salary is a new account type, an Income account. Just like Assets and Liabilities are opposites, so are Income and Expenses. Where Expenses are money leaving your world, Income is money entering it.

Money flowed from Salary (source, negative) to Bank-Checking (destination, positive). The sign feels backwards: “I received money, so why is Income negative?” Because the sign shows direction of flow: income is where the money came from, and your bank is where it went to.

This is the one part of double-entry that takes repetition.⁴ Don’t try to make it intuitive; just trust the invariant: if your transaction sums to zero, you’ve got the signs right. After a dozen transactions, the pattern becomes automatic.

These four types of accounts cover 99% of what you’ll do:

Assets: stuff you own (bank accounts, cash, investments, money owed to you)
Liabilities: stuff you owe (credit cards, loans)
Income: money entering your world (salary, interest, dividends)
Expenses: money leaving your world (groceries, rent, restaurants)

There’s a fifth type, Equity, which is a catch-all “this money doesn’t fit elsewhere” bucket.⁵ Suppose you start tracking an account that already has $1000 in it; that money came from somewhere but you don’t have a record of that. It can’t be income because you already had it, and the other types don’t fit. That’s a good sign it belongs in equity. The good news is that you rarely interact with Equity directly. Generally, the software handles it for you, but there are some exceptions that we’ll cover in Chapter 2.

There’s so much more that could be said about double-entry bookkeeping. For further reading, I particularly like the double entry explainer in the Beancount docs. It goes through some more examples and expands into a bunch of related topics.

But we now have the foundation which ensures that every transaction balances, every dollar is accounted for, and nothing slips through the cracks. But we still need a way to actually record and store these transactions.

Plain text accounting

One of the things I learned from writing my own finance system is that auditability is king. You need the ability to eyeball a transaction, ask yourself “does this look right,” and fix it if it doesn’t. And nothing beats being able to see and edit any transaction you’ve ever made in a text editor.

That’s one of the main things that drew me to the philosophy of Plain Text Accounting. This is a set of principles on using plain text files as the “immutable source of truth” of your finances and then building scripts and tools on top of them to process, analyze, and visualize them.

There are many other advantages to this I’ve come to appreciate over the years:

Everything is version controlled. You can store these transaction files in a git repo, which has powerful effects. You can look at diffs to see what changed on any day. You can git blame any transaction to see when and why it was added. You can git tag important states of the repo (e.g. when taxes were filed, when a new job was started, when a big refactoring happened). You can git checkout any previous state to see e.g. “how did my repo look last year”.

It’s private so you never have to trust any third party with all your financial details. Everything can be stored in locations that you fully control.

There’s no lock-in. Because everything is just a plain text file, it’s trivially easy to change how you want things to be represented: you don’t have to deal with apps with broken or messy CSV exports making it difficult to take your data elsewhere.

It’s scriptable. If you want to refactor something, compute a new breakdown or even rewrite your system entirely, all you need is to write a script. Whether you write it yourself or prompt an LLM to do it for you, the text-based format makes automation trivial.

Plain text gives you the abstract idea of “storing transactions in text” but there’s still a bunch of questions. What’s the transaction syntax? How do you parse your files? How do you validate that everything balances, compute totals, and let you query the results? That’s where Beancount comes in.

Introducing Beancount!

Over the years, people have written many plain-text accounting tools which answer all the questions above. The main ones you’ll find which have gained a lot of popularity are Ledger, hledger, and Beancount. I’ve used each of them at some point in my plain text journey and all are solid choices. But I ended up on Beancount for a few reasons.

It’s a Python library, not just a command-line tool. I can write importers that parse my bank’s PDF statements, generate transactions programmatically, and build custom reports, all in a language I already know.

Strictness by default. Accounts must be declared before use, so typos get caught immediately. Transactions must balance and there are immediate error messages if they don’t. The tool catches mistakes early rather than letting them propagate.

Plugin and tool ecosystem. Beancount has a very rich set of libraries and tools which build on top of and integrate with it. Along with the core project, you get access to any and all of these projects you want to use. We’ll discuss this much more later.

Fava. The web UI which sits on top of the Beancount engine. It’s so good that people convert from other formats (using tools like ledger2beancount or gnucash2beancount) just to use it. Where Beancount gives you the reliable engine, Fava gives you the pretty yet powerful frontend:

Reports: balance sheet, income statement, transaction journal
Query editor: SQL-like queries, exportable to CSV
Charts: spending breakdowns, net worth over time, holdings by currency
Error highlighting: problems highlighted immediately
Extensibility: plugins like fava-dashboards and fava-portfolio-returns

With this, we now understand enough of the basic concepts for us to get started trying out Beancount!

Chapter 2: Getting Started

The best way to learn plain-text accounting is to roll up your sleeves and try it out. So let’s pause the theory for a moment and get a real ledger running on your machine.

I’ve built a companion repository, LalitMaganti/beancount-blog-examples, which contains a cut-down version of the system I use day-to-day. The repo is organized into folders that match the chapters of this post (chapter-2/, chapter-3/, etc.), each building on the previous. It also includes a demo/ folder with 2+ years of synthetic history if you want to see the end result immediately.

To get started, clone the repository:

git clone https://github.com/LalitMaganti/beancount-blog-examples.git
cd beancount-blog-examples

# Run the demo to see the end result
./scripts/quickstart.sh demo

# Or start Chapter 2 to follow the guide
./scripts/quickstart.sh chapter-2

This will set up a Python environment, install dependencies, and open Fava at http://localhost:5000. Here’s what you should see.

Fava's Trial Balance view shows assets, income, and expenses in one place.

Sunburst charts make it easy to spot your biggest expense categories instantly.

Click around the different tabs. You’ll find that Fava gives you the following:

Balance sheet: the state of your accounts at the current point in time. Basically think of it as an aggregate view of all your finances. You can answer questions like “what’s my net worth now?”, “how much money do I owe across my credit cards?” or “how much have I gained from investments?”
Income statement: the sum of all the money flows into/out of your accounts. Think of it as a sum of all the income and expenses across time. You can answer questions like “how much did I earn from my job?”, “how much did I spend on Amazon?” or “did I spend more or less than last year?”
Transaction history: a flat list of all transactions you have in your journal. A way to see and search any transaction you’ve made across any accounts in your system.

The Query Console lets you run SQL-like queries against your financial data.

There’s much more to Fava’s features (queries, multi-currency, plugins) as we’ll see later on in the post.

You can also explore the text journal itself using a text editor. For example, here’s a grocery run and a payslip in the Beancount transaction format:

; chapter-2/src/transactions.beancount

; Beancount auto-fills the second amount when it can be inferred.
2024-01-15 * "Tesco" "Weekly groceries"
 Expenses:Groceries 85.50 GBP
 Assets:Lalit:UK:HSBC:Current:GBP

2024-01-25 * "Google" "January salary"
 Assets:Lalit:UK:HSBC:Current:GBP 3200.00 GBP
 Income:Lalit:UK:Google:Salary

Go through the transactions and get a feel for the format. It might seem alien at first but trust me when I say soon it’ll feel like the most natural thing in the world!

With a VS Code extension, you get syntax highlighting and auto-completion for your accounts.

Now that you have a working system, I want to share the hard-won insights that aren’t in the official docs. This post isn’t going to be a full Beancount tutorial. The official docs are excellent for that (Fava even has a demo that you can try without downloading anything!).

Instead, I want to focus on the architecture: the decisions I wish I’d made correctly from day one. I’m writing this as if I’m speaking to my past self.

Start with one account

You don’t need to track everything on day one. Pick one account. Your main checking account is a good start. Get comfortable with the flow. Import statements, categorize transactions, check that balances match. Once that feels solid, add another account. Then another.

I started with my HSBC current account. Now, I have my whole financial life inside the system and I trust it wholeheartedly. But this happened one account at a time. If I tried to do everything in one go, I would certainly have been overwhelmed and given up on the whole thing.

Opening balances

Once you’ve picked your first account, you face an immediate problem: you may have opened that account years ago and there might already be thousands of transactions over that time. Trying to import them all in one go is another sure path to being overwhelmed and giving up.

Instead, a better idea is to pick a “starting date” at which you say “I will import everything from this day onwards”. But that poses its own problem: you already had money in that account, how do you tell Beancount it exists?

Well, Beancount has a pad directive that creates the balancing entry automatically:

; chapter-2/src/balance.beancount
2024-01-01 open Equity:Opening-Balances
2024-01-01 pad Assets:Lalit:UK:HSBC:Current:GBP Equity:Opening-Balances
2024-01-02 balance Assets:Lalit:UK:HSBC:Current:GBP 1500.00 GBP

The pad directive tells Beancount: “whatever amount is needed to make the balance assertion true, take it from Equity:Opening-Balances and put it in this account”. This is one of the rare cases you actually have to think about equity accounts (though not much beyond blindly using Equity:Opening-Balances!).

Structure That Scales

Adding a first account is easy and the second is straightforward, but adding a third, fourth, fifth… and you can easily find that things start becoming jumbled and messy. Just like code, putting a little bit of thought into the organization upfront goes a long way. This part covers the architectural decisions you’ll thank yourself later for.

Naming Asset and Liability accounts

The structure of asset and liability account names is very important, much more than I initially gave them credit for. It’s a good idea to keep as much information in them as possible. Here’s what I’ve settled on:

; chapter-2/src/accounts.beancount
2024-01-01 open Assets:Lalit:UK:HSBC:Current:GBP GBP
2024-01-01 open Assets:Lalit:UK:Barclays:Current:GBP GBP
2024-01-01 open Liabilities:Lalit:UK:AMEX:GBP GBP

The pattern is:

Type:Person:Region:Institution:Account:Currency

Why this structure? Because it’s a lot easier to remove detail than add it in later! I initially started by not having the country, the currency or my name in the account. But over time, I wanted to understand:

How much money I have in the UK vs the US?
How much cash (i.e. not investments) is in a certain currency?
How much of our household wealth was in my wife’s accounts vs my own (discussed in more detail in Chapter 6)?

Having this information in the account name is great because Beancount’s SQL syntax makes it very easy to filter on account names. Want “all UK assets”? Filter on :UK:. Want “all HSBC accounts”? Filter on :HSBC:. Want “all GBP cash”? Filter on :GBP.

Powering up with Plugins

Once you have a structure, you want to ensure it stays clean. This is where Beancount’s Plugins come in.

Plugins are Python scripts that run when your ledger loads. They can validate data, modify entries, or even generate new transactions automatically. You load them in your journal file like this:

plugin "beancount.plugins.check_commodity"

Remember the transfer-in-flight pattern from Chapter 1? Here’s how it looks in Beancount:

; chapter-2/src/transactions.beancount
; Money leaves on Saturday
2024-03-16 * "Transfer to Barclays"
 Assets:Lalit:UK:HSBC:Current:GBP -1000.00 GBP
 Assets:Lalit:Transfers:Internal 1000.00 GBP

; Money arrives on Monday
2024-03-18 * "Transfer from HSBC"
 Assets:Lalit:Transfers:Internal -1000.00 GBP
 Assets:Lalit:UK:Barclays:Current:GBP 1000.00 GBP

Once both transactions are recorded, the transit account balance returns to zero, confirming the transfer is complete. If you record one leg of a transfer but forget the other, the transit account will simply show a non-zero balance.

Understanding where a non-zero balance in transfers is coming from is handled by my absolute favorite plugin, beancount_reds_plugins.zerosum. It’s responsible for matching both sides of a transaction and moving it to a separate account, meaning my Transfers:Internal account only contains the actual “pending” transactions. Making this account empty is a surprisingly satisfying little “minigame” during my weekly imports (though it never lasts for long!).

There are also a couple more plugins that make handling closed accounts better:

beancount.plugins.close_tree - When you close an account, automatically closes all child accounts too. Useful as you move your banking between institutions.
beancount_checkclosed.check_closed - Validates that closed accounts have zero balance and no transactions after the close date.

The boundary of your system

As you add accounts one by one, you’ll inevitably see money flowing to places you haven’t set up yet. Say you transfer £500 to a Natwest savings account you haven’t added to the system. Where does it go?

Specifically, use a named placeholder account in Equity:Transfers:

; chapter-2/src/transactions.beancount
2024-03-15 * "Transfer to savings (not yet tracked)"
 Assets:Lalit:UK:HSBC:Current:GBP -500.00 GBP
 Equity:Transfers:Natwest-Savings 500.00 GBP

This essentially says: “£500 went to Natwest Savings, which I’m not tracking yet”. Putting it in an equity account means it doesn’t pollute your balance sheet with incomplete information or your income statement with false expenses.

Note also the best practice of using a named equity account per untracked destination, not a generic bucket; this was a mistake I made when I did this initially. You’ll thank yourself when you import your Natwest savings account later, you can just do a search-replace to rename the accounts ⁶.

Organizing your repo

As you add these patterns (transit accounts, multiple institutions, liability accounts) your single journal.beancount file will start to become unwieldy. Just like good software architecture, you want to organize upfront to be easy to maintain as the system continues to grow.

This is what the structure looks like with the concepts we have right now (it’ll get more complicated as we go deeper!):

chapter-2/
├── journal.beancount # Main entry point, includes everything
├── src/
│ ├── accounts.beancount # Account definitions
│ ├── transactions.beancount # Primary transaction ledger
│ └── balance.beancount # Balance assertions

src/ is what you write and edit (your “code”)
journal.beancount is the entry point that includes everything
Later, data/ will hold inputs from the outside world (raw statements: PDFs, OFX, CSVs)

This organization is a small change but can make a big difference in your subconscious feeling about the state of your finances!

Exit ramp

At this point, you have a solid foundation for tracking multiple accounts. If you stop here, you have a robust, auditable system for manual bookkeeping. You could continue adding transactions by hand indefinitely, and you’d still be miles ahead of any spreadsheet in terms of correctness and visibility.

But manual entry is a chore, and as your financial life grows, it becomes a bottleneck. In the next chapter, we’ll see how to automate the tedious part: getting transactions from your bank statements into your ledger without losing the control that plain-text accounting gives you.

Chapter 3: Automated Import

Automation is king, but harder than it looks

Inputting transactions by hand works for some people, but I don’t have the patience for it. Ever since I was young, I’ve always wanted to automate everything: it’s the reason why I became a software engineer in the first place!

But full automation is a dead end. Most banks don’t offer APIs, and scraping breaks constantly. 2FA flows change ⁷, websites get redesigned, sessions expire. I tried this route, and it wasn’t worth it. Even in the US where aggregators like Plaid exist, coverage is patchy.⁸ In the UK, it’s impossible.

The hierarchy of data sources

So what actually works? Well ideally your financial institution gives you something you can write a script against. But what that might be is non-obvious and counter-intuitive. Here’s the hierarchy I’ve landed on over time:

OFX is the gold standard. If your bank offers it, use it. The format is standardized, transactions have unique IDs, and deduplication is straightforward. Life is easy if you have good OFX.
CSV is the deceptive runner-up. It seems like the logical choice. Structured data, right? But in practice, bank CSVs are often afterthoughts. I’ve seen column formats change without notice, “CSVs” that are actually weird custom formats spread over multiple lines, and rows coalesced in ways that lose critical information (like cost basis).

So what do you do when OFX isn’t available and CSV isn’t trustworthy? You turn to an unlikely hero.

Why PDFs beat CSVs

It sounds backwards, but PDFs are often the most reliable data source available.

Banks have a strong incentive to get PDFs right. Customers actually read them. They’re legal documents that get printed and filed. If a bank messes up a PDF statement, they hear about it immediately. If they break a CSV export, it might go months without anyone noticing.

The key insight is that bank statement PDFs are almost always columnar. Of course, this relies on the PDF having a proper text layer; if your bank sends you scanned images, you’re out of luck (though I’ve yet to encounter one that does). When you convert them to text while preserving the layout, you get something that looks like this:

Date Details Paid out Paid in Balance
15 Jan 24 TESCO STORES 1234 42.50 1,457.50
16 Jan 24 TFL TRAVEL 6.80 1,450.70
25 Jan 24 GOOGLE SALARY 3,200.00 4,650.70

The columns are aligned by spaces, which means you can parse them as fixed-width data. The approach works in three steps:

Convert PDF to text: Run pdftotext -layout statement.pdf statement.txt. The -layout flag preserves the original column alignment.
Find the table boundaries: Bank statements have predictable markers. HSBC uses “BALANCE BROUGHT FORWARD” at the start and “BALANCE CARRIED FORWARD” at the end. You extract just the transaction rows between these markers.
Parse with fixed-width columns: Pandas’ read_fwf function is designed exactly for this. You put this logic inside the extract() method of your beangulp importer class, where it converts the text into a DataFrame:

# Inside your Importer class
df = pd.read_fwf(
 io.StringIO(text),
 colspecs=[(0, 10), (10, 40), (40, 52), (52, 64), (64, 80)],
 names=['Date', 'Details', 'Paid out', 'Paid in', 'Balance']
)

The column positions come from inspecting the header row. In practice, I detect them dynamically by finding keywords like “Paid out” and “Paid in” in the header and using their character positions.

Once you have a DataFrame, generating Beancount transactions is straightforward. You write a small class that iterates through this DataFrame and maps each row to a Beancount Transaction object, filling in the date, amount, and payee. See my HSBC importer for a working example.

For 95% of my banks, this approach works great. However, there is one bank where the text spacing becomes very strange and so I need to use something else. That’s when I turn to Tabula, a Java CLI that extracts data tables from PDFs, even very complex ones.

The main reason I don’t use it all the time is that it’s much slower. But it also succeeds in the cases where pdftotext fails. I run it to get a structured JSON output, from which I create transactions (see my HSBC US credit card importer).

I’m sure some readers will have worries about the fragility of what I’ve described here. I can tell you from experience that in three years, neither my UK nor my US HSBC PDF importer has ever broken. Neither has my Schwab one, and my Aviva one has only needed a single change. So I can personally vouch that this approach works and works well.

From raw data to transactions

OK, so now you have statements. What do you do then? There are two pieces to the puzzle: parsing statements into transactions, and categorizing those transactions.

For parsing, Beancount has an official importer framework called beangulp. You write a Python class that knows how to read a particular file format (the skeleton importer in chapter-3/ already uses this API). Beangulp handles the mechanics: identifying which importer handles which file, extracting transactions, and deduplicating against existing entries.

But beangulp just extracts transactions without making any judgment on which account the transaction should be booked against. It doesn’t know that Tesco is groceries or that British Airways is an airline. I go to a new restaurant. How does the system know that is a restaurant?

This leads to a very important conclusion: we cannot fully automate importing transactions for bank accounts and credit cards. However, that doesn’t mean we have to enter things manually either. There’s a middle ground.

Enter beancount-import⁹ (Note: this is a standalone tool, distinct from Fava’s built-in import features). It’s a web UI that uses your beangulp importers and adds a categorization layer on top. You run it, it opens in your browser, and it presents pending transactions one by one for review. Think of it as a staging area where you approve or tweak before anything hits your ledger.

For each transaction, it shows the raw data from your statement alongside a suggested categorization. You can accept the suggestion, override it with a different account, or skip it entirely. Once you decide, it moves to the next one. The interface is simple, mostly keyboard-driven and optimized for speed.

The import UI allows you to manually categorize transactions, like this Tesco grocery run, while the system learns your preferences.

The categorization uses machine learning (old-school decision trees, running locally, no LLMs, no cloud). It learns from your previous choices: the first time you see “Tesco”, you pick “Expenses:Groceries”, and the second time it auto-suggests and you just hit Enter. After a few weeks, the system should know 90% of the types of transactions you make and it’s easy to correct the ones which it doesn’t.

Once you’re proficient, a month’s worth of transactions takes 5-10 minutes. Most are repeats and you’re just hitting Enter. You only pause on genuinely new merchants. This is the core of my weekly ritual, and why I’ve found categorization has not become a chore, even after doing it for years.

Exit ramp

With automated imports and semi-automated categorization, the “hard work” of bookkeeping is mostly solved. For many, this is the endgame: a perfect record of where every penny went, updated in minutes each week.

But your net worth isn’t just cash in a bank account. It’s also the stocks, bonds, and funds that grow (or shrink) over time. Tracking these requires a few more tools to handle cost basis, dividends, and market prices. We’ll tackle those in Chapter 4.

Chapter 4: Investments

Now for investments. It’s where things get more interesting, and I think there’s less material out there covering the nitty-gritty. Here are some lessons I’ve learned over the years.

The Unified Mental Model

The most important thing to realize is that Beancount treats everything as a commodity.

A share of Apple (AAPL) is a commodity. A US Dollar (USD) is a commodity. A British Pound (GBP) is a commodity. While Beancount can infer these on the fly, you’ll typically declare them explicitly in your journal (e.g., 2024-01-01 commodity AAPL).¹⁰

This explicit declaration is a small but critical architectural win: it prevents a simple typo from creating a phantom currency, and it provides the metadata that advanced reporting plugins (like those used to calculate your portfolio performance) rely on to work correctly.

This means you don’t “buy stocks with money”. You simply exchange one commodity for another. The syntax for buying shares is identical to the syntax for exchanging currency.

The Account Structure

Just like with bank accounts, I break investments down by institution and security.

; chapter-4/src/accounts.beancount
2024-01-01 open Assets:Lalit:US:IB:Brokerage:USD USD
2024-01-01 open Assets:Lalit:US:IB:Brokerage:AAPL AAPL
2024-01-01 open Assets:Lalit:UK:Vanguard:ISA:GBP GBP
2024-01-01 open Assets:Lalit:UK:Vanguard:ISA:VWRL VWRL

Cash in a brokerage is just another holding named by currency (GBP, USD), while stock holdings use their ticker (AAPL, VWRL).

But you also need to track the flows generated by these assets: capital gains, dividends, commissions, and withholding taxes. I create specific accounts for each security:

2024-01-01 open Income:Lalit:US:IB:Brokerage:AAPL:Dividends USD
2024-01-01 open Income:Lalit:US:IB:Brokerage:AAPL:Capital-Gains USD

Why so granular? It’s the same reason I name assets fully: aggregation up the tree is trivial; disaggregation after the fact is impossible. If you track all your dividends in a single Income:Dividends account, it’s easy to know “how much dividends did I earn total?”. But if you want to know “what was my AAPL dividend yield this year?”, you’re out of luck. Track at the leaf (Income:IB:AAPL:Dividends), and you can answer both questions.

The Notation

With our accounts defined, we can now record the actual movement of assets. We use {} to denote cost (what we paid per unit) and @ to denote price (what the unit is worth now).

Example 1: Buying Stock Exchanging 1850 USD for 10 shares of Apple.

2024-01-10 * "BUY AAPL"
 Assets:Lalit:US:IB:Brokerage:AAPL 10 AAPL {185.00 USD} @ 185.00 USD
 Assets:Lalit:US:IB:Brokerage:USD -1850.00 USD

The {185.00 USD} is the cost basis and the @ 185.00 USD is the price. Beancount uses the cost basis to track lots and calculate capital gains.

Note: Cost basis rules depend heavily on where you are. In the US, you track cost basis of individual lots. In the UK, we have special “Section 104” pooling rules.¹¹ I discuss this more in Chapter 5.

Example 2: Buying Currency Exchanging 950 GBP for 98,000 INR.

2024-03-02 * "Wise" "GBP to INR"
 Assets:Lalit:UK:Wise:INR 98000.00 INR @@ 950.00 GBP
 Assets:Lalit:UK:Wise:GBP -950.00 GBP

In the second example, @@ specifies the total cost rather than the per-unit cost.

Example 3: Selling Stock Selling 5 shares of Apple at $190 (bought at $185).

2024-01-30 * "SELL AAPL"
 Assets:Lalit:US:IB:Brokerage:AAPL -5 AAPL {185.00 USD} @ 190.00 USD
 Assets:Lalit:US:IB:Brokerage:USD 950.00 USD
 Income:Lalit:US:IB:Brokerage:AAPL:Capital-Gains -25.00 USD

Here we specify the lot we’re selling ({185.00 USD}) and the price we’re selling it at (@ 190.00 USD). The difference is the capital gain (or loss).

But the principle is identical: Assets:Wise:INR and Assets:Brokerage:AAPL are just accounts holding commodities.

Automation

Investments are very different from normal accounts in that they can be fully automated. No categorization needed: a buy is a buy, a dividend is a dividend. You don’t have new merchants to worry about.

This means you can skip beancount-import’s web UI and run your beangulp importers directly with output going straight to the ledger. To help inspire you, I’ve open-sourced my personal collection of importers (IB, Vanguard, Schwab, and more) in the beancount-lalitm repo.

Handling Account Sprawl

However, one annoyance is that creating those granular accounts for every single stock (...:AAPL:Dividends, ...:AAPL:Commissions, etc.) is tedious. To solve this, I wrote the ancillary_accounts plugin. Instead of manual account creation, you just add metadata to the main holding account:

2023-02-01 open Assets:Lalit:US:IB:Brokerage:BAC BAC
 ancillary_commission_currency: "USD"
 ancillary_distribution_currency: "USD"
 ancillary_withholding_tax_currency: "USD"
 ancillary_capital_gains_currency: "USD"

The plugin automatically generates the corresponding income and expense accounts for you.

Corporate Actions

I also use a plugin called stock_split to handle corporate actions. It retroactively adjusts historical transactions when a stock splits, keeping quantities and prices consistent with post-split values so your charts don’t show a sudden, fake drop in value.

The Value of Things (Prices)

We have the quantities (10 AAPL, 98,000 INR), but to calculate a single “Net Worth” number, we need to know what they are worth in your home currency. This requires prices for both:

; chapter-4/src/prices.beancount
2024-01-10 price AAPL 185.50 USD ; Stock price in USD
2024-01-10 price USD 0.79 GBP ; Currency price in GBP

I automate this using a daily CI job. A script fetches the latest stock prices and forex rates from AlphaVantage and commits them to prices.beancount. You can find the script here.

In practice, I actually have three price files:

prices.beancount - auto-fetched daily for as many securities as possible.
prices-manual.beancount - for securities without automatic feeds (like some pension funds). I input these manually once a month.
prices-delisted.beancount - historical prices for securities no longer trading. This saves me from making API calls which would fail anyway.

The Payoff

With this data, Fava comes alive. To see the full potential of these reports, I’ve included a demo/ folder in the companion repo with 2+ years of history. Run ./scripts/quickstart.sh demo and you’ll see the payoff.

The Holdings page now shows your positions with their cost basis and current market value.

The Holdings report automatically calculates the market value of your assets using live price data.

In the demo environment, you can also see how plugins like fava-dashboards build custom visualizations. The plugin uses beanquery (Beancount’s SQL-like query language) to fetch data and renders interactive charts. It’s the best way to track long-term trends and asset allocation at a glance.

Custom dashboards (shown here using the demo data) allow you to track long-term trends and asset allocation at a glance.

And with fava-portfolio-returns, you can calculate your true Time-Weighted Return (TWR) and Internal Rate of Return (IRR) to see if you’re actually beating the market. It accounts for cash flows properly, so adding money mid-year doesn’t inflate your returns.

The portfolio returns plugin (using the demo data) calculates your actual investment performance, net of cash flows.

Exit ramp

You now have a system that tracks your entire financial world: from the coffee you bought this morning to the capital gains in your brokerage account. For most people, this is a complete solution.

However, as you collect more data, you’ll find you want to look at it in different ways. Maybe you want a simplified view for daily use and a detailed one for tax season. Or maybe you’re not the only person in your household. In the final chapters, we’ll see how to scale this system to handle multiple views and multiple people.

Chapter 5: Multiple Views

Day-to-day, I want a simple view of my finances. Take-home pay as a single number, investments without tax calculations cluttering the screen. But at tax time, I need detail. Every payslip line item and capital gains calculated the way HMRC wants them. Recording the same transaction twice would be maintenance hell. So instead, I extract multiple views from a single journal.

I think of these as “lenses” on the data. Some lenses aggregate: rolling up transactions into balances, summaries, or dashboards. Others transform: collapsing detail you don’t need day-to-day, or expanding it when you do. Both read from the same source files; nothing is duplicated.

Aggregated views

Fava is great for interactive exploration, but I also want textual snapshots I can version control. I have a script that generates daily summaries showing account balances, and a CI workflow that commits them automatically. In my setup this runs on a self-hosted Gitea instance on hardware I control, so the raw ledger never leaves machines I own. If you prefer, you can keep everything local-only or push to an encrypted remote; GitHub Actions works the same way if you’re comfortable with that trade-off. This gives me:

A record of how balances changed day to day
An immutable snapshot at tax time of what the system showed
Git as audit trail: “What was my net worth on March 15th 2023?” is answerable with git checkout

The key script is archive.py. It uses the beanquery library to write SQL scripts over your journal and generate textual reports:

# Generate balance sheet in GBP
sql = '''
 SELECT account,
 round(sum(number(convert(value(position, '2024-12-31'), 'GBP', '2024-12-31'))), 2) as value
 FROM OPEN ON 2024-01-01 CLOSE ON 2024-12-31 CLEAR
 GROUP BY account
 HAVING round(sum(number), 2) != 0
 ORDER BY account;
'''

I run this for each calendar year and tax year, generating files like:

networth.txt - Single-line net worth in each currency
balance-sheet.txt - Net worth breakdown by account
holdings.txt - Investment positions with cost basis and market value

Here’s what networth.txt looks like:

 gbp usd
------------- -------------
 15978.42 GBP 19973.02 USD

This is the concrete “one number I trust”: a single net-worth snapshot in my reporting currency, generated from the full ledger and price data.

And holdings.txt:

account units curr avg_cost price book_val mkt_val
--------------------------------- ----- ---- -------- ----- -------- -------
Assets:Lalit:UK:HSBC:Current:GBP 4914.50 GBP 1.00 1.00 4914.50 4914.50
Assets:Lalit:UK:Vanguard:ISA:VWRL 20.00 VWRL 96.00 97.50 1920.00 1950.00
Assets:Lalit:US:IB:Brokerage:AAPL 5.00 AAPL 146.15 150.40 730.75 752.00

My Gitea workflow is very simple too:

on:
 schedule:
 - cron: '00 7 * * *' # Run daily at 7am

jobs:
 update:
 steps:
 - run: uv run scripts/archive.py outputs/ journal.beancount 2024-01-01 2024-12-31
 - run: git commit -am "Regen reports" && git push

Transformed views

Sometimes I want to change how transactions work fundamentally. This is a more advanced technique: while Aggregated views read data, Transformed views temporarily rewrite it in memory to simplify reality.

I have three transformed views, each for a different purpose:

Net - my daily driver. Collapses payslip details into a single take-home number.
Gross - breaks down payslip line items for tax time analysis.
CGT - a view that includes a “virtual currency” tracking capital gains the way my tax authority calculates them.

The linchpin is the rename_accounts plugin. It lets me keep one copy of all transactions and rename accounts on the fly to show or hide detail.

Gross vs net payslip

Let me start with the simpler example. In gross view, my payslip shows every line item:

2024-01-25 * "Google" "January salary"
 Assets:Lalit:UK:HSBC:Current:GBP 3500.00 GBP ; Take-home pay
 Income:Lalit:UK:Google:Salary -5000.00 GBP ; Gross salary
 Expenses:Lalit:UK:Google:Income-Tax 1000.00 GBP ; Tax withheld
 Expenses:Lalit:UK:Google:National-Insurance 400.00 GBP ; NI contribution
 Expenses:Lalit:UK:Google:Pension 100.00 GBP ; Pension contribution

Useful for analyzing my tax situation. But day-to-day, I don’t care about the breakdown. In net view, the same transaction collapses:

; chapter-5/journal-net.beancount
include "journal.beancount"

plugin "beancount_reds_plugins.rename_accounts.rename_accounts" "{
 'Income:Lalit:UK:Google:Salary': 'Income:Lalit:UK:Google:Net-Income',
 'Income:Lalit:UK:Google:Bonus': 'Income:Lalit:UK:Google:Net-Income',
 'Expenses:Lalit:UK:Google:Income-Tax': 'Income:Lalit:UK:Google:Net-Income',
 'Expenses:Lalit:UK:Google:National-Insurance': 'Income:Lalit:UK:Google:Net-Income',
 'Expenses:Lalit:UK:Google:Pension': 'Income:Lalit:UK:Google:Net-Income',
}"

Because Income is stored as a negative number and Expenses as positive numbers, merging them into one account mathematically subtracts the tax from the gross pay, leaving just the net amount. The difference should be obvious if I compare the Income Statements on Fava:

The Gross view is essential for tax season, but the detailed line items for taxes and insurance often dwarf your actual spending data.

The Net view collapses those details into a single take-home number, making your everyday expenses much easier to analyze.

Tracking capital gains for tax

We can use this same renaming technique to handle a much more complex beast: Capital Gains Tax.

Your broker reports one gain number, but your tax authority may calculate another. In the UK, where I live, we have specific rules like “Section 104 pooling” (averaging cost basis) and “bed-and-breakfasting” (wash sale rules).

To handle this, I use a virtual currency called CGT-GBP that represents “pounds of gain HMRC cares about”. My plugin, uk_cgt_lots, calculates this number and automatically appends a self-balancing pair of Equity postings to the original sale transaction:

2024-06-15 * "SELL AAPL"
 Assets:Lalit:US:IB:Brokerage:AAPL -10 AAPL {150.00 USD} @ 175.00 USD
 Assets:Lalit:US:IB:Brokerage:USD 1750.00 USD
 Income:Lalit:US:IB:Brokerage:AAPL:Capital-Gains -250.00 USD
 ; The following postings are generated by the uk_cgt_lots plugin:
 Equity:Taxable-Capital-Gains 195.00 CGT-GBP
 Equity:Taxable-Capital-Gains-Placeholder -195.00 CGT-GBP

Since both legs are in Equity, they remain invisible on my Income Statement in my daily “Net” view. In fact, I use rename_accounts to collapse them into a single account so they net to zero:

; journal-net.beancount
plugin "beancount_reds_plugins.rename_accounts.rename_accounts" "{
 'Equity:Taxable-Capital-Gains-Placeholder' : 'Equity:Taxable-Capital-Gains',
}"

But when I want to see my tax liability, I switch to the CGT View. This view renames the “Placeholder” to a visible Revenue account:

; journal-cgt.beancount
plugin "beancount_reds_plugins.rename_accounts.rename_accounts" "{
 'Equity:Taxable-Capital-Gains-Placeholder' : 'Revenues:Taxable-Capital-Gains',
}"

Now, the -195.00 becomes Revenue, which shows up as profit on my tax report. The matching +195.00 remains in Equity. This allows me to have “Schrödinger’s Capital Gains”: they exist for the taxman, but not for my daily budget, all controlled by which view I load.

I don’t calculate the tax owed since that’s too complicated with allowances, rates, and bands; the system just tracks the gains. At tax time, I sum up the CGT-GBP balance and do the actual calculation on the tax form.

Exit ramp

By separating your “source of truth” from your “lenses,” you get a system that grows with you. You can add new plugins or virtual currencies to solve specific problems (like taxes) without ever touching the raw transactions you’ve already imported.

In the final chapter, we’ll see the ultimate application of this: combining two people’s financial lives into one unified view.

Chapter 6: Two People, One Number

I got married at the start of the year, which brought a fundamental change to how I manage my finances. While many couples use joint accounts, we prefer to keep our individual accounts and perform occasional “normalization” transfers. However, we view our combined resources as shared household wealth.

This created a reporting paradox that I had to solve in Beancount.

The Paradox

When I transfer £500 to my wife for my share of the bills, two things are true simultaneously:

The Individual Truth: From my perspective, £500 is “gone” (an expense). From her perspective, £500 has “arrived” (income).
The Household Truth: For the household, the net worth hasn’t changed. Money just moved from the left pocket to the right pocket.

In a traditional system, you usually have to pick one truth. In Beancount, we can have both.

The Composable Architecture

To solve this, I treat the household as a composable system of three distinct entities: Me, Her, and Shared Definitions. We use Beancount’s include feature to build the specific “lens” we need at any given moment:

Lalit’s View = Shared Definitions + Lalit’s Transactions
Wife’s View = Shared Definitions + Wife’s Transactions
Household View = Shared Definitions + Lalit’s Transactions + Wife’s Transactions + Translation Logic

For example, the household view literally just includes the other files (alongside the translation logic we’ll see in a moment):

; chapter-6/total/journal-net.beancount
include "../common/src/commodities.beancount"
include "../common/src/accounts.beancount"

include "../lalit/src/journal.beancount"
include "../wife/src/journal.beancount"

; ... Translation Logic follows

Implementation: The Directory Structure

This architecture is reflected directly in the repository structure:

chapter-6/
├── common/ # Shared configuration
│ └── src/
│ ├── accounts.beancount # Shared expense accounts (e.g. Expenses:Groceries)
│ └── commodities.beancount
├── lalit/ # My stuff
│ ├── src/ # My ledger
│ └── data/ # My statement PDFs/CSVs
├── wife/ # Wife's stuff
│ ├── src/ # Her ledger
│ └── data/ # Her statement PDFs/CSVs
└── total/ # Combined household view
 └── journal-net.beancount # Entry point with Translation Logic

The Rules of Engagement

For this to work without constant manual adjustment, we follow two simple rules:

Rule 1: Assets and Liabilities are Private. Bank accounts always include the person’s name in the path (e.g., Assets:Lalit:HSBC or Assets:Wife:HSBC). We never use a generic Assets:Checking account. Legal ownership of the cash always matters.

Rule 2: Expenses are Public. Shared expenses like groceries or electricity use a generic name without a person prefix.

; chapter-6/common/src/accounts.beancount
2024-01-01 open Expenses:Groceries GBP

When I buy groceries, I record it in my ledger using the shared account. We don’t track “who owes what” for individual grocery runs; we just track that the household spent the money. We accept that we lose the ability to split shared expenses by person, but the gain in simplicity is worth it.

The “Magic”: Solving the Transfer Paradox

Finally, we use the rename_accounts plugin in the total/ folder to resolve the transfer paradox.

In my ledger, a transfer looks like a simple expense:

; chapter-6/lalit/src/transactions.beancount
2024-01-20 * "Transfer to Wife"
 Expenses:Lalit:Transfers:Wife 500.00 GBP
 Assets:Lalit:UK:HSBC:Current:GBP

In her ledger, it looks like income:

; chapter-6/wife/src/transactions.beancount
2024-01-20 * "Transfer from Lalit"
 Assets:Wife:UK:HSBC:Current:GBP 500.00 GBP
 Income:Wife:Transfers:Lalit

The “Translation Logic” in the combined view renames these into a shared transit account:

; chapter-6/total/journal-net.beancount
plugin "beancount_reds_plugins.rename_accounts.rename_accounts" "{
 'Expenses:Lalit:Transfers:Wife': 'Assets:Household:Transfers:Internal',
 'Income:Wife:Transfers:Lalit': 'Assets:Household:Transfers:Internal',
}"

Now, when Fava loads the combined view, it sees £500 leave my account and enter Assets:Household:Transfers:Internal, and then £500 leave that same account and enter her bank account. The transit account nets to zero, and our household net worth remains unchanged.

The Result

This setup gives us the best of both worlds. I can maintain my own financial autonomy and see my personal “runway,” while we can simultaneously monitor our combined progress toward shared goals.

The final result: a unified household view that tracks legal ownership without sacrificing the "One Number" net worth total.

That’s the full system (see chapter-6 for the complete multi-person structure). But how do I actually use it week to week?

The weekly ritual

Now here’s how I keep it current: the “20 minutes a week” I mentioned at the start:

Getting statements: During the week, banks email me saying a statement is available. Some attach PDFs directly; others require a login. Either way, I snooze the emails (I use inbox zero) until the weekend. For my wife’s accounts, I nudge her once a month and she drops stuff in a shared Drive folder.
Running imports: On the weekend, I work through my snoozed emails. Note that I’m not updating every single account every week. I only download statements for the 3-4 accounts that saw activity; long-term investments often just get a monthly or quarterly check-in. I move each file to the correct data/ subfolder for that institution (e.g., data/hsbc-uk-current/). My importer pipeline automatically runs pdftotext to extract text so the parsers can read it.
Categorizing: I launch beancount-import, which opens a web UI in my browser:
```
python -m beancount_import.webserver --journal lalit/journal-net.beancount
```
As I mentioned before, most transactions auto-categorize and I’m just hitting Enter: I buy groceries from the same place, pay my hosting costs to the same provider etc. New merchants do need some manual work, but it’s a matter of typing a few characters and again pressing Enter.
Formatting and checking: I run bean-format (a Beancount utility that normalizes indentation and aligns amounts) to keep my transactions file tidy. This makes git diffs cleaner. Then I open Fava for a quick sanity check:
```
fava lalit/journal-net.beancount
```
Are transfer accounts zeroed out? Do expenses look legit? How are investments doing? If it all checks out, commit and push.

That’s it. I try to stick to doing this every week, but sometimes I’m on holiday or just have other commitments. In that case, it’s 40 minutes every two weeks. The system is forgiving; I’m never behind for too long.

I also have some automation helping me out: during the week, I have a CI workflow that runs daily, regenerates summaries, and commits them. Whenever I want, I can check the repo and see what the numbers look like. I particularly like this because I can easily see the before/after numbers in a single file, so I can spot check “does this make sense”. And of course at any time, I can also open Fava if I want to go a bit deeper.

Conclusion

That 2021 tax disaster feels like a lifetime ago. What started as “there must be a better way” became a system I actually trust. One number, always current, completely under my control.

The specifics will evolve as life changes and tools improve, but three principles have held for years now. I expect them to hold for decades:

Double-entry everywhere. Every transaction balances. Money never appears from nowhere or vanishes into nothing. When something doesn’t add up, you know immediately.

Plain text as the source of truth. Your financial history lives in files you can read, diff, grep, and version control. No vendor lock-in, no opaque databases, no trusting a third party with your data.

Track at the leaf. Record transactions at the most granular level that makes sense. You can always aggregate up (Income:Dividends from Income:IB:AAPL:Dividends), but you can never disaggregate down. Capture the detail; collapse it later with views.

Finance systems are deeply personal. This post isn’t meant to say this is the system everyone should use, just what’s worked for me over several years.

Each chapter here could be its own post, so if you want me to go deeper on imports, investments, or the multi-person setup, let me know!

It took me dozens of hours spread over months. This post and companion repo should get you there faster. ↩︎
By “normal” I mean any personal finance system which works on the basis of just plain transactions and “tagging” or “categorizing them”. This includes spreadsheets and apps like Mint, Quicken, or Copilot. ↩︎
Since 1494, when Luca Pacioli first codified the method in Summa de arithmetica. The practice existed among Italian merchants before that, but Pacioli was the first to publish it. ↩︎
Real accountants use “Debits” and “Credits” to keep numbers positive. Beancount just uses plus and minus. It’s way more intuitive for coders, even if it makes accountants twitch. ↩︎
Yes I know technically that Equity means ‘Ownership’. But I’m trying to keep things simple for an audience new to this stuff, and equity is confusing enough in this simplified model without the abstract ideas which come from business bookkeeping. ↩︎
As I say, I initially just had an “Equity:Transfers” account and I put 3 different “untracked accounts” in that same bucket. When I finally imported these, I had to go through and recategorize all these transfers manually! Not the most fun day I’ve had… ↩︎
Trust me, I tried for years to maintain Puppeteer scrapers for all my bank statements. It became a second job to keep those up to date and eventually I started dreading my finances because of it. The day I deleted all that code, I felt a real burden had been lifted. ↩︎
Plaid also has privacy tradeoffs. Banks with OAuth support let you authenticate directly, but many still require you to hand Plaid your credentials, which they store and use to access your data. ↩︎
The original beancount-import doesn’t support Beancount v3. I maintain a fork with v3 compatibility. ↩︎
The beangrow plugin (used for performance reporting) specifically requires a name metadata attribute on each commodity. For convenience, I typically set this to the symbol itself, but it can be any descriptive string like “Apple Inc.” or “Vanguard All-World ETF”. ↩︎
US readers will be familiar with tracking individual lots instead. The principle of needing a tax-specific view applies regardless of jurisdiction. ↩︎

Why I Ignore The Spotlight as a Staff Engineer

Thu, 04 Dec 2025 00:15:00 +0000

Lately I’ve been reading Sean Goedecke’s essays on being a Staff+ engineer. His work (particularly Software engineering under the spotlight and It’s Not Your Codebase) is razor-sharp and feels painfully familiar to anyone in Big Tech.

On paper, I fit the mold he describes: I’m a Senior Staff engineer at Google. Yet, reading his work left me with a lingering sense of unease. At first, I dismissed this as cynicism. After reflecting, however, I realized the problem wasn’t Sean’s writing but my reading.

Sean isn’t being bleak; he is accurately describing how to deal with a world where engineers are fungible assets and priorities shift quarterly. But my job looks nothing like that and I know deep down that if I tried to operate in that environment or in the way he described I’d burn out within months.

Instead I’ve followed an alternate path, one that optimizes for systems over spotlights, and stewardship over fungibility.

We Live in Different Worlds

The foundational reason for our diverging paths is that Sean and I operate in entirely different worlds with different laws governing them.

From Sean’s resume, my understanding is that he has primarily worked in product teams ¹ building for external customers. Business goals pivot quarterly, and success is measured by revenue or MAU. Optimizing for the “Spotlight” makes complete sense in this environment. Product development at big tech scale is a crowded room: VPs, PMs and UX designers all have strong opinions. To succeed, you have to be agile and ensure you are working specifically on what executives are currently looking at.

On the other hand, I’ve spent my entire career much more behind the scenes: in developer tools and infra teams.

My team’s customers are thousands of engineers in Android, Chrome, and throughout Google ². End users of Google products don’t even know we exist; our focus is on making sure developers have the tools to collect product and performance metrics and debug issues using detailed traces.

In this environment, our relationship with leadership is very different. We’re never the “hot project everyone wants,” so execs are not fighting to work with us. In fact, my team has historically struggled to hire PMs. The PM career ladder at Google incentivizes splashy external launches so we cannot provide good “promotion material” for them. Also, our feedback comes directly from engineers. Adding a PM in the middle causes a loss in translation, slowing down a tight, high-bandwidth feedback loop.

All of this together means our team operates “bottom-up”: instead of execs telling us “you should do X”, we figure out what we think will have the most impact to our customers and work on building those features and tools. Execs ensure that we’re actually solving these problems by considering our impact on more product facing teams.

Compounding Returns of Stewardship

In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts. But in Infrastructure and Developer Experience, context is the currency.

Treating engineers as fungible assets destroys context. You might gain fresh eyes, but you lose the implicit knowledge of how systems actually break. Stewardship, staying with a system long-term, unlocks compounding returns that are impossible to achieve on a short rotation.

The first is efficiency via pattern matching. When you stay in one domain for years, new requests are rarely truly “new.” I am not just debugging code; I am debugging the intersection of my tools and hundreds of diverse engineering teams. When a new team comes to me with a “unique” problem, I can often reach back in time: “We tried this approach in 2021 with the Camera team; here is exactly why it failed, and here is the architecture that actually works”.

But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons.

Take Bigtrace, a project I recently led; it was a solution that emerged solely because I stuck around long enough to see the shape of the problem:

Start of 2023 (Observation): I began noticing a pattern. Teams across Google were collecting terabytes or even petabytes of performance traces, but they were struggling to process them. Engineers were writing brittle, custom pipelines to parse data, often complaining about how slow and painful it was to iterate on their analysis.
Most of 2023 (Research): I didn’t jump to build a production system. Instead, I spent the best part of a year prototyping quietly in the background while working on other projects. I gathered feedback from these same engineers who had complained and because I had established long-term relationships, they gave me honest and introspective feedback. I learned what sort of UX, latency and throughput requirements they had and figured out how I could meet them.
End of 2023 to Start of 2024 (Execution): We built and launched Bigtrace, a distributed big data query engine for traces. Today, it processes over 2 billion traces a month and is a critical part of the daily workflow for 100+ engineers.

If I had followed the advice to “optimize for fungibility” (i.e. if I had switched teams in 2023 to chase a new project) Bigtrace would not exist.

Instead, I would have left during the research phase and my successor would have seen the same “noise” of engineers complaining. But without the historical context to recognize a missing puzzle piece, I think they would have struggled to build something like Bigtrace.

The Power of “No”

One of the most seductive arguments for chasing the “Spotlight” is that it guarantees resources and executive attention. But that attention is a double-edged sword.

High-visibility projects are often volatile. They come with shifting executive whims, political maneuvering, and often end up in situations where long-term quality is sacrificed for short-term survival. For some engineers, navigating this chaos is a thrill. For those of us who care about system stability, it feels like a trap.

The advantage of stewardship is that it generates a different kind of capital: trust. When you have spent years delivering reliable tools, you earn the political capital to say “No” to the spotlight when it threatens the product.

Recently, the spotlight has been on AI. Every team is under pressure to incorporate it. We have been asked repeatedly: “Why don’t you integrate LLMs into Perfetto?” If I were optimizing for visibility, the answer would be obvious: build an LLM wrapper, demo it to leadership, and claim we are “AI-first.” It would be an easy win for my career.

But as a steward of the system, I know that one of Perfetto’s core values is precision. When a kernel developer is debugging a race condition, they need exact timestamps, not a hallucination. Users trust that when we tell them “X is the problem” that it actually is the problem and they’re not going to go chasing their tail for the next week, debugging an issue which doesn’t exist.

But it’s important not to take this too far: skepticism shouldn’t become obstructionism. With AI, it’s not “no forever” but “not until it can be done right” ³.

A spotlight-seeking engineer might view this approach as a missed opportunity; I view it as protecting what makes our product great: user trust.

The Alternate Currency of Impact

The most common fear engineers have about leaving the “Spotlight” is career stagnation. The logic goes: If I’m not launching flashy features at Google I/O, and my work isn’t on my VP’s top 5 list, how will I ever get promoted to Staff+?

It is true that you lose the currency of “Executive Visibility.” But in infrastructure, you gain two alternate currencies that are just as valuable, and potentially more stable.

Shadow Hierarchy

In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers.

I call this the Shadow Hierarchy. You don’t need your VP to understand the intricacies of your code. You need the Staff+ Engineers in other critical organizations to need your tools.

When a Senior Staff Engineer in Pixel tells their VP, “We literally cannot debug the next Pixel phone without Perfetto”, that statement carries immense weight. It travels up their reporting chain, crosses over at the Director/VP level, and comes back down to your manager.

This kind of advocacy is powerful because it is technical, not political. It is hard to fake. When you are a steward of a critical system, your promotion packet is filled with testimonials from the most respected engineers in the company saying, “This person’s work enabled our success”.

Utility Ledger

While product teams might be poring over daily active users or revenue, we rely on metrics tracking engineering health:

Utility: Every bug fixed using our tools is an engineer finding us useful. It is the purest measure of utility.
Criticality: If the Pixel team uses Perfetto to debug a launch-blocking stutter, or Chrome uses it to fix a memory leak, our impact is implicitly tied to their success.
Ubiquity: Capturing a significant percentage of the engineering population proves you’ve created a technical “lingua franca”. This becomes especially obvious when you see disconnected parts of the company collaborating with each other, using shared Perfetto traces as a “reference everyone understands”.
Scale: Ingesting petabytes of data or processing billions of traces proves architectural resilience better than any design doc.

When you combine Criticality (VIP teams need this) with Utility (bugs are being fixed), you create a promotion case that is immune to executive reorganizations.

Archetypes and Agency

Staff Archetypes

I am far from the first to notice the idea of “there are multiple ways to be a staff software engineer”. In his book Staff Engineer, Will Larson categorizes Staff-plus engineers into four distinct archetypes.

Sean describes the Solver or the Right Hand: engineers who act as agents of executive will, dropping into fires and moving on once the problem is stabilized. I am describing the Architect or the Tech Lead: roles defined by long-term ownership of a specific domain and deep technical context.

The “Luck” Rebuttal

I can hear the criticism already: “You just got lucky finding your team. Most of us don’t have that luxury.”

There are two caveats to all my advice in this post. First, the strategy I have employed so far requires a company profitable enough to sustain long-term infrastructure. This path generally does not exist in startups or early growth companies; it is optimized for Big Tech.

Second, luck does play a role in landing on a good team. It is very hard to accurately evaluate team and company culture from the outside. But while finding the team might have involved luck, staying there for almost a decade was a choice.

And, at least in my experience, my team is not particularly special: I can name five other teams in Android alone ⁴. Sure, they might have a director change here or a VP change there, but the core mission and the engineering team remained stable.

The reason these teams seem rare is not that they don’t exist, but that they are often ignored. Because they don’t offer the rapid, visible “wins” of a product launch nor are they working on the “shiny cool features”, they attract less competition. If you are motivated by “shipping to billions of users” or seeing your friends and family use something you built, you won’t find that satisfaction here. That is the price of admission.

But if you want to build long-term systems and are willing to trade external validation for deep technical ownership, you just need to look behind the curtain.

Conclusion

The tech industry loves to tell you to move fast. But there is another path. It is a path where leverage comes from depth, patience, and the quiet satisfaction of building the foundation that others stand on.

You don’t have to chase the spotlight to have a meaningful, high-impact career at a big company. Sometimes, the most ambitious thing you can do is stay put, dig in, and build something that lasts. To sit with a problem space for years until you understand it well enough to build a Bigtrace.

By product team I don’t mean “frontend team”: even as a backend engineer, you are still working on some part of what is being served directly to end users. ↩︎
This is not exhaustive, Perfetto is open source and we do also care about external developers but that’s not why we get paid. From the company perspective, time we spent on open source bugs is “wasted” time but we do it because we believe in the mission of open source. I talked about this more in a recent post, On Perfetto, Open Source, and Company Priorities. ↩︎
For what it’s worth, LLMs might not even be the best solution to “let’s put AI into Perfetto”: in my opinion there is lots of value with “old school” machine learning techniques like neural networks. A lot of trace analysis is just pattern matching. This is something I’m hoping to explore more in the coming year! ↩︎
Android Kernel, Android System Health, Android Runtime, Android Camera HAL, Android Bionic ↩︎

We stopped roadmap work for a week and fixed 189 bugs

Sat, 22 Nov 2025 18:20:00 +0000

It’s Friday at 4pm. I’ve just closed my 12th bug of the week. My brain is completely fried. And I’m staring at the bug leaderboard, genuinely sad that Monday means going back to regular work. Which is weird because I love regular work. But fixit weeks have a special place in my heart.

What’s a fixit, you ask?

Once a quarter or so, my org with ~45 software engineers stops all regular work for a week. That means no roadmap work, no design work, no meetings or standups.

Instead, we fix the small things that have been annoying us and our users:

an error message that’s been unclear for two years
a weird glitch when the user scrolls and zooms at the same time
a test which runs slower than it should, slowing down CI for everyone

The rules are simple: 1) no bug should take over 2 days and 2) all work should focus on either small end-user bugs/features or developer productivity.

We also have a “points system” for bugs and a leaderboard showing how many points people have. And there’s a promise of t-shirts for various achievements: first bug fix, most points, most annoying bug, etc. It’s a simple structure, but it works surprisingly well.

What we achieved

Some stats from this fixit:

189 bugs fixed
40 people participated
4 was the median number of bugs closed per person
12 was maximum number of bugs closed by one person

Bug Burndown Chart for the Q4'25 Fixit

Here are some of the highlights (sadly many people in my org work in internal-facing things so I cannot share their work!):

I closed a feature request from 2021! It’s a classic fixit issue: a small improvement that never bubbled to the priority list. It took me one day to implement. One day for something that sat there for four years. And it’s going to provide a small but significant boost to every user’s experience of Perfetto.
My colleague made this small change to improve team productivity. Just ~25 lines of code in a GitHub Action to avoid every UI developer taking two extra clicks to open the CI’s build. The response from the team speaks for itself:

Such a simple change but the team loved it!

I also fixed this issue to provide a new “amalgamated” version of our SDK, allowing it to be easily integrated into projects. It’s one of those things that might be the difference between someone deciding to use us or not, but building it took just one hour of work (with liberal use of AI!).

The benefits of fixits

For the product: craftsmanship and care

I care deeply about any product I work on. That means asking big questions like “what should we build?” and “how do we make this fast?” But it also means asking smaller questions: “is this error message actually helpful?” or “would I be frustrated using this?”

A hallmark of any good product is attention to detail: a sense that someone has thought things through, and the pieces fit together to make a cohesive whole. And the opposite is true: a product with rough edges might be tolerated if there are no alternatives, but there will always be a sense of frustration and “I wish I could use something else”.

Fixits are a great chance to work on exactly those details that separate good products from great ones. The small things your average user might not consciously notice, but absolutely will notice if they’re wrong.

For the individual: doing, not thinking

I sometimes miss the feeling I had earlier in my career when I got to just fix things. See something broken, fix it, ship it the same day.

The more senior you get in a big company, the less you do that. Most of your time becomes thinking about what to build next, planning quarters ahead, navigating tradeoffs and getting alignment.

Fixits give me that early-career feeling back. You see the bug, you fix it, you ship it, you close it, you move on. There’s something deeply satisfying about work where the question isn’t “what should we do?” but rather “can I make this better?” And you get to answer that question multiple times in a week.

For the team: morale and spirit

People sharing live updates in the Fixit chatroom

Having 40 people across two time zones all fixing bugs together adds a whole other dimension.

The vibe of the office is different: normally we’re all heads-down on different projects, but during fixit the team spirit comes out strong. People share their bug fixes in chat rooms, post before-and-after screenshots and gather around monitors to demo a new feature or complain about a particularly nasty bug they’re wrestling.

The daily update from Friday

The leaderboard amplifies this energy. There’s a friendly sense of competition as people try and balance quick wins with meatier bugs they can share stories about.

There’s also a short update every morning about how the previous day went:

total bugs fixed
how many people have fixed at least one bug
how many different products we’ve fixed things in
who’s currently at the top of the leaderboard

All of this creates real momentum, and people feel magnetically pulled into the effort.

How to run a fixit

I’ve participated in 6 fixits over the years and I’ve learned a lot about what makes them successful. Here are a few things that matter more than you’d think.

Preparation is key

Most of what makes a fixit work happens before the week even starts.

All year round, we encourage everyone to tag bugs as “good fixit candidates” as they encounter them. Then the week before fixit, each subteam goes through these bugs and sizes them:

small (less than half a day)
medium (less than a day)
large (less than 2 days).

They assign points accordingly: 1, 2, or 4.

We also create a shortlist of high-priority bugs we really want fixed. People start there and move to the full list once those are done. This pre-work is critical: it prevents wasting day one with people aimlessly searching for bugs to fix.

The 2-day hard limit

In one of our early fixits, someone picked up what looked like a straightforward bug. It should have been a few hours, maybe half a day. But it turned into a rabbit hole. Dependencies on other systems, unexpected edge cases, code that hadn’t been touched in years.

They spent the entire fixit week on it. And then the entire week after fixit trying to finish it. What started as a bug fix turned into a mini project. The work was valuable! But they missed the whole point of a fixit. No closing bugs throughout the week. No momentum. No dopamine hits from shipping fixes. Just one long slog.

That’s why we have the 2-day hard limit now. If something is ballooning, cut your losses. File a proper bug, move it to the backlog, pick something else. The limit isn’t about the work being worthless - it’s about keeping fixit feeling like fixit.

Number of people matters

We didn’t always do fixits with 40 people. Early on, this wasn’t an org-wide effort, just my subteam of 7 people. It worked okay: bugs got fixed and there was a sense of pride in making the product better. But it felt a bit hollow: in the bigger picture of our org, it didn’t feel like anyone else noticed or cared.

At ~40 people, it feels like a critical mass that changes things significantly. The magic number is probably somewhere between 7 and 40. And it probably varies based on the team. But whatever the number is, the collective energy matters. If you’re trying this with 5 people, it might still be worth doing, but it probably won’t feel the same.

Gamification

The points and leaderboard are more than a gimmick, but they have to be handled carefully.

What works for us:

Points are coarse, not precise: We deliberately use 1/2/4 points instead of trying to measure exact effort; the goal is “roughly right and fun”, not accurate performance evaluation.
Celebrate breadth, not just volume. We give t-shirts for things like “first bug fix”, “most annoying bug fixed”, not just “most points”. That keeps newer or less experienced engineers engaged.
Visibility over prizes. A shout-out in the daily update or an internal post often matters more than the actual t-shirt.
No attachment to perf reviews. This is important: fixit scores do not feed into performance reviews. The moment they do, people will start gaming it and the good vibe will die.

We’ve had very little “gaming” in practice. Social norms do a good job of keeping people honest and 40 is still small enough that there’s a sense of “loyalty to the cause” from folks.

The AI factor

The big challenge with fixits is context switching. Constantly changing what you’re working on means constantly figuring out new parts of the codebase, thinking about new problems.

AI tools have mitigated this in a big way. The code they write is less important than their ability to quickly search through relevant files and summarize what needs to change. They might be right or wrong, but having that starting point really reduces the cognitive load. And sometimes (rarely) they one-shot a fix.

This docs change was a perfect example of the above: an update to our docs which catches out new contributors and AI was able to one-shot the fix.

On the other hand, in my record page change it was more useful for giving me prototypes of what the code should look like and I had to put in significant effort to correct the bad UX it generated and its tendency to “over-generate” code. Even so, it got me to the starting line much faster.

Criticisms of fixits (and why I still like them anyway)

I’ve definitely come across people who question whether fixits are actually a good idea. Some of the criticisms are fair but overall I still think it’s worth it.

“Isn’t this just admitting you ignore bugs the rest of the time?”

To some extent, yes, this is an admission of the fact that “papercut” bugs are underweighted in importance, both by managers and engineers. It’s all too easy to tunnel on making sure a big project is successful and easier to ignore the small annoyances for users and the team.

Fixits are a way of counterbalancing that somewhat and saying “actually those bugs matter too”. That’s not to say we don’t fix important bugs during regular work; we absolutely do. But fixits recognize that there should be a place for handling the “this is slightly annoying but never quite urgent enough” class of problems.

The whole reason we started fixits in the first place is that we observed these bugs never get actioned. Given this, I think carving out some explicit time for it is a good thing.

“Isn’t it a waste to pause roadmap work for a whole week?”

It’s definitely a tradeoff. 40 engineer-weeks is a lot of manpower and there’s an argument to be made it should be used for actually solving roadmap problems.

But I think this underweights the importance of polish of products to users. We’ve consistently found that the product feels noticeably better afterward (including positive comments from users about things they notice!) and there’s a sense of pride in having a well-functioning product.

A user’s response to my PR to improve the Perfetto UI “record” page

Also, many of the team productivity fixes compound (faster tests, clearer errors, smoother workflows) so the benefits carry forward well beyond the week itself.

This only works at big companies!

I agree that a full week might be too much for tiny teams or startups. But you can still borrow the idea in smaller chunks: a “fixit Friday” once a month, or a 2-day mini-fixit each quarter. The core idea is the same: protected, collective time to fix the stuff people complain about but no one schedules time to address.

Fixits are good for the soul

The official justification for fixits is that they improve product quality and developer productivity. And of course they do this.

But the unofficial reason I love them is simpler: it just feels good to fix things. It takes me back to a simpler time, and putting thought and attention into building great products is a big part of my ethos for how software engineering should be done. I wouldn’t want to work like that all the time. But I also wouldn’t want to work somewhere that never makes time for it.

When Good Technical Writing Isn't Enough

Wed, 12 Nov 2025 20:14:00 +0000

Last week, I came across Don’t Build an Audience. It’s a fascinating post and has been occupying a lot of my “free thinking” time. I strongly suggest reading it as it’s well written and excellently argued.

To summarise what it tries to say:

The market for written content on the web (blog posts, articles etc) is “efficient”: in other words, great work will find its audience without you needing to build a “following” first.
“Liquidity providers” (influencers, curators, algorithms) are incentivized to share good content because it benefits them. They maintain credibility and engagement by surfacing quality work. Specifically called out are Substack, Alexey Guzey and Tyler Cowen.
You can speed up “distribution” of what you write with minimal effort. Just 30 minutes to email a few key people and posts on the “right” platforms.
Therefore, focus on creating excellent work rather than “building an audience”. If it’s truly good, the market will ensure it reaches “everyone who matters”.

I’m a very strong believer in the core message of “make cool things rather than chase followers”. It’s something I’ve done throughout my life and I have zero interest in changing that going forward.

But I don’t agree with the argument that the market is efficient for all types of written content. I think the author is over-indexing on what they write about: topics accessible for a wide audience and do not require background knowledge (e.g. philosophical musings, economic theory, social dynamics).

I write about technical topics. Sometimes this can be so deeply in the weeds that at most a few thousand people in the world would care about it. But it might totally change how these people work: it might teach about how to debug performance in their code, let them know about a new approach to analysing data or even inspires them build an entirely new program.

For technical content, the foundations of the “efficient market” as proposed break down:

There are very few influencers or curators for technical niches. There’s no “Tyler Cowen of performance” or “Alexey Guzey of databases”. The people who would most benefit from these posts are engineers who are heads-down solving problems. They’re probably not sharing content for an audience. There are exceptions (e.g. Simon Willison) but, while technical, they still skew to the “accessible” side. They’re also few and far between.
Substack is not good for “niche” content. Unlike TikTok or YouTube which are incredibly good at matching niche interests to the right people, Substack’s discovery still feels like it significantly prioritises content with wide appeal.
In-depth technical writing is “high-attention”. You can accidentally watch a 10 TikTok videos about woodworking while procrastinating. You can read a blog post about how prediction markets work on a daily commute. You probably wouldn’t accidentally read 5,000 words about database query optimization if you were not already interested.

The obvious rebuttal though is that, while Substack or curators may not work for technical writing, there are alternative “liquidity providers”, namely a) link-aggregators like Hacker News and Lobste.rs and b) technical subreddits and forums c) roundup newsletters like TLDR and Programming Digest.

However, there is a very powerful effect working against you in all of these: if you post lots of content on the same topic, you are punished by all of them. Specifically, there is always a feeling from readers of “didn’t I already see a post on this last week?”. And this comes from the value both the readers and platforms place on novelty. For example, Perfetto: Swiss Army Knife for Linux Client Tracing reached the front page of Hacker News but I think it’s very unlikely that another post mentioning Perfetto will do so within next several months.

This is very weird because, as an expert in an area, you want to write more about what you know about. But all of these technical “liquidity providers” want you to do the opposite: they incentivise you to “broaden out” and write about different topics constantly. Or at the very least, they prefer you “space out” posts so you’re not writing too much on the same topic in a compressed timeframe.

To me, this means you have to be slightly more strategic when thinking about how to get your technical writing out there:

Do build an audience: just not in the the way the original post means. It assumes “audience building” means chasing followers and optimizing for engagement. But for technical writers, it’s instead connecting with a small group who value your specific expertise. When I write my n-th post about Perfetto, I’m writing for the people who are are interested enough in performance tooling to care. They follow precisely because I keep going deep on the similar topics.
Rotate through different platforms when promoting articles. Hacker News for some articles, lobste.rs for others, LinkedIn for yet others. The choice depends on topic (different platforms prefer different “types” of content) but also time since you last promoted something similar on that platform.
Accept that this is a long game. The market might not be efficient in the short term, but over years, good articles will naturally bubble up and become “de-facto references” for people. So instead of virality, measure in terms of “mind-share” over the course of years.
- Of course, this is much easier said than done because you have to write “in the dark” hoping that what you write will have value in the long term without getting any immediate feedback!

So yes, build cool things. That remains the most important guiding principle as without that, there’s little point to the type of technical blog posts I like to write. But I think a little bit of strategic investment can go a long way to increase the chance of the right people seeing them.

Perfetto: Swiss Army Knife for Linux Client Tracing

Fri, 31 Oct 2025 00:00:00 +0000

I gave a talk at the 2025 Tracing Summit last month titled “Perfetto: The Swiss Army Knife of Linux Client/Embedded Tracing”. My goal in this talk was to show how Linux kernel, systems and embedded developers can use Perfetto when debugging and root-causing performance issues in their respective domains. Even though the Perfetto UI is primarily built for viewing Android or Chrome traces, it is a flexible tool and can be used in many other ways!

The talk was recorded and is available on YouTube. Taking inspiration from Simon Willison, this post is an annotated presentation containing my slides and detailed notes on them. The talk also has a lot of UI demos: for these, I’ll have a screenshot but also a link to the relevant part of the video (videos are unbeatable for UI!).

First, what is Perfetto? Perfetto is fundamentally a suite of tools: it’s not just one thing but a bunch of different tools working together to help you debug and root-cause problems. This diagram shows how everything fits together, with the core parts of the Perfetto project in the center.

The recording tools for Perfetto consist of 1) an SDK for C++ apps 2) a daemon that can collect data from ftrace, /proc, /sys, and various kernel interfaces 3) another daemon that amalgamates trace data from multiple processes into a single trace file. These tools all speak the Perfetto protobuf format, a high-performance trace format designed to be very efficient to write but not to analyze or consume directly.

That’s where the trace processor comes in. It’s a C++ library that parses the protobuf format, but also a bunch of other widely used trace formats. It exposes this data via an SQL query interface to any embedding program including Perfetto UI (which is what most of this talk is about) and also to the Python API if you want to do ad-hoc scripting or analysis in Python.

There are also very common tracing/profiling formats used by the Linux community: perf.data, ftrace text format, Firefox profiler format, and many others. Perfetto supports quite a few of those directly. There’s also the Chrome JSON format (AKA the Trace Event Format) which is a simpler interchange format. It’s not the most efficient to read or write, but it does the job for a lot of use cases.

Often people write converters. They have their own tracing format, maybe proprietary, maybe open source but something we don’t know about, and it’s very common that people convert to one of the formats we understand, most commonly our protobuf or Chrome JSON formats.

The Perfetto UI is fundamentally a web-based trace visualizer, combining timeline visualization, user-driven selection/aggregation, and SQL queries all in one interface. Because it has the trace processor as a backend, it works with a bunch of different trace formats.

It’s very important to note that even though the Perfetto UI is web-based, everything happens inside your browser and trace data never leaves your system. You can even build it and host it yourself on any static server: we’ve made it extremely easy to do so!

At the start of 2025, we actually moved our whole development to GitHub. In the past, we used to develop on Android and GitHub was just a mirror. That’s no longer the case, GitHub is actually where we develop and take pull requests.

Most of this talk, I’m going to spend actually showing you how you can use the Perfetto UI to debug performance issues on Linux. I don’t want to show you an Android trace which needs a lot of context about how the Android system works and so you think, “oh, that was cool, but I didn’t really understand what was happening.”

So to make this talk more approachable, I wrote a straightforward demo program you can look at yourself! So it’s obviously not a production system but I’ve tried to make it as representative of the sort of issues we use Perfetto for every day.

It’s a Rust program which generates a Julia set and visualizes it over time. The technologies I used: Vulkan, GPU rendering and also multi-threaded CPU computation. So how it works is that computation of various parameters is happening on background threads, and then that’s being passed to the main thread for rendering.

And then, for demonstration purposes, there is a performance bug; rendering should run at 60 FPS, but every so often, the frame rate drops dramatically. Here’s what that looks like:

The code is on GitHub and if you’re interested in following along. The traces are there as well - you don’t have to collect the traces yourself, but you can if you want. All the instructions and information is in the README.

So the first suspicion we may have is that maybe it’s some CPU problem. A lot of engineers I know would reach for perf immediately whenever they see a problem like this. The main reason is that if perf can capture the problem, they can go straight to the line of code without needing to spend time debugging using more complex approaches.

You can record a perf profile with perf record -k mono -g ./fractal_renderer. The standard post-processing step which I think a lot of people do would be to generate an SVG flame graph out of this with perf script | stack-collapse-perf.pl --all | flamegraph.pl > fractal-frame.svg. Here’s how that looks like for our perf profile:

See also the interactive version of the profile

The flame graph shows the thread names at the bottom, and then stacked above are the call stacks of what the program is doing. The width represents how much time is spent in each function. In this case, the worker threads are spending most of their time in cos and sin operations, doing math computation. Exactly what you’d expect for this type of program. Aside: Interestingly demangling didn’t seem to work for Rust out of the box which I find a bit unusual but I didn’t dig too much into why this was.

Looking at this, nothing really stands out. There’s no smoking gun that immediately reveals the problem. But here’s the fundamental limitation: the flame graph aggregates data across the entire trace, losing the time dimension. The performance problem we’re seeing happens every 2-3 seconds - brief drops in an otherwise normally functioning program.

The flame graph shows me the aggregate, dominated by the correct behavior, making it nearly impossible to spot those occasional problematic moments. So how do I find the places where it’s doing the wrong thing?

Well, that’s maybe where Perfetto can help you out a little bit! The thing I find lots of people don’t know is that perf actually preserves timestamp information about when samples were taken: many tools drop this information but Perfetto is pretty good at showing that to you. I just need post-process the trace with perf script > fractal.perftext to generate a text version of the profile which we can then open in the Perfetto UI

My demo talking through how to open the perf profile and navigate it starts at 9:11 in the video.

Screenshot from 9:50 in the video

The x-axis of what you’re seeing is time. And every horizontal line represents a thread: we call these lanes “tracks”. And each of the small arrowheads on the tracks are CPU samples for the associated thread.

The behavior of the worker threads over time is really interesting. Most of the lifetime of the program, it’s doing continuous computation, basically. And then there’s this period of time in the middle where there’s this very interesting staircase pattern where it seems like only one thread is running at any one time: we’ll want to keep a note of this as this is quite important.

Screenshot from 10:37 in the video

One very cool thing about Perfetto is that it allows me to quickly generate visualizations of whatever I’m selecting on the timeline. We call this “area selection” and it’s where I drag my mouse and select a region both horizontally and vertically. This is on one track but even across multiple tracks. So in the timeline it shows me the selection I’m making at the top, plus a flame graph at the bottom representing the aggregation of the samples in just that time period.

The key advantage is that I can look at individual regions of time interactively. You can also do this with flamegraph.pl but you need to pre-filter the data to what you’re looking for. This assumes you already know what the problem is before you find it. I always find that a bit counterintuitive. I prefer this view where I can see everything first, then decide “I want to look at this specific part in more detail.” That’s what Perfetto lets me do.

Screenshot from 12:31 in the video

So now on to the flame graph itself: in the last year, I spent a bunch of time on improving the flame graph visualization in Perfetto UI. At Google, we have an internal tool called pprof (related to but not the same as the open-source one), and I’ve always loved the visualization it has. So I worked to make Perfetto’s flame graph look and behave very similarly.

There are a bunch of features here. You can zoom in like most flame graph, but you can also say “I only want to look at the stack starting from this point” — it gets rid of everything above and starts fresh from there. If I don’t want to see a particular function like sin, I can just hide the frame and it gets merged into its parent. The search is regex-based so I can also just type the specific things I’m interested in: it’s pretty powerful.

There’s also something I don’t think I’ve seen in other visualizers (I might be wrong, please do correct me!): what I call “bottom up” view. Imagine you take all the leaves across your entire program and visualize who’s calling into them. If you have a leaf function that’s called everywhere in your program but reached from many different places, it’ll be split across your flame chart. This is the inverse — you’re looking at the leaves and asking “who is calling me?” This is particularly useful when you’re trying to optimize very low-level functions that are always at the leaf level — things like memcpy or hashmap operations.

So our main takeaway is that our worker threads all had gaps in their CPU execution in a staircase pattern. This means the threads weren’t actually on the CPU and that’s usually a sign that they’re sleeping.

But sleeping on what? Locks? Disk? Network? What exactly is happening? To answer that, I need a scheduler trace to show me the wake-up patterns, who’s being scheduled when, what type of sleep threads are in, that sort of thing.

We recently added support for trace-cmd’s text format in Perfetto (aside: there’s an open issue to support the binary format too!). For this demo, I’m collecting sched_switch and sched_waking events with sudo trace-cmd record -e sched:sched_switch -e sched:sched_waking -C mono ./fractal-renderer-vk and we can open it in the UI once we do trace-cmd report -N > fractal.sched.

The demo showing off scheduler traces starts at 15:16 in the video.

Screenshot from 15:49 in the video

So this is what a scheduler trace looks like. Again as before, the x-axis is time. At the top of the trace, you’ve got the CPU scheduling tracks. For each CPU, they tell us what was scheduled on that CPU at each point in time. The white regions are where there’s nothing scheduled. As I zoom in, you start to see information like the name of the thread that’s scheduled, the TID of the thread, and so on. This is the CPU view of things, looking at it from a CPU perspective.

Screenshot from 16:48 in the video

There’s also a thread perspective on scheduling: what we call this “thread state”. This shows the scheduling state of each thread at every point in time. Dark green represents the thread running, light green represents runnable (the thread is on the runqueue so it’s eligible to run, but the scheduler hasn’t actually let it run yet), and white represents interruptible sleep (S sleep). There are also orange regions for uninterruptible sleep (D sleep), though that doesn’t show up much in this trace.

You can again see a very clear staircase-like pattern. Again, only one thread seems to be running at any one time. You’re maybe getting the sense of what the problem might be at this point.

Screenshot from 17:43 in the video

Like with the perf visualization, you can do area selections here too. I can select a region on a CPU and get tables showing time spent by each thread (shown in the screenshot above). I can also do this for thread state, you can also see that runnable time is basically negligible compared to running and sleeping, which is where most of the time is being spent.

Screenshot from 18:12 in the video

There are also scheduler-specific visualizations like run queue length and active CPU count. You can see in the problematic region, the active CPU count also went down, which makes sense.

So the scheduler trace shows when threads are sleeping, but it’s not telling us why. What was the program doing in that region? What code was it running? The sequential pattern suggests some sort of serialization—something is causing only one thread to run at a time. But we need application-level visibility.

Since this program was written in Rust, I just used the off-the-shelf tracing and tracing-perfetto crates; I did have to make some small modifications to tracing-perfetto that I’m hoping to upstream. These output a Perfetto trace without needing to use the Perfetto SDK. These libraries are also maintained by others: we don’t have any hand in them.

All I needed to do was integrate these into my program and then add a command line switch to write out the collected trace to a file. So I just do ./fractal_renderer --trace fractal.pftrace

My demo for app tracing starts at 19:37 in the video.

Screenshot from 20:06 in the video

Looking at the app trace: as before x-axis is time and each of the tracks represents a thread. Instead of scheduling activity or CPU stack samples, this time the tracks is userspace instrumentation of the program itself. Each of these rectangles is a named period of time representing what the program was doing at that time.

There are lots of different names for these in the tracing world (e.g. spans, intervals) but we call these “slices”. The main thread is rendering frames as you would expect and the workers are each computing tiles of that frame, which eventually feed back to the main thread and sync to the GPU.

Screenshot from 23:42 in the video

There’s also this thing called flows, which shows the causal links between different pieces. Render frame is calling into all these places and causing them to happen. It’s sort of similar to wakeup graphs but for user space: basically, this is what caused me to start running this function.

Screenshot from 23:51 in the video

So you can see very clearly there’s a normal region and then a region where things are taking 1.8 seconds: almost certainly the cause of the frame drops.

And if we look at the slice in question, it seems to be doing something called “update adaptive quality.” Basically, I wrote some code to dynamically change the rendering quality based on frame rate. If I’m running faster, I can render at higher quality. If I’m running slower, I can do lower quality. That’s what this adaptive quality thing is supposed to do.

But clearly something has gone wrong. I’m causing frame drops because of updates to adaptive quality. A little bit ironic, to say the least. Now we know what the program is actually doing during that time span.

So now we’ve looked at three different sources of debugging data separately. In perf, we got told there are few or no CPU samples and weird staircase patterns. In ftrace, we saw only one worker seems to be active and the rest are sleeping in interruptible sleep. And in app tracing, we see it’s because of adaptive quality updates that workers are working on, and they shouldn’t be doing it this way.

Obviously, the theory is that it’s all the same problem. But we can confirm this theory with a very new (and still experimental) feature of the UI: trace merging.

Note: as this part of Perfetto is still experimental, if you want to try it yourself, you need to go to plugins and enable the “MultiTraceOpen” plugin. Also, unlike I what say in the talk, you do not need to be on Autopush as all the features I showed in my talk are now available in Stable.

The demo showing off merged traces starts at 24:32 in the video.

Screenshot from 25:44 in the video

This is now the merged trace file containing all the different information. You can see the CPU scheduling tracks like we had in the scheduler trace, the stack samples like we had in the perf trace, and the app instrumentation, all beside each other on one timeline. You can very clearly see the adaptive quality update running, then this period where it ran, and then it was done. It’s very cool to be able to see the pattern of how my program is sleeping and where and what it was running on one timeline.

The thing is, you’ve always been able to do this with Perfetto if you collect a Perfetto trace with all of these integrated. The new capability is that this is now being brought to traces you didn’t collect with Perfetto. Collect it with whatever you like, and we’ll still manage to do this visualization. That’s the new capability we have.

Screenshot from 29:58 in the video

There’s also a query interface which we’ve been building out recently and there are some pretty powerful things you can do with these tables. As well as the flat tables of data, there’s also a more dynamic pivot table and if I click the arrow on the left, I can get into a detailed filter table, similar to a spreadsheet but optimize for things people want to do on traces.

For example, by doing an area select, I can first get a list of all the events that happened during a time region. Then using the table I can filter for things - say I only care about slices longer than some duration, like 319 microseconds for whatever reason. I can click and add a filter for things greater than this.

Screenshot from 30:11 in the video

There’s also a feature called “Show Debug Track” that’s also very powerful. The table shows rows with timestamped duration information, and this feature lets you visualize that data as a track on the timeline. It adds a track at the top showing the regions of time where these events happened—in this case, where ComputeTile took longer than a certain threshold.

This is particularly useful for correlation analysis. For example, in Android, we’re often looking at system A and trying to understand its effect on system B. We find all the places where one thing is slow, then look for correlations. Being able to see a track in the UI where you can quickly have context and say “oh, during this period of time, this other thing was happening” is invaluable. It’s probably one of our most-used features.

You can also copy the SQL query from these tables and start doing your own aggregations on top of it. This eases the burden of starting with a completely blank canvas. Instead of wondering “where do I even start querying my data? What tables is my data in?”, the UI gives you a starting query that you can build on without needing to write something from scratch.

So the dynamic quality updates were stopping the world. Perf profiling showed the problem, scheduler traces found the sleeping pattern, app tracing confirmed it was the quality adjustment code, and the single timeline view let me see everything happening at once.

Interestingly, I tried to fix this by removing the lock contention—which I thought was the bug I had introduced. But it turns out I shouldn’t have been doing this code in the workers in the first place. Even after removing the lock contention, just the CPU activity of doing that work was enough to cause frame drops. The right solution was to move it to a background thread. As part of debugging this for the demo, I discovered something even better that I could be doing.

Android and Chrome are our bread and butter—that’s what we officially support as a team, and why Google staffs us. But there are many other interesting uses.
Mesa uses Perfetto as one of its tracing systems. One thing I could have shown is collecting a Mesa trace alongside all the other traces we looked at—you can actually see what the GPU is doing at the same time, which would have been very cool, but I just didn’t have time.
VizTracer is function tracing for Python, similar to uftrace, but you don’t have to recompile anything or do anything special.
pthread_trace is for visualizing pthread mutex contention. The author has a very efficient way of doing this and writes protobuf using heavy use of constexpr to make it very low overhead. It’s a very interesting project.
magic-trace uses Intel Processor Trace for function-level tracing at the processor level with lower overhead. They wrote a converter from that to the Perfetto format so you can visualize that data in Perfetto.
Qais Yousef’s sched-analyzer enriches Perfetto traces with scheduler internals. It’s a very cool project that I find particularly interesting.
John Stultz’s all-in-one Perfetto recording script for kernel developers addresses a common complaint: “Perfetto is so complicated, I don’t know what events I want to record.” He just wrote a script that configures everything for you.
Finally, Josef Bacik’s systing is a bit experimental but fascinating. He re-implemented tracing daemons on top of BPF instead of Perfetto’s native implementation, combining BPF-based tracing with perf stack traces in a single binary. He has a blog post explaining why. I thought it was a fascinating use of Perfetto.

If you want to try this yourself, you can use the demo program I’ve provided. But even better: record traces on your own programs. Open scheduler traces and perf at the same time. Instrument your program, convert it to Perfetto format, and visualize everything together.

If you want to convert ad-hoc timestamped data to Perfetto, we wrote a comprehensive tutorial with Python snippets for everything you might want to visualize. It covers all the features I showed and how to write Python code to generate them. We have a library for writing these traces: besides this one library, you don’t need to install anything else. You can go ahead and convert your own data to Perfetto.

We’re very happy to accept contributions and review pull requests pretty quickly. We’ve had a lot of open-source contributors over the years and have been quite lucky with some very high-quality contributions. If you want to contribute yourself or have a feature you feel is missing, send it our way.

Demo repo: github.com/LalitMaganti/fractal-renderer-vk
Learn more: docs.perfetto.dev
Perfetto UI: ui.perfetto.dev
Contact: lalitm@google.com

On Perfetto, Open Source, and Company Priorities

Fri, 17 Oct 2025 15:00:00 +0100

I recently stumbled across this post on lobste.rs about a project called traceboot which allows visualizing the Linux boot process using lightweight ftrace events and Perfetto. The author had some commentary about their experience trying to order tracks in Perfetto:

Ordering tracks with perfetto has been ridiculously complicated. It has taken the majority of the time of this project! Upstream’s answers are basically that the main user is Android (Perfetto is a Google project) so others come second if at all. While I get the reasons to do so, I read that as a caution against depending on it as a third-party. Google is notorious for (…) completely killing projects

Honestly? All of these points are right:

It is really unfortunate that doing something so simple took so much effort.
It’s true that external users are supported at a lower priority than Android users.
It’s also the case that Google has historically wound down projects when priorities shift

The good news is that we just landed support for trace writers to specify explicitly how traces should be ordered with the JSON format in Perfetto without any extreme workarounds! This feature is already available on the “Canary” UI channel and on “Stable” within 3-4 weeks.

process_sort_index and thread_sort_index are now respected by Perfetto!

So we solved the imediate problem! But here’s what concerns me: I found this issue totally by chance while browsing lobste.rs. There were several issues filed in the Perfetto repo over several years asking for this feature (#555, #764, #378) but none of them managed to communicate to us just how painful this issue was and the lengths people were willing to go to work around it. Seeing the annoyance in such stark terms definitely put into perspective the importance of this feature.

You might be thinking: “surely if a feature is requested that much, you should consider it important and work on fixing it”. And for a general open source project, I’d agree with you. But there are some unique challenges which come with being an open source project under the umbrella of a Big Tech company.

It’s all about how the Perfetto team fits into Google. We’re part of the Android Diagnostics organization whose mandate is to build tools and systems to root-cause problems on Android. Performance debugging would be impossible without a system-wide tracing tool like Perfetto (which is why we even exist at all!). But because we are funded by Android, it’s also expected that we prioritise its needs.

However, I, and others in the team, also deeply believe in the mission of open source. We want to build something useful not just for Android engineers inside Google but for the broader community as a whole. That creates a tension we struggle with constantly: we value external users and their use cases, but the reality of our team’s existence means Google’s priorities often win out.

This tension is compounded by the fact that different types of users want fundamentally different things from Perfetto. If you use Perfetto purely as a trace viewer, the expectation is “I write the trace file and the Perfetto UI should display exactly what I want”. But Android OS and app developers expect “Perfetto should intelligently figure out what is most important in my trace and highlight that to me.”

In the early days of Perfetto, we took on a lot of technical debt to meet the immediate needs of Android developers. We built an ‘intelligence’ layer that automatically reordered tracks based on Android-specific heuristics. This was great for our primary users, but it made it immensely difficult to later support features for the ‘pure trace viewer’ use case, like manually ordering tracks via the JSON file, because the Android logic would always override it.

The good news is that over the past 12 months, we’ve spent a lot of time trying to decouple the “core” layer of Perfetto UI from the “intelligence” layer for Android. This separation means we can now preserve the original intent from the trace format (like “this track should be ordered here”) all the way to the UI, rather than having it get lost or overridden by Android-specific heuristics along the way. This allows us to build features like JSON track ordering without creating impossible-to-maintain code that hurts us in the long run.

It’s also understandable that people worry about Google’s project longevity. Thankfully Perfetto’s core design principles of a) being 100% open source b) fully client side significantly mitigate these risks. For example, running the Perfetto UI is as simple as cloning the repo, running tools/install-build-deps --ui && ui/run-dev-server and navigating to localhost:10000. The site is entirely static capable of being hosted anywhere. There is no “backend server” to be taken down, everything runs in your browser only. So if for some reason, if the Perfetto team was to disappear, it would be trivially easy to host it yourself or even fork the project completely and start maintaining it.

With all that said, here are my takeaways from this experience:

There are still painpoints lurking in issues from the past that external users are still hitting today. We should maybe do a pass over historical bugs and figure out what issues are hiding there.
Moving forward, we need to be more transparent and communicative about why we’re not prioritising a certain issue. I feel our terse replies about our priorities are giving misleading indications on the value we place on external users. Maybe we should also be more open about what we are prioritizing, even if that means we have to say “we’re not going to be able to do much for external users this quarter”.
We also need to set up a regular routine for going through our bug tracker and checking “has anything changed which now makes this bug possible” and if so, we should follow up and post on the issue to make everyone aware of this and keep bugs up to date.

Here’s also where community help would be invaluable: if you’re facing an issue with Perfetto, tell us on Github or on our mailing list! Comment on old bugs if one already exists or file new ones if they don’t. Bonus points if you can tell us why the issue is so painful for you and what you’re doing to work around it if it’s not fixed. The more information we have, the better judgement calls we can make on prioritization.

V4L2 and Hardware Encoding on the Raspberry Pi

Mon, 01 Feb 2021 23:00:00 +0000

TLDR: Explain how the V4L2 M2M API works through the use-case of implementing hardware video encoding on the Raspberry Pi. This knowledge is generally useful as V4L2 is the de-facto generic API for hardware decoding and encoding on Linux.

Background

My journey started at this video on the excellent Craft Computing YouTube channel which showed how to setup TinyPilot, a Python app for KVM over IP which runs on a Raspberry Pi. Behind the scenes, TinyPilot uses ustreamer to read frames from a HDMI capture card and either exposes it over HTTP or writes it to shared memory. Along with the MJPEG output, support was recently added for encoding video using H264.

Even after messing with the source code, I could not get the H264 encoding working on my Pi running 64-bit Ubuntu with an error message of Can't create MMAL wrapper. Digging further, I ran into some insurmountable roadblocks with the approach taken by ustreamer and discovered the complex state of hardware encoding on the Pi.

In short, there are three userspace APIs for accessing the HW encoding and decoding on the Pi’s Broadcom chip:

OpenMAX Integration Layer (OpenMAX IL)

This API came directly from Broadcom and was the original way to access hardware codecs.
Support was dropped by Broadcom a long time ago and there appears to be no hope of it ever working on 64-bit OSes.
ustreamer uses this API to perform MJPEG encoding when specifically requested by the user.

Multimedia Abstraction Layer (MMAL)

This API was developed by the Pi developers as a replacement for OpenMAX and backs tools like raspistill and raspivid.
While support for 64-bit was added at one point, it was subsequently reverted and, currently, does not work.
ustreamer uses this API to perform H264 encoding to shared memory.

Video4Linux2 Memory to Memory (V4L2 M2M)

V4L2 is Linux API which was historically used for video capture and output. In 2010, the M2M API was added, significantly extending V4L2 and adding support for video codecs.
The Raspberry Pi kernel has a driver which uses MMAL APIs in kernelspace and integrates with the rest of V4L2.
Since this is just a generic kernel API, it works out of the box on 64-bit OSes with programs including ffmpeg using it.

Since digging into the internals of MMAL seemed complex, even to knowledgable folks in the Pi community, I decided to try to moving ustreamer’s H264 encoding to use V4L2 instead. As a side benefit, since we’re using Linux kernel APIs, this should also futureproof this code to any new revisions of the Pi.

I quickly ran into the fact that, while lots of people have used V42L for cameras and webcams, only the big projects like GStreamer and ffmpeg use it for encoding. This meant there was no simple guide or example I could follow. However, with a lot of poring over kernel documentation and trial and error, I learnt enough to successfully perform H264 hardware encoding using V4L2 and modified ustreamer to support it.

Using V4L2: step by step

Generally, V4L2 works in either “capture” (frames being read off a device e.g. camera or webcam) or “output” (frames being sent to a device e.g. a graphics card) M2M combines “capture” and “output” into a single mode which allows arbitrary transformations of frames. Video encoding is the obvious example of this but decoding is also supported.

All communication with the hardware encoder on the Pi is done by opening a special device in /dev (/dev/video11 to be precise), performing ioctls on the file descriptor to control the encoding and using mmap-ed memory to pass the frames between userspace and kernelspace.

There are two phases to the encoding process: setup and streaming. In the setup phase, we tell the encoder about things like the pixel format, resolution, frame rate and setup the buffers to pass the frames. The streaming phase involves writing the next raw frame of data to the kernel and reading back the encoded frame.

Note: all error handling is omitted to make the code snippets smaller.

Setup

First, we open the special device for the video encoder. This gives an fd we can use for ioctls.

#include <linux/videodev2.h>

int fd = open("/dev/video11", O_RDWR);

Next, we setup the “output” and “capture” devices; the most important things to set are resolution and pixel format. The “capture” device will be implicitly set to produce H264 frames with the same pixel format as the output but the resolution needs to be set manually.

Note: confusingly “output” here refers to the raw frames being encoded and “capture” to the encoded H264 output frames. A good way to think about this is to consider how V4L2 was originally designed: frames are sent to output devices like graphics cards and read from capture devices like cameras.

For 1080p raw frames in YUYV format:

struct v4l2_format fm;
struct v4l2_pix_format_mplane *mp = &fm.fmt.pix_mp;

fm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
ioctl(fd, VIDIOC_G_FMT, &fm);

mp->width = 1920;
mp->height = 1080;
mp->pixelformat = V4L2_PIX_FMT_YUYV;
ioctl(fd, VIDIOC_S_FMT, &fm);

fm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
ioctl(fd, VIDIOC_G_FMT, &fm);

mp->width = 1920;
mp->height = 1080;
ioctl(fd, VIDIOC_S_FMT, &fm);

Now, we set the inter-frame interval to indirectly set the frame rate. For a 30FPS video:

struct v4l2_streamparm stream;
memset(&stream, 0, sizeof(stream));
stream.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
stream.parm.output.timeperframe.numerator = 1;
stream.parm.output.timeperframe.denominator = 30;

ioctl(fd, VIDIOC_S_PARM, &stream);

The most complex part of the setup is configuring the mmap buffers to send and receive frames. Since we’re not trying to be performant, we use one capture buffer and one output buffer.

struct buffer {
 void* start;
 int length;
 struct v4l2_buffer inner;
 struct v4l2_plane plane;
};

// mmaps the buffers for the given type of device (capture or output).
void map(int fd, uint32_t type, struct buffer* buffer) {
 struct v4l2_buffer *inner = &buffer->inner;

 memset(inner, 0, sizeof(*inner));
 inner->type = type;
 inner->memory = V4L2_MEMORY_MMAP;
 inner->index = 0;
 inner->length = 1;
 inner->m.planes = &buffer->plane;

 ioctl(fd, VIDIOC_QUERYBUF, inner);

 buffer->length = inner->m.planes[0].length;
 buffer->start = mmap(NULL, buffer->length, PROT_READ | PROT_WRITE,
 MAP_SHARED, fd, inner->m.planes[0].m.mem_offset);
}

struct v4l2_requestbuffers buf;
buf.memory = V4L2_MEMORY_MMAP;
buf.count = 1;

struct buffer output;
buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
ioctl(fd, VIDIOC_REQBUFS, &buf);
map(fd, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, &output);

struct buffer capture;
buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
ioctl(fd, VIDIOC_REQBUFS, &buf);
map(fd, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE, &capture);

Next, we “queue” up the capture and output buffers. This tells the encoder that it has exclusive access to these buffers until we dequeue them.

ioctl(fd, VIDIOC_QBUF, &capture->inner);
ioctl(fd, VIDIOC_QBUF, &output->inner);

Finally, we start the streaming process; this tells the kernel that it should start reading raw frames from the output buffer and writing encoded frames to the capture buffer:

int type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
ioctl(fd, VIDIOC_STREAMON, &type);

type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
ioctl(fd, VIDIOC_STREAMON, &type);

Streaming

The actual encoding process is a straightforward loop where we:

dequeue an output frame, write the next frame and queue it again.
dequeue a capture frame, read the encoded frame and queue it again.

// Assumed functions for reading a raw frame and writing an encoded frame.
// Depending on application this can be from a file, a device or anywhere else.
int read_raw(void* buf, size_t len);
void write_encoded(void* buf, size_t len);

int bytes;
do {
 struct v4l2_buffer buf;
 buf.memory = V4L2_MEMORY_MMAP;
 buf.length = 1;

 struct v4l2_plane out_planes;
 memset(&out_planes, 0, sizeof(out_planes));
 buf.m.planes = &out_planes;

 // Dequeue the output buffer, read the next frame and queue it back.
 buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
 ioctl(fd, VIDIOC_DQBUF, &buf);

 bytes = read_raw(output->start, output->length);
 output->plane.bytesused = bytes;

 ioctl(fd, VIDIOC_QBUF, &output->inner);

 // Dequeue the capture buffer, write out the encoded frame and queue it back.
 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
 ioctl(fd, VIDIOC_DQBUF, &buf);

 int encoded_len = buf.m.planes[0].bytesused;
 write_encoded(capture->start, encoded_len);

 ioctl(fd, VIDIOC_QBUF, &capture->inner);
} while (bytes != 0);

What I learned

There are several useful things I learned while working on this project:

The kernel documentation had some excellent high-level explanations about how the API worked
- Specifically this docs page was an exceptional resource as it lays out all the steps needed to perform encoding
- Unfortunately, it only rarely provides code samples so a lot of trial and error was still needed.
The kernel source was invaluable to figure out why error codes were being returned.
- An ENOSYS error code is not helpful when it is implementation specific…
I discovered quite late in the project that loading the bcm2835_codec kernel module with debugging turned on (by setting the parameter debug=3) allowed visibility into log messages which I could cross reference with the kernel sources.
- This would have saved a bunch of time if I’d known this at the start
Looking at strace of ffmpeg trying to encode the same video as my program allowed me to see if there was any hidden or different syscalls.
- Instead of an explicit error being returned, I would sometimes end up being unable to dequeue a capture or output frame, hanging ustreamer.
- This included several times when I had missed a required field in a setup ioctl and when I had messed up the order of the queue/dequeue loop.

Conclusion

Hopefully, this article has showed how to approach doing video encoding on the Raspberry Pi from first principles. I’ve also tried to give an overview of the V4L2 API; the beauty of a standard API means much of what I’ve said here is not specific to the Pi but could be reused on any number of situations.

There is a lot more interesting ground I could have covered including non-blocking streaming, DMA and draining the encoder but this post has already run long enough…