Lalit Maganti (Atom, Tag: Software-Engineering)

Don't answer the first question

2026-05-16T14:19:00+01:00

In my work on Perfetto, a performance debugging tool, one question I get often is: “how do I split a Perfetto trace into multiple files?” Instead of answering directly, I say: “there isn’t an easy way to do that, but what’s leading you to collect traces large enough to want to split?”

This is one of my golden rules at work. When a user asks me something “weird”: don’t answer the first version of the question.

On the surface this might appear like I’m talking about the XY problem, but that stops one step short. It treats the user’s stated question as a puzzle to decode: figure out what they really meant, answer that, move on. I think we can go much further.

Instead, the confusion that produced the wrong question is itself an opening, and the conversation it sparks is valuable to both sides. The user walks away with a better mental model of the tool. I walk away with a clearer picture of where the product confuses people. And sometimes, between us, we figure out that the product itself needs to change.

I’ve written before about how I can still be a successful engineer while avoiding the spotlight. While that covered the general strategy, this is one of the concrete tactics that makes it work. I should also say this post is aimed at people who build things for other engineers. If you’re building a consumer product, or a B2B service, it will translate less directly, but the underlying instinct might still be useful.

Diagnosing the ask

Some questions are easy, routine, and purely a matter of pointing at documentation; those don’t merit much discussion here. The interesting cases are where something is out of the ordinary, and it’s rare that the user will have given me enough information in their first ask¹.

So I run a mental checklist to figure out where to go next:

Have I seen this before? If so, I might already have an answer to hand. If not, it’s uncommon enough that I want to slow down.
Does the question even sound reasonable compared to others I’ve seen? If not, why might they be asking it, and is there a more normal question underneath?
Does it fit the shape of the tool? Or is the user fighting the architecture without realizing it?

Once I’ve figured out what feels off, the next step is asking something that will surface the missing context. I might say something like “well the answer to your immediate question is X but that’s a pretty strange thing to ask for because of reason Y. Can you tell me more about the wider problem you’re trying to solve?”

This will probably be the start of a back and forth. How quickly it moves depends on how well the user can communicate their thoughts. But we’ll usually end up in one of a few places: they’re missing the philosophy of the tool, the product is hiding the right path or the product itself needs to change.

When they’re missing the philosophy

It’s quite common for users to come to us not knowing what they want, or not understanding the problem they’re trying to solve.

To be clear, I’m not criticizing them for this; teams are often trying to solve problems with limited time or resources, and they turn to new debugging tools when they’re struggling to make progress. As a result, they often find the tool, find it does most of what they want but doesn’t match their model of “how it should work”. So they file a feature request.

A common version of this: people come to Perfetto, see that a trace is a highly detailed recording of what a device did over a window of time, realize you can compute metrics from a Perfetto trace, and treat it as a holy grail solution to all their problems. Want a frame rate? Count frames in the trace. Memory used by an app? Look at the allocations and frees. In principle, any metric could be computed from a trace.

But this is a bad idea for a simple reason: traces are expensive to collect and process: you’re collecting all the data about the system rather than samply a single number. You’re going to waste a lot of resources when instead, a dedicated metric collection system would do the job much more efficiently².

My overarching point is that there’s a certain philosophy to how tools are designed, and users often miss it because they’re focused on their immediate problem.

A big part of my job is teaching the team how to approach performance engineering in the first place, not just explaining how to use Perfetto. It means making people aware of the tools they have available, how to think about things like startup, frame drops, memory, and power, and how to work with them both in normal situations and when something goes wrong.

When the right path is hidden

Other times the team understands the problem; they just don’t see how to put existing tools together. Our tools are powerful by design, and we have to be mindful that other teams might not understand the full range of what we’ve built. It’s my job to figure out what they actually want. Often, something we built for a different purpose can be repurposed to meet their needs.

A perfect example here is what I already discussed: trace splitting. The conversation goes something like, “…what’s leading you to collect traces large enough to want to split?” They say it’s because they have periods of interest in a long trace and want to slice it up. Partly for performance, partly to make visualizing easier.

But then I point out that Perfetto already supports periodic trace snapshots, short repeated recordings instead of one long one, which removes the need to collect a long trace at all. They’re trying to solve a problem they shouldn’t be having in the first place.

It’s always satisfying to see people say “that’s exactly what I needed!” even though it’s not what they asked for. It means I successfully figured out what they actually wanted rather than what they thought they did.

When the product needs to change

Occasionally, the response reveals something genuinely new, something that could set us on the path to building something big. These cases are tricky: even when the ask is novel, the asker often can’t tell you what they actually need.

The cost of getting things wrong in foundational software is high, so I err on the side of not building something until not having it hurts: multiple teams have come to me saying “we want this.” Ideally by then we’ve found the essence of what’s actually worth building. This is very unlikely to happen after just one ask, so waiting is really powerful.

We’ve made this mistake at Perfetto. Take ad-hoc UI customization. People wanted to hack on the UI to suit their workflows, and they kept complaining about how hard that was. So we let them, and immediately took on a huge amount of technical debt. Every new feature had to interact with every existing one, and the whole thing quickly became impossible to scale.

It took us roughly one year to dig ourselves out of this hole by designing a proper plugin API. The real need, which no one could articulate up front, was a way to personalize the UI to the needs of their team or use case without affecting every other user. But of course no one was able to name this until it was too late. So it was our responsibility to spot this early as the requests flowed in.

Allowing folks to “merge” Perfetto traces was one we got right. People asked about it constantly, but we held off. We pointed people at workarounds and said “we’ll see.” We knew doing it properly was a lot of work and easy to get wrong, so we waited. We finally built it last year in a maintainable way, but only because we understood the problem space so well in the first place.

The point

The first version of a question is rarely the real one. Ask why before you answer. Sometimes the answer teaches the user how the tool is meant to be used. Sometimes it tells you the product is hiding the right path. Sometimes it tells you there’s nothing worth building yet. And sometimes it sets you up to build the next big thing once, not twice.

The technique is simple, but easy to skip because the pull to be responsive is constant, and every quick answer feels productive. Take the small step back anyway. Both sides almost always walk away with more than they came in with.

It’s always a pleasant surprise when that does happen! ↩︎
Computing metrics from traces works fine for local development and most lab tests. It breaks down when you need fine-grained measurement with low noise, or when you’re collecting in the field on real users’ devices. ↩︎

Eight years of wanting, three months of building with AI

2026-04-05T13:00:00+01:00

For eight years, I’ve wanted a high-quality set of devtools for working with SQLite. Given how important SQLite is to the industry¹, I’ve long been puzzled that no one has invested in building a really good developer experience for it².

A couple of weeks ago, after ~250 hours of effort over three months³ on evenings, weekends, and vacation days, I finally released syntaqlite (GitHub), fulfilling this long-held wish. And I believe the main reason this happened was because of AI coding agents⁴.

Of course, there’s no shortage of posts claiming that AI one-shot their project or pushing back and declaring that AI is all slop. I’m going to take a very different approach and, instead, systematically break down my experience building syntaqlite with AI, both where it helped and where it was detrimental.

I’ll do this while contextualizing the project and my background so you can independently assess how generalizable this experience was. And whenever I make a claim, I’ll try to back it up with evidence from my project journal, coding transcripts, or commit history⁵.

Why I wanted it

In my work on Perfetto, I maintain a SQLite-based language for querying performance traces called PerfettoSQL. It’s basically the same as SQLite but with a few extensions to make the trace querying experience better. There are ~100K lines of PerfettoSQL internally in Google and it’s used by a wide range of teams.

Having a language which gets traction means your users also start expecting things like formatters, linters, and editor extensions. I’d hoped that we could adapt some SQLite tools from open source but the more I looked into it, the more disappointed I was. What I found either wasn’t reliable enough, fast enough⁶, or flexible enough to adapt to PerfettoSQL. There was clearly an opportunity to build something from scratch, but it was never the “most important thing we could work on”. We’ve been reluctantly making do with the tools out there but always wishing for better.

On the other hand, there was the option to do something in my spare time. I had built lots of open source projects in my teens⁷ but this had faded away during university when I felt that I just didn’t have the motivation anymore. Being a maintainer is much more than just “throwing the code out there” and seeing what happens. It’s triaging bugs, investigating crashes, writing documentation, building a community, and, most importantly, having a direction for the project.

But the itch of open source (specifically freedom to work on what I wanted while helping others) had never gone away. The SQLite devtools project was eternally in my mind as “something I’d like to work on”. But there was another reason why I kept putting it off: it sits at the intersection of being both hard and tedious.

What makes it hard and tedious

If I was going to invest my personal time working on this project, I didn’t want to build something that only helped Perfetto: I wanted to make it work for any SQLite user out there⁸. And this means parsing SQL exactly like SQLite.

The heart of any language-oriented devtool is the parser. This is responsible for turning the source code into a “parse tree” which acts as the central data structure anything else is built on top of. If your parser isn’t accurate, then your formatters and linters will inevitably inherit those inaccuracies; many of the tools I found suffered from having parsers which approximated the SQLite language rather than representing it precisely.

Unfortunately, unlike many other languages, SQLite has no formal specification describing how it should be parsed. It doesn’t expose a stable API for its parser either. In fact, quite uniquely, in its implementation it doesn’t even build a parse tree at all⁹! The only reasonable approach left in my opinion is to carefully extract the relevant parts of SQLite’s source code and adapt it to build the parser I wanted¹⁰.

This means getting into the weeds of SQLite source code, a fiendishly difficult codebase to understand. The whole project is written in C in an incredibly dense style; I’ve spent days just understanding the virtual table API¹¹ and implementation. Trying to grasp the full parser stack was daunting.

There’s also the fact that there are >400 rules in SQLite which capture the full surface area of its language. I’d have to specify in each of these “grammar rules” how that part of the syntax maps to the matching node in the parse tree. It’s extremely repetitive work; each rule is similar to all the ones around it but also, by definition, different.

And it’s not just the rules but also coming up with and writing tests to make sure it’s correct, debugging if something is wrong, triaging and fixing the inevitable bugs people filed when I got something wrong…

For years, this was where the idea died. Too hard for a side project¹², too tedious to sustain motivation, too risky to invest months into something that might not work.

How it happened

I’ve been using coding agents since early 2025 (Aider, Roo Code, then Claude Code since July) and they’d definitely been useful but never something I felt I could trust a serious project to. But towards the end of 2025, the models seemed to make a significant step forward in quality¹³. At the same time, I kept hitting problems in Perfetto which would have been trivially solved by having a reliable parser. Each workaround left the same thought in the back of my mind: maybe it’s finally time to build it for real.

I got some space to think and reflect over Christmas and decided to really stress test the most maximalist version of AI: could I vibe-code the whole thing using just Claude Code on the Max plan (£200/month)?

Through most of January, I iterated, acting as semi-technical manager and delegating almost all the design and all the implementation to Claude. Functionally, I ended up in a reasonable place: a parser in C extracted from SQLite sources using a bunch of Python scripts, a formatter built on top, support for both the SQLite language and the PerfettoSQL extensions, all exposed in a web playground.

But when I reviewed the codebase in detail in late January, the downside was obvious: the codebase was complete spaghetti¹⁴. I didn’t understand large parts of the Python source extraction pipeline, functions were scattered in random files without a clear shape, and a few files had grown to several thousand lines. It was extremely fragile; it solved the immediate problem but it was never going to cope with my larger vision, never mind integrating it into the Perfetto tools. The saving grace was that it had proved the approach was viable and generated more than 500 tests, many of which I felt I could reuse.

I decided to throw away everything and start from scratch while also switching most of the codebase to Rust¹⁵. I could see that C was going to make it difficult to build the higher level components like the validator and the language server implementation. And as a bonus, it would also let me use the same language for both the extraction and runtime instead of splitting it across C and Python.

More importantly, I completely changed my role in the project. I took ownership of all decisions¹⁶ and used it more as “autocomplete on steroids” inside a much tighter process: opinionated design upfront, reviewing every change thoroughly, fixing problems eagerly as I spotted them, and investing in scaffolding (like linting, validation, and non-trivial testing¹⁷) to check AI output automatically.

The core features came together through February and the final stretch (upstream test validation, editor extensions, packaging, docs) led to a 0.1 launch in mid-March.

But in my opinion, this timeline is the least interesting part of this story. What I really want to talk about is what wouldn’t have happened without AI and also the toll it took on me as I used it.

AI is why this project exists, and why it’s as complete as it is

Overcoming inertia

I’ve written in the past about how one of my biggest weaknesses as a software engineer is my tendency to procrastinate when facing a big new project. Though I didn’t realize it at the time, it could not have applied more perfectly to building syntaqlite.

AI basically let me put aside all my doubts on technical calls, my uncertainty of building the right thing and my reluctance to get started by giving me very concrete problems to work on. Instead of “I need to understand how SQLite’s parsing works”, it was “I need to get AI to suggest an approach for me so I can tear it up and build something better"¹⁸. I work so much better with concrete prototypes to play with and code to look at than endlessly thinking about designs in my head, and AI lets me get to that point at a pace I could not have dreamed about before. Once I took the first step, every step after that was so much easier.

Faster at churning code

AI turned out to be better than me at the act of writing code itself, assuming that code is obvious. If I can break a problem down to “write a function with this behaviour and parameters” or “write a class matching this interface,” AI will build it faster than I would and, crucially, in a style that might well be more intuitive to a future reader. It documents things I’d skip, lays out code consistently with the rest of the project, and sticks to what you might call the “standard dialect” of whatever language you’re working in¹⁹.

That standardness is a double-edged sword. For the vast majority of code in any project, standard is exactly what you want: predictable, readable, unsurprising. But every project has pieces that are its edge, the parts where the value comes from doing something non-obvious. For syntaqlite, that was the extraction pipeline and the parser architecture. AI’s instinct to normalize was actively harmful there, and those were the parts I had to design in depth and often resorted to just writing myself.

But here’s the flip side: the same speed that makes AI great at obvious code also makes it great at refactoring. If you’re using AI to generate code at industrial scale, you have to refactor constantly and continuously²⁰. If you don’t, things immediately get out of hand. This was the central lesson of the vibe-coding month: I didn’t refactor enough, the codebase became something I couldn’t reason about, and I had to throw it all away. In the rewrite, refactoring became the core of my workflow. After every large batch of generated code, I’d step back and ask “is this ugly?” Sometimes AI could clean it up. Other times there was a large-scale abstraction that AI couldn’t see but I could; I’d give it the direction and let it execute²¹. If you have taste, the cost of a wrong approach drops dramatically because you can restructure quickly²².

Teaching assistant

Of all the ways I used AI, research had by far the highest ratio of value delivered to time spent.

I’ve worked with interpreters and parsers before but I had never heard of Wadler-Lindig pretty printing²³. When I needed to build the formatter, AI gave me a concrete and actionable lesson from a point of view I could understand and pointed me to the papers to learn more. I could have found this myself eventually, but AI compressed what might have been a day or two of reading into a focused conversation where I could ask “but why does this work?” until I actually got it.

This extended to entire domains I’d never worked in. I have deep C++ and Android performance expertise but had barely touched Rust tooling or editor extension APIs. With AI, it wasn’t a problem: the fundamentals are the same, the terminology is similar, and AI bridges the gap²⁴. The VS Code extension would have taken me a day or two of learning the API before I could even start. With AI, I had a working extension within an hour.

It was also invaluable for reacquainting myself with parts of the project I hadn’t looked at for a few days²⁵. I could control how deep to go: “tell me about this component” for a surface-level refresher, “give me a detailed linear walkthrough” for a deeper dive, “audit unsafe usages in this repo” to go hunting for problems. When you’re context switching a lot, you lose context fast. AI let me reacquire it on demand.

More than I’d have built alone

Beyond making the project exist at all, AI is also the reason it shipped as complete as it did. Every open source project has a long tail of features that are important but not critical: the things you know theoretically how to do but keep deprioritizing because the core work is more pressing. For syntaqlite, that list was long: editor extensions, Python bindings, a WASM playground, a docs site, packaging for multiple ecosystems²⁶. AI made these cheap enough that skipping them felt like the wrong trade-off.

It also freed up mental energy for UX²⁷. Instead of spending all my time on implementation, I could think about what a user’s first experience should feel like: what error messages would actually help them fix their SQL, how the formatter output should look by default, whether the CLI flags were intuitive. These are the things that separate a tool people try once from one they keep using, and AI gave me the headroom to care about them. Without AI, I would have built something much smaller, probably no editor extensions or docs site. AI didn’t just make the same project faster. It changed what the project was.

Where AI had its costs

The addiction

There’s an uncomfortable parallel between using AI coding tools and playing slot machines²⁸. You send a prompt, wait, and either get something great or something useless. I found myself up late at night wanting to do “just one more prompt,” constantly trying AI just to see what would happen even when I knew it probably wouldn’t work. The sunk cost fallacy kicked in too: I’d keep at it even in tasks it was clearly ill-suited for, telling myself “maybe if I phrase it differently this time.”

The tiredness feedback loop made it worse²⁹. When I had energy, I could write precise, well-scoped prompts and be genuinely productive. But when I was tired, my prompts became vague, the output got worse, and I’d try again, getting more tired in the process. In these cases, AI was probably slower than just implementing something myself, but it was too hard to break out of the loop³⁰.

Losing touch

Several times during the project, I lost my mental model of the codebase³¹. Not the overall architecture or how things fitted together. But the day-to-day details of what lived where, which functions called which, the small decisions that accumulate into a working system. When that happened, surprising issues would appear and I’d find myself at a total loss to understand what was going wrong. I hated that feeling.

The deeper problem was that losing touch created a communication breakdown³². When you don’t have the mental thread of what’s going on, it becomes impossible to communicate meaningfully with the agent. Every exchange gets longer and more verbose. Instead of “change FooClass to do X,” you end up saying “change the thing which does Bar to do X”. Then the agent has to figure out what Bar is, how that maps to FooClass, and sometimes it gets it wrong³³. It’s exactly the same complaint engineers have always had about managers who don’t understand the code asking for fanciful or impossible things. Except now you’ve become that manager.

The fix was deliberate: I made it a habit to read through the code immediately after it was implemented and actively engage to see “how would I have done this differently?”.

Of course, in some sense all of the above is also true of code I wrote a few months ago (hence the sentiment that AI code is legacy code), but AI makes the drift happen faster because you’re not building the same muscle memory that comes from originally typing it out.

The slow corrosion

There were some other problems I only discovered incrementally over the three months.

I found that AI made me procrastinate on key design decisions³⁴. Because refactoring was cheap, I could always say “I’ll deal with this later.” And because AI could refactor at the same industrial scale it generated code, the cost of deferring felt low. But it wasn’t: deferring decisions corroded my ability to think clearly because the codebase stayed confusing in the meantime. The vibe-coding month was the most extreme version of this. Yes, I understood the problem, but if I had been more disciplined about making hard design calls earlier, I could have converged on the right architecture much faster.

Tests created a similar false comfort³⁵. Having 500+ tests felt reassuring, and AI made it easy to generate more. But neither humans nor AI are creative enough to foresee every edge case you’ll hit in the future; there are several times in the vibe-coding phase where I’d come up with a test case and realise the design of some component was completely wrong and needed to be totally reworked. This was a significant contributor to my lack of trust and the decision to scrap everything and start from scratch.

Basically, I learned that the “normal rules” of software still apply in the AI age: if you don’t have a fundamental foundation (clear architecture, well-defined boundaries) you’ll be left eternally chasing bugs as they appear.

No sense of time

Something I kept coming back to was how little AI understood about the passage of time³⁶. It sees a codebase in a certain state but doesn’t feel time the way humans do. I can tell you what it feels like to use an API, how it evolved over months or years, why certain decisions were made and later reversed.

The natural problem from this lack of understanding is that you either make the same mistakes you made in the past and have to relearn the lessons or you fall into new traps which were successfully avoided the first time, slowing you down in the long run. In my opinion, this is a similar problem to why losing a high-quality senior engineer hurts a team so much: they carry history and context that doesn’t exist anywhere else and act as a guide for others around them.

In theory, you can try to preserve this context by keeping specs and docs up to date. But there’s a reason we didn’t do this before AI: capturing implicit design decisions exhaustively is incredibly expensive and time-consuming to write down. AI can help draft these docs, but because there’s no way to automatically verify that it accurately captured what matters, a human still has to manually audit the result. And that’s still time-consuming.

There’s also the context pollution problem. You never know when a design note about API A will echo in API B. Consistency is a huge part of what makes codebases work, and for that you don’t just need context about what you’re working on right now but also about other things which were designed in a similar way. Deciding what’s relevant requires exactly the kind of judgement that institutional knowledge provides in the first place.

Relativity

Reflecting on the above, the pattern of when AI helped and when it hurt was fairly consistent.

When I was working on something I already understood deeply, AI was excellent. I could review its output instantly, catch mistakes before they landed and move at a pace I’d never have managed alone. The parser rule generation is the clearest example³⁷: I knew exactly what each rule should produce, so I could review AI’s output within a minute or two and iterate fast.

When I was working on something I could describe but didn’t yet know, AI was good but required more care. Learning Wadler-Lindig for the formatter was like this: I could articulate what I wanted, evaluate whether the output was heading in the right direction, and learn from what AI explained. But I had to stay engaged and couldn’t just accept what it gave me.

When I was working on something where I didn’t even know what I wanted, AI was somewhere between unhelpful and harmful. The architecture of the project was the clearest case: I spent weeks in the early days following AI down dead ends, exploring designs that felt productive in the moment but collapsed under scrutiny. In hindsight, I have to wonder if it would have been faster just thinking it through without AI in the loop at all.

But expertise alone isn’t enough. Even when I understood a problem deeply, AI still struggled if the task had no objectively checkable answer³⁸. Implementation has a right answer, at least at a local level: the code compiles, the tests pass, the output matches what you asked for. Design doesn’t. We’re still arguing about OOP decades after it first took off.

Concretely, I found that designing the public API of syntaqlite was where this hit home the hardest. I spent several days in early March doing nothing but API refactoring, manually fixing things any experienced engineer would have instinctively avoided but AI made a total mess of. There’s no test or objective metric for “is this API pleasant to use” and “will this API help users solve the problems they have” and that’s exactly why the coding agents did so badly at it.

This takes me back to the days I was obsessed with physics and, specifically, relativity. The laws of physics look simple and Newtonian in any small local area, but zoom out and spacetime curves in ways you can’t predict from the local picture alone. Code is the same: at the level of a function or a class, there’s usually a clear right answer, and AI is excellent there. But architecture is what happens when all those local pieces interact, and you can’t get good global behaviour by stitching together locally correct components.

Knowing where you are on these axes at any given moment is, I think, the core skill of working with AI effectively.

Wrap-up

Eight years is a long time to carry a project in your head. Seeing these SQLite tools actually exist and function after only three months of work is a massive win, and I’m fully aware they wouldn’t be here without AI.

But the process wasn’t the clean, linear success story people usually post. I lost an entire month to vibe-coding. I fell into the trap of managing a codebase I didn’t actually understand, and I paid for that with a total rewrite.

The takeaway for me is simple: AI is an incredible force multiplier for implementation, but it’s a dangerous substitute for design. It’s brilliant at giving you the right answer to a specific technical question, but it has no sense of history, taste, or how a human will actually feel using your API. If you rely on it for the “soul” of your software, you’ll just end up hitting a wall faster than you ever have before.

What I’d like to see more of from others is exactly what I’ve tried to do here: honest, detailed accounts of building real software with these tools; not weekend toys or one-off scripts but the kind of software that has to survive contact with users, bug reports, and your own changing mind.

Why Senior Engineers Let Bad Projects Fail

2026-01-13T10:48:00Z

When I was a junior engineer, my manager would occasionally confide his frustrations to me in our weekly 1:1s. He would point out a project another team was working on and say, “I don’t believe that project will go anywhere, they’re solving the wrong problem.” I used to wonder, “But you are very senior, why don’t you just go and speak to them about your concerns?” It felt like a waste of his influence to not say anything.

So it’s quite ironic that I found myself last week explaining to a mentee why I thought a sister team’s project would have to pivot because they’d made a poor early design choice. And he rightfully asked me the same question I had years ago: “why don’t you just tell them your opinion?” It’s been on my mind ever since because I realized I’d changed my stance on it a lot over the years.

The answer is that being right and being effective are different.

In large companies, speaking up about what you see as a “bad project” is a good thing. But only in moderation. Sometimes the mark of seniority is realizing that arguing with people who won’t listen isn’t worth it; it’s better to save your counsel.

Bad projects

What I mean by a “bad project” is many things:

UX: making product complicated, solving a problem which doesn’t exist, breaking existing workflows
Technical: overcomplicated design, wrong library, poor performing architecture
Political: chasing hype cycles, exists primarily to justify a promotion

It’s important to point out that for much of the lifecycle of a project, whether it’s “bad” is highly subjective. Software engineering is largely a game of tradeoffs and making decisions which are not perfect but the best possible with the information available. There often can be disagreements on whether correct choices are made and it only becomes obvious much later on, potentially years after a project has shipped.

But as you become more senior, you’ll start to have “taste” when it comes to software projects and that will cause you to look at some fraction of the software projects and feel “this doesn’t make sense”. And this gut feeling is the sign to me of a “bad project”, one which you can see in advance of when it’s obvious to everyone.

Drawing on my personal experience, the most memorable example was a few years ago at Google . There was a high-profile announcement internally of a “game changer” project that sat right at the intersection of two extremely large organizations. It was technically amazing and elegant, and full of clever ideas for really hard problems.

But I distinctly remember sitting in the room for the announcement, turning to my lead and whispering, “This project has no chance of succeeding, right?” He turned to me and just said, “Yup.” We both realized the problem immediately. The project was entirely based on a platform team asking a flagship product team to give up control of their core user flow: technically the right move, but no lead or PM would ever cede ownership of something that central to another team. Politically, this project was a total fantasy.

The project kept quietly chugging away in the background for almost two years. Every time it got close to launch, it would get pushed back as “not ready yet.” Over time, we heard less and less about it until, eventually, the inevitable “strategic pivot” email appeared in my inbox. Resources were reallocated and the code was deleted. We were told the company “learned a lot from the effort,” but to me it felt like it was doomed from the beginning. Politics and solving the correct problem matter just as much as technical beauty.

Why you cannot stop them all

When I started noticing “bad projects” and I felt that I had some expertise to share, the temptation for me was to start calling them out. Reach out to the team doing it, tell them “this doesn’t make sense” and explain to them why. Use facts and logic to persuade.

And I did do this. But only for a very short time before I realized that there are a lot of costs to doing this that I just wasn’t thinking about.

Firstly, software companies have an inherent bias for action. They value speed and shipping highly. Concerns, by definition, slow things down and mean people have to look at things which they hadn’t budgeted for. And so unless your concern is big enough to overcome the “push for landing”, there’s little chance for any meaningful change to come from you saying something. In fact, it’s very likely that you’ll be largely ignored.

Related to this, even if the team does take your concern seriously, you have to be careful not to do it too often. Once or twice, you might be seen as someone who is upholding “quality”. But do it too often and you quickly move to being seen as a “negative person”, someone who is constantly a problem maker, not a problem “fixer”. You rarely get credit for the disasters you prevented. Because nothing happened, people forget about it quickly.

There’s also the problem that every time you push back, you are potentially harming someone’s promotion packet or a VP’s “pet project.” You are at risk of burning bridges and creating “enemies”, at least of a sort. Having a few people who disagree in a big company with you is the cost of doing business, but if you have too many, it starts affecting your main work too.

Finally, there is also the psychological impact. There is one of you and hundreds of engineers working in spaces that your expertise might help with. Your attention is finite, but the capacity for a large company to generate bad ideas is infinite. Speaking from experience, getting too involved in stopping these quickly can make you very cynical about the state of the world. And this is really not a good place to be.

Manage influence like a bank account

So if you cannot stop all the bad projects, what do you do? You get strategic. Instead of trying to fix everything, view your influence as a bank account. You have a certain amount of “influence” coming in every month as you do your job, help people, ship successful projects, and generally remain low friction.

Then, when it matters, you should be ready to make “withdrawals.” Every time you block something or raise concerns, no matter how small, you are writing a check against your balance. But not all checks are the same size:

The $5 Check: A nitpick on a code review. Cheap, daily expense.
The $500 Check: Challenging an architectural decision or pushing back on a timeline. Requires some savings.
The $50,000 Check: Trying to kill a VP’s pet project. This is a massive spend. You might only afford this once every few years.

The problem comes if you spend $5 on every minor inefficiency you see. If you are constantly saying “no” to small things, your account will be empty when you need to write the big check to stop a true disaster.

If you “go overdrawn,” you enter political bankruptcy. People stop inviting you to meetings, they stop asking for your opinion, they essentially start working around you. Once you are bankrupt, your influence drops to zero and you not only harm your ability to influence things but also start hurting your own ability to get things done.

When to spend influence

Given that we’ve now accepted that we cannot weigh in on everything, we need to figure out when it does make sense to do so.

The most important thing to do first is to be humble and evaluate whether you actually have the expertise to make a judgment. Seniority often brings opinions, but those are not always informed opinions. For example, while I have some frontend experience, I do not feel qualified to give deep advice on it because my knowledge is “enough to get by” rather than deep expertise that comes from long term ownership. It is easy to lose sight of the fact that high-quality judgments require informed opinions. If you find yourself in this position, see yourself as an opinionated observer and stop there.

You must also internalize the fact that just because you say something does not make it the truth. You are raising awareness of a point of view, not issuing a decree. So if some team doesn’t listen to your concerns and decides to go ahead with what they were doing anyway, then you have to accept that and move on: at the end of the day, you’re an engineer, not a CEO with authority over them!

Given these points, I use three main factors to decide when to speak up:

How close is the project to my team?
If it goes wrong, how much impact will it have on my team?
If it goes wrong, how big will the problem be for the company?

Proximity. If a project is close to you, the “price tag” of saying something is lower. If it is within your own team, the cost is near zero because you have high trust and a quick conversation often solves it. If it is in your broader organization, the price goes up; you have to spend social capital and potentially stake your reputation. If it is outside your org? The cost is often prohibitive. You have zero leverage, different reporting chains, and stopping it would require a massive withdrawal.

Team Impact. Sometimes another org does something that deeply affects your work. For example, because Perfetto (the performance tool I work on) has users throughout Google, sometimes a team will ask us to sign off on a very complex integration. This is a classic risk: if things go right, they get the credit, but if things go wrong, your leadership might expect you to help solve a problem you didn’t create. In these cases, the payoff of speaking up is high because you are protecting your team.

Company Scale. Finally, consider the blast radius. Some projects are self-contained; if they fail, they only take themselves down. Others are so intertwined with core systems that their failure causes widespread damage or creates technical debt that persists for years. These can be deadly to the long-term health of a project.

How to act with bad projects

It’s also not just about when you put your opinions forward but how you do it. There’s a very wide range of actions you can take depending on what you’re facing.

When you intervene

The nuclear option is to directly say “we should not do this” and try to shut the project down. This almost always requires escalation to your leads and the leads of the owning team, requiring great conviction in both the fact that you’re right and that this project will be actively harmful. But on some occasions, this is the right thing to do, especially if the cost of not saying something can be existential to your project or team.

A slightly softer but still quite risky variant of this is, instead of doing a direct escalation, you raise concerns in directly with the team. Usually this is done with a meeting with the team or a strongly worded “concern” or “rebuttal” doc. The goal is to speak in strong enough terms that the team themselves conclude that this the project might not be a good idea.

Then there are the smaller interventions, nudging things in the right direction. These are perfect for when a team is about to do something that makes sense from a high level but they are going about this the wrong way. I see this often with Perfetto: a team sends a design doc proposing a complex use of Perfetto that I know will cause them pain later. I sit down with them, understand their actual problem, and guide them to a better solution. It costs an hour but saves them months. If you do it right, you can even be seen as a helper rather than a hindrance, even if you do slow down the team.

When you don’t

Sometimes you conclude that the ROI just isn’t there to do anything direct: the political momentum is too strong, or the issue is too small to justify spending any influence. At this point, what you do depends on how much your team is involved.

If it overlaps with your team’s work heavily then it might be best to make some subtle contingency plans: reducing your dependency on it or building abstractions to cope if it goes away. There is also a long game trick here. Even a bad project usually has an “essence” of a good idea, a specific problem it was trying to solve or an insight it was based on. If it fits with your job, it’s often a good idea to take that essence and see if you can naturally incorporate a better version of that specific solution into your own project. That way, if the bad project stalls or gets canceled, you can be proactive instead of reactive to the fallout.

Alternatively, if you’re not involved, it’s easy: just stay out of the picture. Vent to friendly colleagues in private, commiserate, but in public, live with the reality.

Managing your team through it

Finally, you must manage your own team through the process. If you can see the flaws in a project, other senior engineers probably see them too. Don’t try to gaslight them or “walk the company line” by pretending a bad project is actually good. It destroys trust.

Instead, be honest about the facts on the ground without going into unnecessary political details. Tell them that you will do the best you can under these constraints.

Conclusion

So what did I tell my mentee? “I’ve learned that being right and being effective are different things. I could go tell them my concerns. They probably wouldn’t listen. I’d burn some goodwill. And in six months, nobody will remember that I called it, they’ll just remember I was the guy who tried to block their work”.

When you’re earlier in your career, you want to believe that good ideas win on merit, that if you just explain clearly enough, people will see reason. It took me quite some time to accept that big companies don’t work that way.

But this doesn’t mean you stop caring. It means you get strategic about when to spend your credibility. Pick the battles where you can actually change the outcome, where your team will be hurt if you stay silent, where the cost of being wrong is low but the cost of the project failing is high.

And for everything else? You vent to colleagues, you make quiet contingency plans, and you watch. Sometimes you learn something. Sometimes you’re wrong and the project actually works. And sometimes you get to feel that grim satisfaction of predicting exactly how things would fall apart.

None of this is as satisfying as fixing everything. But it works and keeps me sane.

Why I Ignore The Spotlight as a Staff Engineer

2025-12-04T00:15:00Z

Lately I’ve been reading Sean Goedecke’s essays on being a Staff+ engineer. His work (particularly Software engineering under the spotlight and It’s Not Your Codebase) is razor-sharp and feels painfully familiar to anyone in Big Tech.

On paper, I fit the mold he describes: I’m a Senior Staff engineer at Google. Yet, reading his work left me with a lingering sense of unease. At first, I dismissed this as cynicism. After reflecting, however, I realized the problem wasn’t Sean’s writing but my reading.

Sean isn’t being bleak; he is accurately describing how to deal with a world where engineers are fungible assets and priorities shift quarterly. But my job looks nothing like that and I know deep down that if I tried to operate in that environment or in the way he described I’d burn out within months.

Instead I’ve followed an alternate path, one that optimizes for systems over spotlights, and stewardship over fungibility.

We Live in Different Worlds

The foundational reason for our diverging paths is that Sean and I operate in entirely different worlds with different laws governing them.

From Sean’s resume, my understanding is that he has primarily worked in product teams ¹ building for external customers. Business goals pivot quarterly, and success is measured by revenue or MAU. Optimizing for the “Spotlight” makes complete sense in this environment. Product development at big tech scale is a crowded room: VPs, PMs and UX designers all have strong opinions. To succeed, you have to be agile and ensure you are working specifically on what executives are currently looking at.

On the other hand, I’ve spent my entire career much more behind the scenes: in developer tools and infra teams.

My team’s customers are thousands of engineers in Android, Chrome, and throughout Google ². End users of Google products don’t even know we exist; our focus is on making sure developers have the tools to collect product and performance metrics and debug issues using detailed traces.

In this environment, our relationship with leadership is very different. We’re never the “hot project everyone wants,” so execs are not fighting to work with us. In fact, my team has historically struggled to hire PMs. The PM career ladder at Google incentivizes splashy external launches so we cannot provide good “promotion material” for them. Also, our feedback comes directly from engineers. Adding a PM in the middle causes a loss in translation, slowing down a tight, high-bandwidth feedback loop.

All of this together means our team operates “bottom-up”: instead of execs telling us “you should do X”, we figure out what we think will have the most impact to our customers and work on building those features and tools. Execs ensure that we’re actually solving these problems by considering our impact on more product facing teams.

Compounding Returns of Stewardship

In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts. But in Infrastructure and Developer Experience, context is the currency.

Treating engineers as fungible assets destroys context. You might gain fresh eyes, but you lose the implicit knowledge of how systems actually break. Stewardship, staying with a system long-term, unlocks compounding returns that are impossible to achieve on a short rotation.

The first is efficiency via pattern matching. When you stay in one domain for years, new requests are rarely truly “new.” I am not just debugging code; I am debugging the intersection of my tools and hundreds of diverse engineering teams. When a new team comes to me with a “unique” problem, I can often reach back in time: “We tried this approach in 2021 with the Camera team; here is exactly why it failed, and here is the architecture that actually works”.

But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons.

Take Bigtrace, a project I recently led; it was a solution that emerged solely because I stuck around long enough to see the shape of the problem:

Start of 2023 (Observation): I began noticing a pattern. Teams across Google were collecting terabytes or even petabytes of performance traces, but they were struggling to process them. Engineers were writing brittle, custom pipelines to parse data, often complaining about how slow and painful it was to iterate on their analysis.
Most of 2023 (Research): I didn’t jump to build a production system. Instead, I spent the best part of a year prototyping quietly in the background while working on other projects. I gathered feedback from these same engineers who had complained and because I had established long-term relationships, they gave me honest and introspective feedback. I learned what sort of UX, latency and throughput requirements they had and figured out how I could meet them.
End of 2023 to Start of 2024 (Execution): We built and launched Bigtrace, a distributed big data query engine for traces. Today, it processes over 2 billion traces a month and is a critical part of the daily workflow for 100+ engineers.

If I had followed the advice to “optimize for fungibility” (i.e. if I had switched teams in 2023 to chase a new project) Bigtrace would not exist.

Instead, I would have left during the research phase and my successor would have seen the same “noise” of engineers complaining. But without the historical context to recognize a missing puzzle piece, I think they would have struggled to build something like Bigtrace.

The Power of “No”

One of the most seductive arguments for chasing the “Spotlight” is that it guarantees resources and executive attention. But that attention is a double-edged sword.

High-visibility projects are often volatile. They come with shifting executive whims, political maneuvering, and often end up in situations where long-term quality is sacrificed for short-term survival. For some engineers, navigating this chaos is a thrill. For those of us who care about system stability, it feels like a trap.

The advantage of stewardship is that it generates a different kind of capital: trust. When you have spent years delivering reliable tools, you earn the political capital to say “No” to the spotlight when it threatens the product.

Recently, the spotlight has been on AI. Every team is under pressure to incorporate it. We have been asked repeatedly: “Why don’t you integrate LLMs into Perfetto?” If I were optimizing for visibility, the answer would be obvious: build an LLM wrapper, demo it to leadership, and claim we are “AI-first.” It would be an easy win for my career.

But as a steward of the system, I know that one of Perfetto’s core values is precision. When a kernel developer is debugging a race condition, they need exact timestamps, not a hallucination. Users trust that when we tell them “X is the problem” that it actually is the problem and they’re not going to go chasing their tail for the next week, debugging an issue which doesn’t exist.

But it’s important not to take this too far: skepticism shouldn’t become obstructionism. With AI, it’s not “no forever” but “not until it can be done right” ³.

A spotlight-seeking engineer might view this approach as a missed opportunity; I view it as protecting what makes our product great: user trust.

The Alternate Currency of Impact

The most common fear engineers have about leaving the “Spotlight” is career stagnation. The logic goes: If I’m not launching flashy features at Google I/O, and my work isn’t on my VP’s top 5 list, how will I ever get promoted to Staff+?

It is true that you lose the currency of “Executive Visibility.” But in infrastructure, you gain two alternate currencies that are just as valuable, and potentially more stable.

Shadow Hierarchy

In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers.

I call this the Shadow Hierarchy. You don’t need your VP to understand the intricacies of your code. You need the Staff+ Engineers in other critical organizations to need your tools.

When a Senior Staff Engineer in Pixel tells their VP, “We literally cannot debug the next Pixel phone without Perfetto”, that statement carries immense weight. It travels up their reporting chain, crosses over at the Director/VP level, and comes back down to your manager.

This kind of advocacy is powerful because it is technical, not political. It is hard to fake. When you are a steward of a critical system, your promotion packet is filled with testimonials from the most respected engineers in the company saying, “This person’s work enabled our success”.

Utility Ledger

While product teams might be poring over daily active users or revenue, we rely on metrics tracking engineering health:

Utility: Every bug fixed using our tools is an engineer finding us useful. It is the purest measure of utility.
Criticality: If the Pixel team uses Perfetto to debug a launch-blocking stutter, or Chrome uses it to fix a memory leak, our impact is implicitly tied to their success.
Ubiquity: Capturing a significant percentage of the engineering population proves you’ve created a technical “lingua franca”. This becomes especially obvious when you see disconnected parts of the company collaborating with each other, using shared Perfetto traces as a “reference everyone understands”.
Scale: Ingesting petabytes of data or processing billions of traces proves architectural resilience better than any design doc.

When you combine Criticality (VIP teams need this) with Utility (bugs are being fixed), you create a promotion case that is immune to executive reorganizations.

Archetypes and Agency

Staff Archetypes

I am far from the first to notice the idea of “there are multiple ways to be a staff software engineer”. In his book Staff Engineer, Will Larson categorizes Staff-plus engineers into four distinct archetypes.

Sean describes the Solver or the Right Hand: engineers who act as agents of executive will, dropping into fires and moving on once the problem is stabilized. I am describing the Architect or the Tech Lead: roles defined by long-term ownership of a specific domain and deep technical context.

The “Luck” Rebuttal

I can hear the criticism already: “You just got lucky finding your team. Most of us don’t have that luxury.”

There are two caveats to all my advice in this post. First, the strategy I have employed so far requires a company profitable enough to sustain long-term infrastructure. This path generally does not exist in startups or early growth companies; it is optimized for Big Tech.

Second, luck does play a role in landing on a good team. It is very hard to accurately evaluate team and company culture from the outside. But while finding the team might have involved luck, staying there for almost a decade was a choice.

And, at least in my experience, my team is not particularly special: I can name five other teams in Android alone ⁴. Sure, they might have a director change here or a VP change there, but the core mission and the engineering team remained stable.

The reason these teams seem rare is not that they don’t exist, but that they are often ignored. Because they don’t offer the rapid, visible “wins” of a product launch nor are they working on the “shiny cool features”, they attract less competition. If you are motivated by “shipping to billions of users” or seeing your friends and family use something you built, you won’t find that satisfaction here. That is the price of admission.

But if you want to build long-term systems and are willing to trade external validation for deep technical ownership, you just need to look behind the curtain.

Conclusion

The tech industry loves to tell you to move fast. But there is another path. It is a path where leverage comes from depth, patience, and the quiet satisfaction of building the foundation that others stand on.

You don’t have to chase the spotlight to have a meaningful, high-impact career at a big company. Sometimes, the most ambitious thing you can do is stay put, dig in, and build something that lasts. To sit with a problem space for years until you understand it well enough to build a Bigtrace.

By product team I don’t mean “frontend team”: even as a backend engineer, you are still working on some part of what is being served directly to end users. ↩︎
This is not exhaustive, Perfetto is open source and we do also care about external developers but that’s not why we get paid. From the company perspective, time we spent on open source bugs is “wasted” time but we do it because we believe in the mission of open source. I talked about this more in a recent post, On Perfetto, Open Source, and Company Priorities. ↩︎
For what it’s worth, LLMs might not even be the best solution to “let’s put AI into Perfetto”: in my opinion there is lots of value with “old school” machine learning techniques like neural networks. A lot of trace analysis is just pattern matching. This is something I’m hoping to explore more in the coming year! ↩︎
Android Kernel, Android System Health, Android Runtime, Android Camera HAL, Android Bionic ↩︎