Lalit Maganti (Atom, Tag: Perfetto)

Perfetto v57: fixing PyTorch traces, plus journald logs and an AI skill

2026-07-03T00:23:00+01:00

We just released Perfetto v57 and I wanted to share the new things I’m most excited about. This is something I wanted to do for past releases but I just never quite got round to it. It’s also something I plan on doing more of going forward: there might even be dedicated pieces if I think the feature deserves it!

PyTorch traces reliably work!

What I’m most excited about in this release isn’t a feature but a bugfix. If you used the PyTorch profiler and opened the resulting trace in Perfetto, there was a decent chance some of your events would just not show up; specifically this would happen when these events overlapped each other on a single track.

Technically, PyTorch is in the wrong here. The Chrome Trace Event (JSON) format says duration events on a track have to nest and can’t overlap; if you need overlap, you’re supposed to use async events. chrome://tracing appears to handle them, but its rendering is actually buggy as soon as a trace has real overlaps; people just learned to live with it. So when a bug came in January about overlapping events being broken, I closed it as working-as-intended because I couldn’t see any easy fix from our end.

Then over time, I realized it wasn’t just one person with a weird trace. The bug had affected many different PyTorch traces, and even JAX traces. I even discovered that people were maintaining forks and hosting old versions of Perfetto just to deal with this issue. This is basically Hyrum’s Law: it doesn’t much matter what the spec says, once enough people rely on some behaviour, that behaviour is your contract.

This is also a very similar situation to what I discussed in Perfetto and open source: it was genuinely hard for me to tell just how many people it was affecting at the beginning: if I had understood, I would have dealt with this very differently.

In any case, in v57, Perfetto now imports the overlapping events instead of dropping them, overflowing the extras onto a different line in the same UI track. I do have to credit @i_komarov who helped me understand how big this issue was and to converge on the fix.

Recording systemd journald logs

Perfetto has always tried to give you as much context as possible about what the system is doing and put it all on a single timeline. One missing piece of that on Linux has been in logs from journald; there’s a ton of useful information there but until now it was hard to correlate that with e.g. scheduling or other app level information.

With v57, there’s now a linux.systemd_journald data source that records journald entries straight into the trace, on the same timeline as everything else. You can filter by minimum syslog priority, by SYSLOG_IDENTIFIER, or by systemd unit, so you are not pulling in the entire journal when you only care about one service.

To show it in action, here’s a trace I recorded on my own machine. Alongside a few log lines I generated myself, it happened to capture a real udisks SMART error that fired while I was recording; exactly the kind of thing you’d otherwise miss.

I have to shout out @staticfloat who contributed this feature; thanks a lot for doing this!

An AI skill for recording and analysis

For a while now, people have been building their own AI tooling around Perfetto. SmartPerfetto and perfetto-mcp are two public examples I’ve come across but many teams inside Google have built their versions of these which each handle a subset of Perfetto’s functionality. To us, this is a sign that there’s demand out there to build something upstream so that everyone can benefit.

v57 includes an installable AI skill that works with Claude Code, Codex, Pi, Opencode, Antigravity, and anything else that speaks the Agent Skills format. It explains to the agent how to record traces (Android only for now!) and then query them. It also includes dedicated workflows for debugging Android memory, a specific area of focus for us; we plan on expanding these workflows over time including to Linux.

The skill is just one piece of a much larger AI plan, which I wrote up as an RFC. We will be doing much more in this space in the coming months.

State tracks

State tracks are a new kind of primitive in Perfetto, sitting alongside slices and counters. They represent something that is always in exactly one state: a connection that is open or closed, a device that is on or off, a worker that is idle or busy. You could already model this with slices, by emitting a slice that starts when a state begins and ends when it changes. This works but you have to be careful to emit events exactly at the same timestamps or you end up having gaps.

With a state track you don’t emit start and end events at all. At each point where the state changes, you emit one event that says what the state is now. The previous state ends automatically. Clearing the track marks it idle. This just makes it a lot simpler to manage.

While the simpler emission model is the obvious win, the reason we made it a first-class primitive goes further than that. Because a state track holds exactly one value at any moment, it can never overlap itself, and the trace processor can lean on that guarantee to treat it specially. You can attach arguments that change as the state does. And in ring-buffer traces, where old events get overwritten as the buffer wraps, you can periodically re-emit the current state, so a trace that starts mid-stream still knows what state everything is in.

State tracks are still quite new so there are likely to be some rough edges here, but we think they’ll become an important primitive: because they’re guaranteed never to overlap, you can do things with them you simply couldn’t before.

Querying traces over HTTPS

I recently discovered that quite a few folks have written scripts which download a trace and then run queries against it with the trace_processor CLI. Which made me think: why don’t we just support that natively inside the CLI itself?

So in v57, trace_processor takes an http(s) URL or a ui.perfetto.dev share link directly. It streams the trace and runs your query with no local copy. Here it is against the journald trace from the previous section:

$ trace_processor query \
 "https://lalitm.com/traces/journald-demo.pftrace" \
 "select log_source, count(*) as n from logs where msg is not null group by log_source"

"log_source","n"
"systemd_journald",30

It’s one of those small polish features which you wouldn’t necessarily think of immediately but when you have it, you’d never want to go back.

Lightning round

A few smaller things also landed:

Shift + mouse wheel scrolls the timeline horizontally. A tiny change that removes a daily papercut when you are panning around a wide trace.
The query results grid got a real overhaul. You can sort, reorder, and hide columns, the sidebar collapses, and SQL errors now show up in the grid instead of disappearing.
Adjacent slices that share a boundary render with a 1px gap, so a run of back-to-back slices reads as separate slices rather than one solid block.
Saved textproto record configs are editable, saving the trouble of downloading the config, deleting it, editing it locally, and then reuploading.

Wrap up

Overall this was a pretty solid release of Perfetto; it’s always satisfying to fix a bug blocking a lot of people. AI skills and state tracks are both first steps in a much longer journey, and journald logs are an important new capability that improves our Linux support. The rest are mostly a bunch of small quality of life improvements which are always nice to see!

Security analysis is finally reaching software's long tail

2026-06-07T16:26:00+01:00

This post was formerly titled “17 bugs in 10 weeks from AI security scanning”. It was changed upon feedback from trusted readers who suggested, while accurate, it did not give a good sense of what I’m actually arguing in the post.

Over the last several weeks, I’ve been receiving more security bug reports for Perfetto’s trace processor than I ever have before, all of them found by AI. And I’m very happy about it! These are bugs that would almost certainly not have been found a year ago and it feels good to close these loopholes even though trace processor is by no means security critical.

For years, security researchers concentrated their time on the highest-stakes targets: kernels, cryptography libraries, password managers. But there’s a lot of code out there which is security-relevant but not truly security-critical. In my experience, these sorts of projects didn’t draw much attention. Now systems in the long tail can get that attention which they wouldn’t have before.

Why is this happening

Trace processor is a project which sits squarely in that long tail. It’s a C++ library (yes, Rust would be the obvious choice today but it’s not practical to rewrite, see footnote ¹) for processing recorded traces of various formats. These are typically traces you collected yourself or in your test infra and process offline so “untrusted input” isn’t much of a concern.

However, some people do process traces they didn’t collect themselves (e.g. user bug reports, automated collection from dogfood users). For those cases we’ve strongly recommended sandboxing trace processor (e.g. gvisor, sandbox2, or minijail) or, for even more sensitive use cases, a VM.

Beyond sandboxing, for catching issues proactively, we mainly relied on fuzzing running internally in Google. These fuzzers occasionally surfaced real, actionable bugs: we set them up to pass in arbitrary trace bytes (as this is the main “attack surface”) but over time these became quite rare as they discovered much of the low hanging fruit, which we quickly fixed. The bugs that remained tend to live deep in the internals, reachable only with a very precisely crafted sequence of bytes that a fuzzer is unlikely to hit by mutation alone.

Apart from this, there has rarely been any bandwidth or resources for a human, either a security expert or someone from my team, to spend lots of time finding security issues in trace processor. There were always other parts of Perfetto more worth spending security time on (e.g. the tracing service, on-device profilers) as they’re actively running in production systems.

All of this changed as of a couple of months ago. We started receiving bug reports filed by some central team which appears to be running AI-based security scanning against various projects throughout Google. Unfortunately, I have to be hand wavy about what exactly they’re doing as their work doesn’t appear to be public.

Starting in early April, we had a slow drip of 1 bug a week, but since the end of April this increased to a rate of several a week, with some days having 3 or 4 being opened in quick succession. This lasted until mid-May, at which point it started tapering back to 1-2 a week with some weeks having none.

I also want to say that the quality of the bugs is high. They’re well-described, often with the relevant attacker model already worked out and even minimal fixes proposed: basically everything I could ask for from a bug report. This matches what both curl and Linux kernel maintainers have noted about security bugs they’ve received, especially how sharply quality has improved in the last few months.

As I can only see the bugs that get filed against me, not the raw output of the AI scanner, I don’t know exactly how much triage happens upstream. My guess is there’s a human doing a light pass to drop obvious noise before reports reach client teams, but judging from the rate at which bugs are opened and the way they’re filed, I doubt anyone is deeply triaging each one.

In total, we’ve received 21 bugs (17 real issues and 4 not actionable), which can be broken down into the following categories:

10 bounds checking: arbitrary trace data flowing into fixed-size buffers or unchecked array indices, leading to out-of-bounds reads or writes.
5 use-after-free: back-pointers, pointer snapshots, or hashmap keys outliving the object they refer to.
1 stack overflow: unbounded recursion when input is deeply nested.
1 access control: not enforcing allowlists on some rare codepaths.
4 closed as not actionable: either where the chance of exploit was purely hypothetical or where fixing would have required fundamental design changes which were not worth the tiny security risk.

All 17 real issues have been fixed, almost all shipping in Perfetto v56.0 ².

How it feels to get a report

How does receiving one of these reports actually feel? Well not as bad as you’d think. Unlike a security-critical application like OpenSSL or curl, in trace processor, a security issue is very unlikely to be a P0 I have to drop everything to fix. Don’t get me wrong, it’s still a priority but one where I have the luxury of taking a few days to figure out the right answer and can release fixes according to our normal schedule, instead of trying to rush out a CVE and get everyone to patch immediately.

Also thankfully, because the majority of the issues are mechanical, the fixes are generally quite straightforward. Take this PR, for example:

We build a key string into a fixed-size stack buffer while parsing some metadata.
The bounds check only runs in debug builds, and the metadata name comes straight from the trace. Putting a long enough name means you would escape the buffer.
The fix is a simple matter of swapping the stack buffer for a std::string. The code path is very cold (only once or twice in a trace) so the extra heap allocation doesn’t matter.

In fact, these sorts of issues are so mechanical that I trust a coding agent to just fix them with minimal guidance: take the well-written report, feed it to the agent, and within ~10 minutes there’s a 10-20 line PR which fixes it. I review every line thoroughly and make sure I understand it, but these tasks are not difficult and firmly inside the “jagged frontier” of what AI can do.

I want to stress though that not every issue is mechanical or can be left to AI; a few reports actually point more to design problems than incorrect function implementations. This use after free is a good example:

The problem was a state object held a back-pointer that could end up pointing to freed memory given certain data appearing in the trace.
The immediate dangling case was easy to patch by just having a callback which invalidated the back-pointer on free. But this is a horrible hack which makes the lifetimes of the objects involved impossible to reason about.
The real problem here is that you had a child object whose parent could go away before it, which really shouldn’t happen if this code is properly architected.
Fixing it properly meant restructuring the ownership model so the lifetime was correct by construction.

The interesting thing was that this was a problem I was aware of and that I had been meaning to clean up for close to a year but never got round to: the security bug just gave me the push and justification to do it. This applied in a couple of other bugs as well and made me internalize that security issues can sometimes be correlated with deeper design flaws or hacky code so there are wider benefits to “security scanning” than just the direct bugs they find.

Will this last?

One thing I am wary of is how long this stream of bugs will keep up; I’m feeling good about it given it’s only been going on for a couple of months, but I can easily imagine that if this goes on for several more months, it might become mentally exhausting.

But my suspicion is that this will go to zero. Why? It’s to do with the pattern of how these bugs are being filed. Each part of the codebase seems like it’s getting a day or two of attention (and associated bugs) before moving on to a different part. Repeats are rare, and the pace of bugs has slowed especially in the last couple of weeks: we had a lot more in the start of May (several a week) but now we’re down to 1-2 a week. There are a finite number of files, so eventually my gut tells me they will run out.

An important consideration is whether we’ll add new bugs faster than the scanner can find old ones. My suspicion is no; the 17 real issues so far are from scanning across 9 years of development. Even if that number triples before things settle, the scanner is still working through years of accumulated code. And we wrote a lot more code, a lot faster, in the earlier years of the project, so the rate of new code being added now is lower than it once was.

The other question is whether new model releases will find more complex design issues rather than the simple issues we’re finding today. Those take significantly more time and effort to fix and so would be a lot more painful if we were to get many of those. I’m very unsure on this so we’ll just have to wait and see!

Where this leaves us

I feel Daniel Stenberg (curl maintainer) phrased it well in this post:

Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools.

This rings very true to me. More broadly, I think folks will have one of three experiences:

Untrusted input + security critical (e.g. curl, kernel, OpenSSL): many complex reports, with a higher false positive rate than the other categories, because there’s a lot of attention on the project and much of the low hanging fruit would already have been picked in the critical codepaths. Though codepaths for lesser-used functionality (e.g. legacy drivers) could end up in category 2 instead.
Untrusted input + not previously audited (e.g. trace processor): a wave of mechanical bugs at a manageable pace and low individual stress because the project is not on a security critical code path. This is where both Daniel and I expect AI security scanners to have the most impact.
No untrusted input (internal tools, math libs, anything operating only on trusted data): you probably won’t notice this shift at all.

My own case sits squarely in that second bucket. But I don’t want to over-generalize from my experience, because there are three things that make this manageable for me that wouldn’t be true for everyone: a) I’m paid to maintain trace processor as part of my full time job; b) someone else is taking the effort to run the AI scans and discover the bugs in the first place; c) the reports appear to be lightly filtered by an upstream human reviewer, enough to strip obvious noise but probably not a deep triage.

To me, this points to a gap in the ecosystem: most open-source projects cannot afford to have a dedicated team doing security scanning for them, and telling a maintainer to stand up their own pipeline when their security risk is marginal will restrict this to only the most motivated projects. I would guess we’re going to see a lot more innovation in this space, including from the big AI labs.

All in all, I’m cautiously positive about my own experience: most of the bugs are mechanical, a few have nudged long-overdue design cleanups, and the pace is manageable. There’s plenty I don’t know about how this evolves: whether the pace holds, whether future models start finding harder design issues. So this should very much be treated as a snapshot, not a forecast!

A common response I expect is: “if it’s parsing arbitrary binary data like traces, it should be written in Rust.” In a vacuum I agree and if I was writing trace processor from scratch today, I would definitely use Rust. But switching to Rust is unfortunately quite impractical; the library is a significant amount of code and is embedded in hundreds of downstream tools, many in environments that don’t have a Rust toolchain. Asking all our embedders to start using Rust would be a significant burden and one I don’t want to impose. Not to mention that our team doesn’t actually have any Rust expertise so reviewing this code at a standard I want from trace processor would be a significant productivity hit until folks got up to speed. And unlike others in the industry, I don’t feel comfortable just rewriting the whole project in one shot and calling it a day… ↩︎
The couple of remaining bugs were found after the v56.0 release was cut and are low-priority enough that it’s not worth rushing out a release for them. They have already been fixed on main and will be fixed in an upcoming point release. ↩︎

syntaqlite: high-fidelity devtools that SQLite deserves

2026-03-17T05:35:45Z

Most SQL tools treat SQLite as a “flavor” of a generic SQL parser. They approximate the language, which means they break on SQLite-exclusive features like virtual tables, miss syntax like UPSERT, and ignore the 22 compile-time flags that change the syntax SQLite accepts.

So I built syntaqlite: an open-source parser, formatter, validator, and LSP built directly on SQLite’s own Lemon-generated grammar. It sees SQL exactly how SQLite sees it, no matter which version of SQLite you’re using or which feature flags you compiled with.

It ships as a CLI, VS Code extension, Claude Code LSP plugin, and C/Rust libraries.

There’s also a web playground which you can try now: paste any SQLite SQL and see parsing, formatting, and validation live in the browser, no install needed. Full documentation is available here.

Here’s syntaqlite in action:

Formatting with the CLI

> syntaqlite fmt -e "select u.name,u.email,count(e.id) as events from users u join events e on e.user_id=u.id where u.signed_up_at>=date('now','-30 days') group by u.name,u.email having count(e.id)>10 order by events desc"

SELECT u.name, u.email, count(e.id) AS events
FROM users AS u
JOIN events AS e ON e.user_id = u.id
WHERE
 u.signed_up_at >= date('now', '-30 days')
GROUP BY
 u.name,
 u.email
HAVING
 count(e.id) > 10
ORDER BY
 events DESC;

Validation with the CLI

> syntaqlite --sqlite-version 3.37.0 validate \
 -e "SELECT json_extract(data, '$.name') FROM events;"

error: unknown function 'json_extract'
 --> <expression>:1:8
 |
1 | SELECT json_extract(data, '$.name') FROM events;
 | ^~~~~~~~~~~~

LSP support in VSCode

LSP in Claude Code

Why it exists

I maintain PerfettoSQL, a dialect of SQLite SQL used for trace analysis in Perfetto. Across Google there are 200K+ lines of the language. At that scale, you hit every corner of it. I tried every SQLite tool I could find and time after time I was disappointed: false positives on valid syntax, misparsed statements, formatters that silently corrupted the SQL by changing semantics. It always came down to a hand-written or generic grammar approximating SQLite rather than matching it to the letter.

Moreover, SQLite’s SQL is not one fixed language. Apart from flags that change the syntax, there are another 12 flags that gate built-in functions, as well as APIs for programs to register custom functions and virtual tables. In Perfetto, we make extensive use of both virtual tables and functions and most tools just don’t give us the flexibility to treat these as first-class extensions of the language.

The SQLite language also constantly evolves: in the last several years, 3.25.0 added window functions, 3.35.0 added RETURNING, 3.38.0 added built-in JSON functions to the amalgamation. And because SQLite is embedded, you can’t assume everyone is on the latest version. This is certainly the case for Perfetto: people link against different versions of SQLite and we need to make sure our standard library is resilient to that.

Now with all of this said, I could have built all these tools just for PerfettoSQL and called it a day. But the more I thought about it, the more I felt that the difference between “good tooling for PerfettoSQL” and “good tooling for SQLite” is so small that it’s worth solving for the wider use case. SQLite is ubiquitous, yet it still lacks the quality tooling other languages have had for over a decade.

Android is one case I think could especially benefit. SQLite is integral to almost every Android app not to mention the platform itself. But the ecosystem is also famously fragmented: if you’re an Android app developer, you need to reason about older platform versions, which in turn means older versions of SQLite.

As an example, Android 13 ships with SQLite 3.32 (link). If you use the RETURNING keyword and your users are on Android 13, it will fail at runtime even though it passes fine on your test device. Wouldn’t it be nice to statically verify this on your host instead of discovering it on a real device?

No tool I found handled all of this: the precise SQLite grammar, version matching, compile flags, dialect extensions. And that’s exactly why syntaqlite exists.

Highlights

Validate semantics with great error messages

Parsing alone is table stakes. But I also wanted to catch common mistakes like reaching for user_name when the column is actually username. For this reason, syntaqlite also resolves table, column, and function references, catching errors like misspelled column names or references to tables that don’t exist.

I’ve also tried to make the error messages really good: at the quality level you would expect to see from modern tools in other languages like rustc or tsc. This means both accurate, line/column level indication with snippets of the error but also “did you mean” hints, pointing you in the right direction. The error messages should be significantly better than SQLite’s and it’s something I plan on making even better with time.

As for correctness, I’ve tested syntaqlite against ~396K statements from SQLite’s upstream test suite with ~99.7% agreement on parse acceptance ¹. Before we get to 1.0, I fully expect this to reach 100%.

There’s also a detailed comparison against other tools covering parser accuracy, formatter correctness, validator quality, and performance. Of course the benchmarks and stress tests were written by me, so take it with a grain of salt. But hopefully the results speak for themselves.

LSP for editors and coding agents

syntaqlite ships a full language server: completions, hover, go-to-definition, find references, rename, diagnostics, and formatting. Unlike most languages, SQL queries depend on an external schema that isn’t in the source file. The LSP needs to know about that schema to give you useful diagnostics.

The most common way to provide this is with a syntaqlite.toml at the root of your project ². Point it at your CREATE TABLE definitions (which you can dump using .schema on your SQLite database; see the project setup guide for details) and syntaqlite uses those for autocompletion, diagnostics, and find references.

[schemas]
"**/*.sql" = ["db/schema.sql"]

This tells syntaqlite: for every .sql file in the project, validate against the definitions in db/schema.sql. You can point at multiple schema files, and scope different schemas to different directories.

syntaqlite.toml can be used to configure other preferences like indentation for formatting and the SQLite version as well. The idea is that you check this into your repo and the whole team gets the same experience.

It validates SQL inside other languages (experimental)

SQL is unusual in that it often lives inside other languages: string literals in Python, template literals in TypeScript, raw strings in C++, and so on.

As an experimental feature (and with caveats!), syntaqlite can extract and validate SQL inside other languages, even if you have string interpolation. For example, in a Python f-string:

# app.py
def get_user_stats(user_id: int):
 return conn.execute(
 f"SELECT name, ROUDN(score, 2) FROM users WHERE id = {user_id}"
 )

syntaqlite validate --experimental-lang python app.py

warning: unknown function 'ROUDN'
 --> app.py:4:23
 |
4 | f"SELECT name, ROUDN(score, 2) FROM users WHERE id = {user_id}"
 | ^~~~~
 = help: did you mean 'round'?

It handles the {user_id} interpolation hole and still catches the function typo with a did-you-mean suggestion.

And more

SQLite version- and flag-aware. Tell syntaqlite which SQLite version and compile-time flags your target has; the formatter, validator, and LSP all respect them.
Custom dialect support. Define grammar, formatting, and validation rules, compile to a shared library, load at runtime. I built this for PerfettoSQL, but it works for any engine that extends SQLite’s syntax. For example, a mobile analytics engine with custom aggregation functions or a local-first app framework with its own replication primitives.
Fast. The formatter processes 3,500 lines of SQL in ~5ms. Full benchmarks on the comparison page.

How it works

I plan on writing a detailed followup on the technical side, but I did want to give a short summary here.

Fundamentally syntaqlite is a C/Rust/C sandwich:

+-------------+---------------+--------------+-------+--------------------+
| VS Code | Claude Code | Playground | CLI | C bindings |
| Extension | / Agents | (WASM) | | (Go, Kotlin...) |
+-------------+---------------+--------------+-------+--------------------+
| Language Server (LSP) |
+-------------------------------------------------------------------------+
| Formatter · Validator · Analyzer (Rust) |
+-------------------------------------------------------------------------+
| Parser · Tokenizer · AST Arena (C, from SQLite) |
+-------------------------------------------------------------------------+

The bottom layer is C: SQLite’s Lemon-generated grammar and tokenizer, plus an arena storing a super flat AST. This layer has no Rust dependency and can be used entirely on its own, which matters if you’re building a database engine or other tool in C/C++ that just needs to parse SQLite SQL ³.

The middle layer is Rust. It reads the C arena directly with no copying or deserialization. The formatter generates a Wadler-Lindig document using an AST-walking interpreter; the semantic analyzer walks the same AST, resolving references and producing diagnostics with byte-accurate source locations.

The top layer is C again, completing the sandwich: the Rust tools are exported back through FFI so C/C++ projects can link syntaqlite as a plain C library. This also makes bindings for other languages possible: Go, Kotlin/Java, and Python are planned before 1.0. Of course, if you’re using Rust directly you can just use the Rust library, skipping the top C layer.

The obvious question I’m expecting is “why not just write everything in Rust”? I wanted the parser to work exactly as SQLite does, which meant literally extracting SQLite’s own code and adapting it to my needs. And writing it in C with the same constraints as SQLite itself (no dependencies beyond the C standard library) means it can be linked into practically anything: database engines, C++ projects like Perfetto’s trace processor, or any environment where SQLite runs.

On the other hand, I really did not want to write a formatter, semantic analyzer, and LSP in C. While I do like the “simplicity” of C, writing a full JSON-RPC server and protocol sounds like a nightmare.

Where this is going

In my head, there’s a clear roadmap for the project side:

100% upstream parity. The ~0.3% gap against SQLite’s test suite is mostly from not fully tracking built-in tables and virtual tables. It just needs another round of “source code introspection” across all the SQLite versions and integrating those tables into syntaqlite’s builtin catalog.
LSP enhancements. Code actions, document symbols, inlay hints for inferred column types. All of these basically making the editor experience even better.
Robust embedded SQL. The Python/TypeScript support is really just hacked together with regexes as a proof of concept. My goal is to extend this to use proper parser libraries and add support for more languages.
Polish and performance. The focus for 0.1 was getting the architecture right. There’s room to improve performance, API surface, and documentation across the whole stack.

As for the blog, this project was the reason I’ve been completely silent over the last month, but now I’m excited to start writing a lot more again. For one, there’s the technical followup for the decisions I took and what I learned along the way.

For another, I used coding agents (mainly Claude Code) extensively while building syntaqlite. There were plenty of moments where following them blindly would have sunk the project, but I can also say with certainty that syntaqlite would not exist without them. I want to dive into that tension: what worked, what didn’t, and the complicated feelings I came away with.

So if you work with SQLite SQL at any scale, I’d appreciate you giving syntaqlite a try and telling me what breaks! SQLite is a project which famously plans to be supported until 2050. My hope is that syntaqlite will be there as a companion every step of the way, making the experience of writing SQLite SQL that much nicer.

syntaqlite on GitHub | Docs | Playground

Per-statement results are in tests/upstream_baselines/. The remaining 0.3% is largely down to not perfectly matching the builtin tables and then edge cases around where and when column aliases come into scope: it’s actually really complicated! ↩︎
You can also pass schema files directly via --schema on the command line without needing a config file. ↩︎
The plan is to replace the PerfettoSQL parser in Perfetto’s trace processor with this C layer. ↩︎

Rendering 100k trace events faster with exponential search

2026-02-03T21:07:04Z

We’ve recently been looking into optimizing rendering performance of the Perfetto UI on large traces. We discovered that there was some inefficiency in our data fetching logic, especially when you’re very zoomed out.

In this case, there can be a lot of slices (spans) which are so small that they take less than one pixel of width. So for each pixel, we need to figure out “what is the event which we should draw for this pixel”. Over time we’ve come to the conclusion that the best thing to draw is the slice with the largest duration in that pixel.

We can break this into two sub-problems:

What is the range of events which correspond to each pixel?
What is the event with the maximum duration for that pixel?

We’re going to focus on 1) in this post as that’s where the slowdown was. 2) is fascinating but also surprisingly orthogonal. If you’re interested, I would suggest reading this excellent post from Tristan Hume explaining the basic algorithm we use.

So let’s formalize our setup for 1): we have a sorted array containing N (where N is O(100k)) int64 timestamps. The timestamps come directly from the Perfetto trace file so we cannot make any assumptions on values or the distribution. But generally speaking, traces tend to have a mix of dense clusters of events (i.e. lot of activity) and large gaps between these clusters (i.e. not much going on). So it makes sense to optimize for this sort of data pattern.

We then need to find the indexes corresponding to the “lower bound” of M regularly-spaced timestamps (A, A+S, A+2S… B), where each corresponds to a pixel boundary and S corresponds to the “time in the trace represented by one pixel”.

Data: | | | · · | | | | | | | | | | · · · | | · · · · | | | | | | |
 dense sparse dense

Queries: ↓ ↓ ↓ ↓ ↓
 A A+S A+2S A+3S B
 │ │ │ │ │
 ▼ ▼ ▼ ▼ ▼
Result: lb(A) lb(A+S) lb(A+2S) lb(A+3S) lb(B)

The obvious solution, and what we’ve been doing for the last few years, is to just binary search each of the M values independently giving us O(M × log N). This is very fast when N is reasonable but when we get into O(100k) on high resolution screens (meaning M is also relatively large), you end up taking several milliseconds to do these queries.

My first approach was to use “static search trees” to speed up the searches; ever since I had come across this amazing post last year, I had been itching to find a place to use them. They worked great and definitely made things faster, but it was a lot of code and required hand-written AVX and a deep understanding, which I knew would be difficult to get through review. And I had this nagging feeling there was something much simpler I could do.

Eventually I had a key realization: I wasn’t using the fact that the queries themselves are sorted! By using this, we can do much better and achieve O(M + log(N)) performance instead!

How? Using a cool technique called exponential search!

You start by binary searching for A just like before giving you an index I. But for every timestamp after that, instead of binary searching, you do something which looks like a hybrid between linear search and binary search.

Found A at index i, now searching for A+S:

Index: i i+1 i+2 i+3 i+4 i+5 i+6 i+7 i+8
Data: [A] [ ] [ ] [ ] [ ] [A+S] [ ] [ ] [ ]
 ↑ ↑ ↑ ↑ ↑
 start +1 +2 +4 +8
 <A+S <A+S <A+S >A+S
 overshot!

 └───────────────────────┘
 binary search here

Take A + S: at I+1, I+2, I+4, I+8…, you check if the timestamp at that position is greater than A + S. If not, you continue to the next one. If it is, then you stop and instead binary search between the previous position you checked and the current one.

This is much faster than just binary searching both theoretically and practically. Theoretically, notice how we scan the same memory at most twice; once for the exponential scan, once in the binary search afterwards. This is unlike the “binary search everything” approach where the same piece of memory could be touched M times, if the data layout was pathological. This is where the O(M × log N) -> O(M + log N) improvement comes from.

Note that this also applies to static search trees which also have O(M × log N) complexity; exponential search turned out to beat them too!

Practically, the initial exponential scan is great for CPUs which love scanning through memory linearly instead of jumping around randomly like binary search does. By significantly reducing the binary search range, you significantly reduce the chance of cache misses.

A final micro-optimization you can do is, when the number of indices you are binary searching is fewer than 16 (empirically chosen), you can just replace with a linear scan instead. At those sizes, you find that the branching and looping from binary search is actually slower than brute force checking everything (especially with the auto-vectorizer helping you out!).

We switched to exponential search in this Perfetto PR and in our microbenchmark (on real trace data!), we saw an improvement from 1.5ms to 180us - a speedup of 8x! This, along with a bunch of other improvements we made, should help make the Perfetto UI faster and smoother on large traces!

Perfetto: Swiss Army Knife for Linux Client Tracing

2025-10-31T00:00:00Z

I gave a talk at the 2025 Tracing Summit last month titled “Perfetto: The Swiss Army Knife of Linux Client/Embedded Tracing”. My goal in this talk was to show how Linux kernel, systems and embedded developers can use Perfetto when debugging and root-causing performance issues in their respective domains. Even though the Perfetto UI is primarily built for viewing Android or Chrome traces, it is a flexible tool and can be used in many other ways!

The talk was recorded and is available on YouTube. Taking inspiration from Simon Willison, this post is an annotated presentation containing my slides and detailed notes on them. The talk also has a lot of UI demos: for these, I’ll have a screenshot but also a link to the relevant part of the video (videos are unbeatable for UI!).

First, what is Perfetto? Perfetto is fundamentally a suite of tools: it’s not just one thing but a bunch of different tools working together to help you debug and root-cause problems. This diagram shows how everything fits together, with the core parts of the Perfetto project in the center.

The recording tools for Perfetto consist of 1) an SDK for C++ apps 2) a daemon that can collect data from ftrace, /proc, /sys, and various kernel interfaces 3) another daemon that amalgamates trace data from multiple processes into a single trace file. These tools all speak the Perfetto protobuf format, a high-performance trace format designed to be very efficient to write but not to analyze or consume directly.

That’s where the trace processor comes in. It’s a C++ library that parses the protobuf format, but also a bunch of other widely used trace formats. It exposes this data via an SQL query interface to any embedding program including Perfetto UI (which is what most of this talk is about) and also to the Python API if you want to do ad-hoc scripting or analysis in Python.

There are also very common tracing/profiling formats used by the Linux community: perf.data, ftrace text format, Firefox profiler format, and many others. Perfetto supports quite a few of those directly. There’s also the Chrome JSON format (AKA the Trace Event Format) which is a simpler interchange format. It’s not the most efficient to read or write, but it does the job for a lot of use cases.

Often people write converters. They have their own tracing format, maybe proprietary, maybe open source but something we don’t know about, and it’s very common that people convert to one of the formats we understand, most commonly our protobuf or Chrome JSON formats.

The Perfetto UI is fundamentally a web-based trace visualizer, combining timeline visualization, user-driven selection/aggregation, and SQL queries all in one interface. Because it has the trace processor as a backend, it works with a bunch of different trace formats.

It’s very important to note that even though the Perfetto UI is web-based, everything happens inside your browser and trace data never leaves your system. You can even build it and host it yourself on any static server: we’ve made it extremely easy to do so!

At the start of 2025, we actually moved our whole development to GitHub. In the past, we used to develop on Android and GitHub was just a mirror. That’s no longer the case, GitHub is actually where we develop and take pull requests.

Most of this talk, I’m going to spend actually showing you how you can use the Perfetto UI to debug performance issues on Linux. I don’t want to show you an Android trace which needs a lot of context about how the Android system works and so you think, “oh, that was cool, but I didn’t really understand what was happening.”

So to make this talk more approachable, I wrote a straightforward demo program you can look at yourself! So it’s obviously not a production system but I’ve tried to make it as representative of the sort of issues we use Perfetto for every day.

It’s a Rust program which generates a Julia set and visualizes it over time. The technologies I used: Vulkan, GPU rendering and also multi-threaded CPU computation. So how it works is that computation of various parameters is happening on background threads, and then that’s being passed to the main thread for rendering.

And then, for demonstration purposes, there is a performance bug; rendering should run at 60 FPS, but every so often, the frame rate drops dramatically. Here’s what that looks like:

The code is on GitHub and if you’re interested in following along. The traces are there as well - you don’t have to collect the traces yourself, but you can if you want. All the instructions and information is in the README.

So the first suspicion we may have is that maybe it’s some CPU problem. A lot of engineers I know would reach for perf immediately whenever they see a problem like this. The main reason is that if perf can capture the problem, they can go straight to the line of code without needing to spend time debugging using more complex approaches.

You can record a perf profile with perf record -k mono -g ./fractal_renderer. The standard post-processing step which I think a lot of people do would be to generate an SVG flame graph out of this with perf script | stack-collapse-perf.pl --all | flamegraph.pl > fractal-frame.svg. Here’s how that looks like for our perf profile:

See also the interactive version of the profile

The flame graph shows the thread names at the bottom, and then stacked above are the call stacks of what the program is doing. The width represents how much time is spent in each function. In this case, the worker threads are spending most of their time in cos and sin operations, doing math computation. Exactly what you’d expect for this type of program. Aside: Interestingly demangling didn’t seem to work for Rust out of the box which I find a bit unusual but I didn’t dig too much into why this was.

Looking at this, nothing really stands out. There’s no smoking gun that immediately reveals the problem. But here’s the fundamental limitation: the flame graph aggregates data across the entire trace, losing the time dimension. The performance problem we’re seeing happens every 2-3 seconds - brief drops in an otherwise normally functioning program.

The flame graph shows me the aggregate, dominated by the correct behavior, making it nearly impossible to spot those occasional problematic moments. So how do I find the places where it’s doing the wrong thing?

Well, that’s maybe where Perfetto can help you out a little bit! The thing I find lots of people don’t know is that perf actually preserves timestamp information about when samples were taken: many tools drop this information but Perfetto is pretty good at showing that to you. I just need post-process the trace with perf script > fractal.perftext to generate a text version of the profile which we can then open in the Perfetto UI

My demo talking through how to open the perf profile and navigate it starts at 9:11 in the video.

Screenshot from 9:50 in the video

The x-axis of what you’re seeing is time. And every horizontal line represents a thread: we call these lanes “tracks”. And each of the small arrowheads on the tracks are CPU samples for the associated thread.

The behavior of the worker threads over time is really interesting. Most of the lifetime of the program, it’s doing continuous computation, basically. And then there’s this period of time in the middle where there’s this very interesting staircase pattern where it seems like only one thread is running at any one time: we’ll want to keep a note of this as this is quite important.

Screenshot from 10:37 in the video

One very cool thing about Perfetto is that it allows me to quickly generate visualizations of whatever I’m selecting on the timeline. We call this “area selection” and it’s where I drag my mouse and select a region both horizontally and vertically. This is on one track but even across multiple tracks. So in the timeline it shows me the selection I’m making at the top, plus a flame graph at the bottom representing the aggregation of the samples in just that time period.

The key advantage is that I can look at individual regions of time interactively. You can also do this with flamegraph.pl but you need to pre-filter the data to what you’re looking for. This assumes you already know what the problem is before you find it. I always find that a bit counterintuitive. I prefer this view where I can see everything first, then decide “I want to look at this specific part in more detail.” That’s what Perfetto lets me do.

Screenshot from 12:31 in the video

So now on to the flame graph itself: in the last year, I spent a bunch of time on improving the flame graph visualization in Perfetto UI. At Google, we have an internal tool called pprof (related to but not the same as the open-source one), and I’ve always loved the visualization it has. So I worked to make Perfetto’s flame graph look and behave very similarly.

There are a bunch of features here. You can zoom in like most flame graph, but you can also say “I only want to look at the stack starting from this point” — it gets rid of everything above and starts fresh from there. If I don’t want to see a particular function like sin, I can just hide the frame and it gets merged into its parent. The search is regex-based so I can also just type the specific things I’m interested in: it’s pretty powerful.

There’s also something I don’t think I’ve seen in other visualizers (I might be wrong, please do correct me!): what I call “bottom up” view. Imagine you take all the leaves across your entire program and visualize who’s calling into them. If you have a leaf function that’s called everywhere in your program but reached from many different places, it’ll be split across your flame chart. This is the inverse — you’re looking at the leaves and asking “who is calling me?” This is particularly useful when you’re trying to optimize very low-level functions that are always at the leaf level — things like memcpy or hashmap operations.

So our main takeaway is that our worker threads all had gaps in their CPU execution in a staircase pattern. This means the threads weren’t actually on the CPU and that’s usually a sign that they’re sleeping.

But sleeping on what? Locks? Disk? Network? What exactly is happening? To answer that, I need a scheduler trace to show me the wake-up patterns, who’s being scheduled when, what type of sleep threads are in, that sort of thing.

We recently added support for trace-cmd’s text format in Perfetto (aside: there’s an open issue to support the binary format too!). For this demo, I’m collecting sched_switch and sched_waking events with sudo trace-cmd record -e sched:sched_switch -e sched:sched_waking -C mono ./fractal-renderer-vk and we can open it in the UI once we do trace-cmd report -N > fractal.sched.

The demo showing off scheduler traces starts at 15:16 in the video.

Screenshot from 15:49 in the video

So this is what a scheduler trace looks like. Again as before, the x-axis is time. At the top of the trace, you’ve got the CPU scheduling tracks. For each CPU, they tell us what was scheduled on that CPU at each point in time. The white regions are where there’s nothing scheduled. As I zoom in, you start to see information like the name of the thread that’s scheduled, the TID of the thread, and so on. This is the CPU view of things, looking at it from a CPU perspective.

Screenshot from 16:48 in the video

There’s also a thread perspective on scheduling: what we call this “thread state”. This shows the scheduling state of each thread at every point in time. Dark green represents the thread running, light green represents runnable (the thread is on the runqueue so it’s eligible to run, but the scheduler hasn’t actually let it run yet), and white represents interruptible sleep (S sleep). There are also orange regions for uninterruptible sleep (D sleep), though that doesn’t show up much in this trace.

You can again see a very clear staircase-like pattern. Again, only one thread seems to be running at any one time. You’re maybe getting the sense of what the problem might be at this point.

Screenshot from 17:43 in the video

Like with the perf visualization, you can do area selections here too. I can select a region on a CPU and get tables showing time spent by each thread (shown in the screenshot above). I can also do this for thread state, you can also see that runnable time is basically negligible compared to running and sleeping, which is where most of the time is being spent.

Screenshot from 18:12 in the video

There are also scheduler-specific visualizations like run queue length and active CPU count. You can see in the problematic region, the active CPU count also went down, which makes sense.

So the scheduler trace shows when threads are sleeping, but it’s not telling us why. What was the program doing in that region? What code was it running? The sequential pattern suggests some sort of serialization—something is causing only one thread to run at a time. But we need application-level visibility.

Since this program was written in Rust, I just used the off-the-shelf tracing and tracing-perfetto crates; I did have to make some small modifications to tracing-perfetto that I’m hoping to upstream. These output a Perfetto trace without needing to use the Perfetto SDK. These libraries are also maintained by others: we don’t have any hand in them.

All I needed to do was integrate these into my program and then add a command line switch to write out the collected trace to a file. So I just do ./fractal_renderer --trace fractal.pftrace

My demo for app tracing starts at 19:37 in the video.

Screenshot from 20:06 in the video

Looking at the app trace: as before x-axis is time and each of the tracks represents a thread. Instead of scheduling activity or CPU stack samples, this time the tracks is userspace instrumentation of the program itself. Each of these rectangles is a named period of time representing what the program was doing at that time.

There are lots of different names for these in the tracing world (e.g. spans, intervals) but we call these “slices”. The main thread is rendering frames as you would expect and the workers are each computing tiles of that frame, which eventually feed back to the main thread and sync to the GPU.

Screenshot from 23:42 in the video

There’s also this thing called flows, which shows the causal links between different pieces. Render frame is calling into all these places and causing them to happen. It’s sort of similar to wakeup graphs but for user space: basically, this is what caused me to start running this function.

Screenshot from 23:51 in the video

So you can see very clearly there’s a normal region and then a region where things are taking 1.8 seconds: almost certainly the cause of the frame drops.

And if we look at the slice in question, it seems to be doing something called “update adaptive quality.” Basically, I wrote some code to dynamically change the rendering quality based on frame rate. If I’m running faster, I can render at higher quality. If I’m running slower, I can do lower quality. That’s what this adaptive quality thing is supposed to do.

But clearly something has gone wrong. I’m causing frame drops because of updates to adaptive quality. A little bit ironic, to say the least. Now we know what the program is actually doing during that time span.

So now we’ve looked at three different sources of debugging data separately. In perf, we got told there are few or no CPU samples and weird staircase patterns. In ftrace, we saw only one worker seems to be active and the rest are sleeping in interruptible sleep. And in app tracing, we see it’s because of adaptive quality updates that workers are working on, and they shouldn’t be doing it this way.

Obviously, the theory is that it’s all the same problem. But we can confirm this theory with a very new (and still experimental) feature of the UI: trace merging.

Note: as this part of Perfetto is still experimental, if you want to try it yourself, you need to go to plugins and enable the “MultiTraceOpen” plugin. Also, unlike I what say in the talk, you do not need to be on Autopush as all the features I showed in my talk are now available in Stable.

The demo showing off merged traces starts at 24:32 in the video.

Screenshot from 25:44 in the video

This is now the merged trace file containing all the different information. You can see the CPU scheduling tracks like we had in the scheduler trace, the stack samples like we had in the perf trace, and the app instrumentation, all beside each other on one timeline. You can very clearly see the adaptive quality update running, then this period where it ran, and then it was done. It’s very cool to be able to see the pattern of how my program is sleeping and where and what it was running on one timeline.

The thing is, you’ve always been able to do this with Perfetto if you collect a Perfetto trace with all of these integrated. The new capability is that this is now being brought to traces you didn’t collect with Perfetto. Collect it with whatever you like, and we’ll still manage to do this visualization. That’s the new capability we have.

Screenshot from 29:58 in the video

There’s also a query interface which we’ve been building out recently and there are some pretty powerful things you can do with these tables. As well as the flat tables of data, there’s also a more dynamic pivot table and if I click the arrow on the left, I can get into a detailed filter table, similar to a spreadsheet but optimize for things people want to do on traces.

For example, by doing an area select, I can first get a list of all the events that happened during a time region. Then using the table I can filter for things - say I only care about slices longer than some duration, like 319 microseconds for whatever reason. I can click and add a filter for things greater than this.

Screenshot from 30:11 in the video

There’s also a feature called “Show Debug Track” that’s also very powerful. The table shows rows with timestamped duration information, and this feature lets you visualize that data as a track on the timeline. It adds a track at the top showing the regions of time where these events happened—in this case, where ComputeTile took longer than a certain threshold.

This is particularly useful for correlation analysis. For example, in Android, we’re often looking at system A and trying to understand its effect on system B. We find all the places where one thing is slow, then look for correlations. Being able to see a track in the UI where you can quickly have context and say “oh, during this period of time, this other thing was happening” is invaluable. It’s probably one of our most-used features.

You can also copy the SQL query from these tables and start doing your own aggregations on top of it. This eases the burden of starting with a completely blank canvas. Instead of wondering “where do I even start querying my data? What tables is my data in?”, the UI gives you a starting query that you can build on without needing to write something from scratch.

So the dynamic quality updates were stopping the world. Perf profiling showed the problem, scheduler traces found the sleeping pattern, app tracing confirmed it was the quality adjustment code, and the single timeline view let me see everything happening at once.

Interestingly, I tried to fix this by removing the lock contention—which I thought was the bug I had introduced. But it turns out I shouldn’t have been doing this code in the workers in the first place. Even after removing the lock contention, just the CPU activity of doing that work was enough to cause frame drops. The right solution was to move it to a background thread. As part of debugging this for the demo, I discovered something even better that I could be doing.

Android and Chrome are our bread and butter—that’s what we officially support as a team, and why Google staffs us. But there are many other interesting uses.
Mesa uses Perfetto as one of its tracing systems. One thing I could have shown is collecting a Mesa trace alongside all the other traces we looked at—you can actually see what the GPU is doing at the same time, which would have been very cool, but I just didn’t have time.
VizTracer is function tracing for Python, similar to uftrace, but you don’t have to recompile anything or do anything special.
pthread_trace is for visualizing pthread mutex contention. The author has a very efficient way of doing this and writes protobuf using heavy use of constexpr to make it very low overhead. It’s a very interesting project.
magic-trace uses Intel Processor Trace for function-level tracing at the processor level with lower overhead. They wrote a converter from that to the Perfetto format so you can visualize that data in Perfetto.
Qais Yousef’s sched-analyzer enriches Perfetto traces with scheduler internals. It’s a very cool project that I find particularly interesting.
John Stultz’s all-in-one Perfetto recording script for kernel developers addresses a common complaint: “Perfetto is so complicated, I don’t know what events I want to record.” He just wrote a script that configures everything for you.
Finally, Josef Bacik’s systing is a bit experimental but fascinating. He re-implemented tracing daemons on top of BPF instead of Perfetto’s native implementation, combining BPF-based tracing with perf stack traces in a single binary. He has a blog post explaining why. I thought it was a fascinating use of Perfetto.

If you want to try this yourself, you can use the demo program I’ve provided. But even better: record traces on your own programs. Open scheduler traces and perf at the same time. Instrument your program, convert it to Perfetto format, and visualize everything together.

If you want to convert ad-hoc timestamped data to Perfetto, we wrote a comprehensive tutorial with Python snippets for everything you might want to visualize. It covers all the features I showed and how to write Python code to generate them. We have a library for writing these traces: besides this one library, you don’t need to install anything else. You can go ahead and convert your own data to Perfetto.

We’re very happy to accept contributions and review pull requests pretty quickly. We’ve had a lot of open-source contributors over the years and have been quite lucky with some very high-quality contributions. If you want to contribute yourself or have a feature you feel is missing, send it our way.

Demo repo: github.com/LalitMaganti/fractal-renderer-vk
Learn more: docs.perfetto.dev
Perfetto UI: ui.perfetto.dev
Contact: lalitm@google.com

On Perfetto, Open Source, and Company Priorities

2025-10-17T15:00:00+01:00

I recently stumbled across this post on lobste.rs about a project called traceboot which allows visualizing the Linux boot process using lightweight ftrace events and Perfetto. The author had some commentary about their experience trying to order tracks in Perfetto:

Ordering tracks with perfetto has been ridiculously complicated. It has taken the majority of the time of this project! Upstream’s answers are basically that the main user is Android (Perfetto is a Google project) so others come second if at all. While I get the reasons to do so, I read that as a caution against depending on it as a third-party. Google is notorious for (…) completely killing projects

Honestly? All of these points are right:

It is really unfortunate that doing something so simple took so much effort.
It’s true that external users are supported at a lower priority than Android users.
It’s also the case that Google has historically wound down projects when priorities shift

The good news is that we just landed support for trace writers to specify explicitly how traces should be ordered with the JSON format in Perfetto without any extreme workarounds! This feature is already available on the “Canary” UI channel and on “Stable” within 3-4 weeks.

process_sort_index and thread_sort_index are now respected by Perfetto!

So we solved the imediate problem! But here’s what concerns me: I found this issue totally by chance while browsing lobste.rs. There were several issues filed in the Perfetto repo over several years asking for this feature (#555, #764, #378) but none of them managed to communicate to us just how painful this issue was and the lengths people were willing to go to work around it. Seeing the annoyance in such stark terms definitely put into perspective the importance of this feature.

You might be thinking: “surely if a feature is requested that much, you should consider it important and work on fixing it”. And for a general open source project, I’d agree with you. But there are some unique challenges which come with being an open source project under the umbrella of a Big Tech company.

It’s all about how the Perfetto team fits into Google. We’re part of the Android Diagnostics organization whose mandate is to build tools and systems to root-cause problems on Android. Performance debugging would be impossible without a system-wide tracing tool like Perfetto (which is why we even exist at all!). But because we are funded by Android, it’s also expected that we prioritise its needs.

However, I, and others in the team, also deeply believe in the mission of open source. We want to build something useful not just for Android engineers inside Google but for the broader community as a whole. That creates a tension we struggle with constantly: we value external users and their use cases, but the reality of our team’s existence means Google’s priorities often win out.

This tension is compounded by the fact that different types of users want fundamentally different things from Perfetto. If you use Perfetto purely as a trace viewer, the expectation is “I write the trace file and the Perfetto UI should display exactly what I want”. But Android OS and app developers expect “Perfetto should intelligently figure out what is most important in my trace and highlight that to me.”

In the early days of Perfetto, we took on a lot of technical debt to meet the immediate needs of Android developers. We built an ‘intelligence’ layer that automatically reordered tracks based on Android-specific heuristics. This was great for our primary users, but it made it immensely difficult to later support features for the ‘pure trace viewer’ use case, like manually ordering tracks via the JSON file, because the Android logic would always override it.

The good news is that over the past 12 months, we’ve spent a lot of time trying to decouple the “core” layer of Perfetto UI from the “intelligence” layer for Android. This separation means we can now preserve the original intent from the trace format (like “this track should be ordered here”) all the way to the UI, rather than having it get lost or overridden by Android-specific heuristics along the way. This allows us to build features like JSON track ordering without creating impossible-to-maintain code that hurts us in the long run.

It’s also understandable that people worry about Google’s project longevity. Thankfully Perfetto’s core design principles of a) being 100% open source b) fully client side significantly mitigate these risks. For example, running the Perfetto UI is as simple as cloning the repo, running tools/install-build-deps --ui && ui/run-dev-server and navigating to localhost:10000. The site is entirely static capable of being hosted anywhere. There is no “backend server” to be taken down, everything runs in your browser only. So if for some reason, if the Perfetto team was to disappear, it would be trivially easy to host it yourself or even fork the project completely and start maintaining it.

With all that said, here are my takeaways from this experience:

There are still painpoints lurking in issues from the past that external users are still hitting today. We should maybe do a pass over historical bugs and figure out what issues are hiding there.
Moving forward, we need to be more transparent and communicative about why we’re not prioritising a certain issue. I feel our terse replies about our priorities are giving misleading indications on the value we place on external users. Maybe we should also be more open about what we are prioritizing, even if that means we have to say “we’re not going to be able to do much for external users this quarter”.
We also need to set up a regular routine for going through our bug tracker and checking “has anything changed which now makes this bug possible” and if so, we should follow up and post on the issue to make everyone aware of this and keep bugs up to date.

Here’s also where community help would be invaluable: if you’re facing an issue with Perfetto, tell us on Github or on our mailing list! Comment on old bugs if one already exists or file new ones if they don’t. Bonus points if you can tell us why the issue is so painful for you and what you’re doing to work around it if it’s not fixed. The more information we have, the better judgement calls we can make on prioritization.