On Perfetto, Open Source, and Company Priorities
I recently stumbled across this post on lobste.rs about a project called traceboot which allows visualizing the Linux boot process using lightweight ftrace events and Perfetto. The author had some commentary about their experience trying to order tracks in Perfetto:
Ordering tracks with perfetto has been ridiculously complicated. It has taken the majority of the time of this project! Upstream’s answers are basically that the main user is Android (Perfetto is a Google project) so others come second if at all. While I get the reasons to do so, I read that as a caution against depending on it as a third-party. Google is notorious for (…) completely killing projects
Honestly? All of these points are right:
- It is really unfortunate that doing something so simple took so much effort.
- It’s true that external users are supported at a lower priority than Android users.
- It’s also the case that Google has historically wound down projects when priorities shift
The good news is that we just landed support for trace writers to specify explicitly how traces should be ordered with the JSON format in Perfetto without any extreme workarounds! This feature is already available on the “Canary” UI channel and on “Stable” within 3-4 weeks.
process_sort_index and thread_sort_index are now respected by Perfetto!
So we solved the imediate problem! But here’s what concerns me: I found this issue totally by chance while browsing lobste.rs. There were several issues filed in the Perfetto repo over several years asking for this feature (#555, #764, #378) but none of them managed to communicate to us just how painful this issue was and the lengths people were willing to go to work around it. Seeing the annoyance in such stark terms definitely put into perspective the importance of this feature.
You might be thinking: “surely if a feature is requested that much, you should consider it important and work on fixing it”. And for a general open source project, I’d agree with you. But there are some unique challenges which come with being an open source project under the umbrella of a Big Tech company.
It’s all about how the Perfetto team fits into Google. We’re part of the Android Diagnostics organization whose mandate is to build tools and systems to root-cause problems on Android. Performance debugging would be impossible without a system-wide tracing tool like Perfetto (which is why we even exist at all!). But because we are funded by Android, it’s also expected that we prioritise its needs.
However, I, and others in the team, also deeply believe in the mission of open source. We want to build something useful not just for Android engineers inside Google but for the broader community as a whole. That creates a tension we struggle with constantly: we value external users and their use cases, but the reality of our team’s existence means Google’s priorities often win out.
This tension is compounded by the fact that different types of users want fundamentally different things from Perfetto. If you use Perfetto purely as a trace viewer, the expectation is “I write the trace file and the Perfetto UI should display exactly what I want”. But Android OS and app developers expect “Perfetto should intelligently figure out what is most important in my trace and highlight that to me.”
In the early days of Perfetto, we took on a lot of technical debt to meet the immediate needs of Android developers. We built an ‘intelligence’ layer that automatically reordered tracks based on Android-specific heuristics. This was great for our primary users, but it made it immensely difficult to later support features for the ‘pure trace viewer’ use case, like manually ordering tracks via the JSON file, because the Android logic would always override it.
The good news is that over the past 12 months, we’ve spent a lot of time trying to decouple the “core” layer of Perfetto UI from the “intelligence” layer for Android. This separation means we can now preserve the original intent from the trace format (like “this track should be ordered here”) all the way to the UI, rather than having it get lost or overridden by Android-specific heuristics along the way. This allows us to build features like JSON track ordering without creating impossible-to-maintain code that hurts us in the long run.
It’s also understandable that people worry about Google’s project longevity.
Thankfully Perfetto’s core design principles of a) being 100% open source b)
fully client side significantly mitigate these risks. For example, running the
Perfetto UI is as simple as cloning the repo, running
tools/install-build-deps --ui && ui/run-dev-server and navigating to
localhost:10000. The site is entirely static capable of being hosted anywhere.
There is no “backend server” to be taken down, everything runs in your browser
only. So if for some reason, if the Perfetto team was to disappear, it would be
trivially easy to host it yourself or even fork the project completely and start
maintaining it.
With all that said, here are my takeaways from this experience:
- There are still painpoints lurking in issues from the past that external users are still hitting today. We should maybe do a pass over historical bugs and figure out what issues are hiding there.
- Moving forward, we need to be more transparent and communicative about why we’re not prioritising a certain issue. I feel our terse replies about our priorities are giving misleading indications on the value we place on external users. Maybe we should also be more open about what we are prioritizing, even if that means we have to say “we’re not going to be able to do much for external users this quarter”.
- We also need to set up a regular routine for going through our bug tracker and checking “has anything changed which now makes this bug possible” and if so, we should follow up and post on the issue to make everyone aware of this and keep bugs up to date.
Here’s also where community help would be invaluable: if you’re facing an issue with Perfetto, tell us on Github or on our mailing list! Comment on old bugs if one already exists or file new ones if they don’t. Bonus points if you can tell us why the issue is so painful for you and what you’re doing to work around it if it’s not fixed. The more information we have, the better judgement calls we can make on prioritization.