israelo.io

Coming for Cancer with Quantum-Accelerated Pipelines

Thu, 27 Nov 2025 03:01:35 +0000

Coming for Cancer With Quantum Power

An excerpt from my diary.

This is my next mission, the thing I’m building toward, not just funding research or writing cheques but actually building the infrastructure, the partnerships, and the quantum-accelerated pipelines that will make cancer was.

Cancer took too much from me already. I’m not waiting for it to take more.

My next big mission is to make cancer was. The way smallpox was a global killer. The way polio was a childhood terror. Cancer will be something our grandchildren read about in history books, not something they watch destroy their families.

“Cancer was a terrible disease” –> makes a great PR-FAQ.

Cancer killed my grandmother. It killed my mum. My sister fought it and won, but she lost a part of herself in the process. Thank God she’d finished having kids by then.

This is personal. I have two daughters. Cancer came for three generations of women in my family. My daughters won’t be next.

So I’m coming for it first, with everything I’ve got.

You don’t feel the pain and urgency until it happens to you. It kills slow. It leaves flashbacks that wake you up in the middle of the night with tears running down your cheeks.

I believe the current paradigms around cancer research, diagnostics, and treatment are too politicized and too incremental. If you think otherwise, I respect it, but you don’t feel the urgency if it hasn’t quietly dismantled your world. The world needs urgency, boldness, and radical clarity. And that urgency has to be channeled through advanced technology, especially the rising frontier of quantum computing.

Quantum computing offers a chance to reimagine what’s possible: simulating molecular and cellular dynamics at speeds and scales we’ve never had before; detecting cancer earlier; designing smarter therapies; personalizing treatments in ways old systems can’t approach. Researchers have already used quantum-classical models to design molecules targeting proteins once considered “undruggable.”[1] Quantum computers have solved problems in minutes that would take classical supercomputers thousands of years.[2]

The infrastructure is emerging. Research institutions and technology companies around the world are proving what can be done. But we need more. We need speed. We need people who won’t settle for incremental progress while people die waiting.

I’m coming for it. I believe we can use technology to make cancer was. This is my life’s mission. The results may come long after I’m gone, but I have to fight.

If you’ve got tips, or want to join me in forming an interest group, let’s connect.

References

[1] Aspuru-Guzik, A. et al. (2025). “AI and quantum computing used to target ‘undruggable’ cancer protein.” University of Toronto Faculty of Arts & Science. https://www.artsci.utoronto.ca/news/ai-and-quantum-computing-used-target-undruggable-cancer-protein

[2] “Quantum Computing Cancer Research: Transforming Healthcare.” (2024). Infiniti Research. https://www.infinitiresearch.com/thoughts/quantum-computing-in-cancer-research-a-new-era-of-drug-discovery-and-treatment/

The Hidden Curse of Love

Fri, 10 Oct 2025 03:01:35 +0000

Love feels natural, effortless, and beautiful. Yet hidden within it is a curse that only reveals itself when you lose someone you love deeply.

Put another way, buried deep within love is something we rarely talk about — its hidden cost. The harder you love, the deeper the pain of loss.

I’ve been thinking a lot about my mum. I loved her deeply. I went back to her home in Nigeria this September, the first time since we lost her. It took days before I could walk into her room. Every corner of that house carried her presence. Her voice. Her scent. Her peace. I could almost hear her call my name. It was rough.

I kept asking why the pain felt heavier than the memories. Why grief seemed to drown the love we had. I needed to process this some more, so I pulled out a pen and paper.

A realization occurred. The hardest part wasn’t just losing her. It was her final five months. Cancer is a bastard. It strips away strength and light, leaving only pain. Watching her fade and groan in agony broke something inside me. It replaced my belief in a kind universe with the cold reality of its indifference.

Since then, I’ve been reflecting on love’s complexity. It feels so beautiful and natural when it flows, but it a heavy burden that only reveals itself in loss. Grief can feel heavier than memory, louder than joy. It threatens to overshadow everything that once made love feel safe.

Maybe that’s the truth of it. The same love that brings immense joy also creates a deep void when it’s gone. Grief doesn’t end; it transforms. Sometimes it sits quietly beside you. Other times, it swallows the whole day. I suppose I could define grief as the shadow of a jewel.

The very light that fills our lives can also cast the deepest shadow when it fades. Maybe that’s the hidden curse of love — that to love deeply is to accept the risk of breaking completely. When you sign up to love and care for anyone, you are implicitly signing up to give them pain or to suffer the pain of an endless grief. Either way, we are cursed.

Here’s the thing that worries me thiough, I love my kids so deeply, and I know that they love me. Now that’s a puzzle.

But here’s what I’m learning to hold onto — the curse is also the blessing. Yes, grief is the price of love. Yes, it breaks us. But would I trade away the years with my mum to avoid this pain? Never. Would I love my children less to spare us all future heartbreak? Not for a second.

The weight of grief is simply love with nowhere to go. It’s not the opposite of love; it’s love’s continuation in a different form. And if I must carry this shadow for the rest of my life, then so be it — because it means the light was real. She was real. This love is real.

Maybe we’re not cursed after all. Maybe we’re just human. And being human means choosing to love fully, knowing the cost, and deciding it’s worth it anyway. Every single time.

Yeah, vanity. everything is vanity.

User Journeys Are Dead. Enter the Agentic Journey Era

Tue, 30 Sep 2025 03:01:35 +0000

For years, product managers and founders (myself included) built products around user journeys. We mapped every click, tap, and drop-off point. We obsessed over funnels, session duration, flows, DAU (Daily Active Users), MAU (Monthly Active Users).

That era is ending. The PRD structure must evolve.

The next generation of amazing products won’t be defined by users (painstakingly) clicking or tapping through flows. They’ll be powered by AI agentic systems that understand intent, plan across platforms, and execute on behalf of the user.

The BLUF is: user journeys as we know them are fading, giving way to the agentic journey. When designing for product, you must now consider AI agents and how they achieve outcomes. Go further by factoring in token economics and create AI eval frameworks for measuring success. Traditional metrics like clicks, session duration and active sessions are headed for obsolescence, just like the journeys themselves.

The future belongs to products that orchestrate outcomes, not just user interactions.

Why This Hit Me

Early in my career at AppDynamics (Cisco), I spent nearly all my time onsite with customers, supporting both pre-sales and post-sales activities (depending on the quarter). Sitting shoulder to shoulder, I could see every hesitation, every workaround, every aha moment. That intimacy shaped how I thought about product.

During COVID, when I moved into product management full-time, that contact disappeared. PRDs, metrics, and dashboards replaced real-time user insight. FWIW, user interviews are never the same as watching users interact with your product—their body language, the subtle patterns in how they work—you notice things you can never capture in Zoom meetings, metrics, or dashboards.

Recently, onsite with a zymtrace customer, I felt that old rush again. What struck me wasn’t just how people used the product, but how naturally they leaned on AI agents in their IDEs, CLIs, and workflows. Tasks that once required manual steps were now offloaded to agents. It drove home a truth: the way we build and measure products has to evolve.

The contrast between these experiences—then and now—makes one thing absolutely clear: product design is no longer just about user interaction; it must now include considerations on how AI agents can achieve the intended outcome. We must rethink product success metrics.

What Really Is the Agentic Journey?

Agentic journeys are fundamentally different from traditional user flows. Unlike conventional journeys, they don’t require users to follow predefined paths—they understand the user’s goals from the outset. Agents interpret intent, navigate complexity, and execute tasks on behalf of the user, turning multi-step workflows into seamless outcomes.

When I say agent here, I mean an AI system that, once given a goal, autonomously chooses its next step—selecting and invoking tools, planning or revising its approach, and using real-world feedback (API responses, errors, intermediate state) to iteratively reach an outcome instead of executing a fixed, hardcoded sequence (ref: Anthropic: Building effective agents)

Key characteristics of agentic journeys include:

Fewer apps, fewer tabs — Agents eliminate context-switching by fetching, integrating, and delivering results directly. Users no longer juggle multiple windows; the agent handles it.
Agent-to-agent communication — Agents coordinate with one another to delegate tasks and orchestrate workflows. Workflows become a collaborative mesh of agents, not a linear sequence of clicks.
Adaptive workflows — Agents adjust plans in real-time as goals or data change, rather than following rigid, predefined paths.
Cost per outcome matters — Efficiency is critical. Excessive compute, tokens, or latency can break the journey.

For example, the Agent2Agent (A2A) Protocol, introduced by Google in April 2025, enables secure communication and collaboration between AI agents across different platforms and vendors. It allows them to share context and coordinate actions seamlessly. Additionally, the Agent Payments Protocol (AP2), announced in September 2025, provides a standardized framework for secure, compliant transactions conducted by AI agents on behalf of users.

These protocols illustrate how agentic AI systems are evolving to manage complexity, collaborate autonomously, and perform transactions securely.

Product Metrics Must Change Too

When I was at Elastic (as Principal Product Manager), I managed BI for a while. I spent a lot of time diving into metrics: MAU, DAU, product segmentation, retention curves. We lived and died by understanding how users moved through the product.

If I were to do it all over again, my approach would be different. The questions aren’t “How many people logged in today?” or “Which feature saw the most clicks?” Instead, they become:

Did the agent complete the intended outcome?
How many tokens did it cost to get there?
Did the product provide the right context, quickly and efficiently?

These are outcome-based metrics—they reflect the economic reality of building in an agentic world. Engagement for its own sake doesn’t matter anymore. What matters is efficiency and cost per completed outcome.

To make these metrics actionable, I’d build evals to measure success: automated tests, benchmarks, and simulations that validate whether agents achieve outcomes reliably, efficiently, and safely. This shifts the focus from counting clicks or logins to understanding real impact.

Token Economics and Efficiency

Agents don’t run for free. Every reasoning step, API call, and chunk of text processed burns tokens—and tokens translate directly into dollars.

That means product teams can’t just design for functionality. We need to design for economic viability. How many tokens does it take to reach an outcome? Is the workflow efficient enough to scale?

APIs sit at the center of this. Agents don’t care about a beautiful UI—they care about clean, reliable endpoints. That means:

Expose everything via API — Every product feature available to a user should also be accessible programmatically.
Predictable responses — Agents need consistency, not cleverness.
Minimal payloads — Return only what’s necessary. Bloated responses waste tokens and context.
Composable endpoints — Agents should be able to chain actions without brittle hacks.

Products that ignore these principles will quickly become too expensive to use. Products that design for compression, context, and efficiency will win.

Gartner’s research underscores this point: more than 40% of agentic AI projects are expected to be scrapped by 2027 because costs spiral out of control and business value remains unclear (Reuters). In other words, ignoring token economics isn’t just inefficient—it can be fatal to the viability of entire product lines.

Anthropic’s Model Context Protocol (MCP) is an early signal of what’s coming. It shows how agents can manage multi-step processes across tools in a structured, context-aware way.

This isn’t just a technical shift—it’s a product strategy shift. The ecosystem is moving toward interoperability and agent-first design, where efficiency and cost per outcome drive product decisions.

What You Must Now Do

If you’re building or leading product today, here’s where I’d start:

Audit your APIs — Can an agent realistically complete your core use case?
Re-examine pricing and unit economics — Factor in token costs and operational efficiency.
Position your product as part of an agent-driven ecosystem — Don’t treat it as a standalone UI.
Train your team to think in “agentic journeys” — Go beyond traditional user flows.
Redefine your metrics — Measure outcomes and token efficiency, not just logins and clicks.

Closing Thought

Watching customers use zymtrace in real time reminded me that building product is never static.

We must shift, albeit gradually, from product journeys designed for people to journeys carried out by AI agents. It’s a new model, and if we don’t adapt, our products won’t just feel clunky; they’ll risk becoming irrelevant.

The next time you discuss product user journeys with your peers or team, ask: What does the agentic journey look like, and how do we build evals to measure its success?

Code & Communication: The Keys My 8-Year-Old Uses to Design Apps

Tue, 03 Sep 2024 22:01:35 +0000

“The success or failure of a product strongly depends on the user experience”, By Rochelle Ogbole

I’m constantly amazed by my daughter Rochelle, and today was no exception 🤩. At just 8 years old, she’s already thinking up ideas for mobile apps, explaining them with a level of clarity that could rival a seasoned developer. But maybe I shouldn’t be too surprised—after all, she’s been coding on Scratch since she was 4! 👩‍💻

Recently, Rochelle came up with a brilliant concept: a kid-friendly mobile app that serves as a creativity canvas for children. Her idea? To unlock kids’ imaginations and bring their creativity to life in a fun, interactive way. We discussed what motivates her to create this app, and I was blown away by how well-thought-out her vision was 😲. It was like she had been planning this for years.

I didn’t expect the Twitter to X reaction

She wants kids to be able to share their creations on social media as a feature. The look on her face when I told her that Twitter is now called X was priceless—she was genuinely surprised. “Why would they do that? What does ‘X’ even mean?” she asked. Then she added, “Well, I’m going to stick with Twitter, Daddy.” I’ll respect her opinion on that!

Watch the trailer for yourself.

Wireframing Her Vision

To start the process, we wireframed her app on paper ✏️. Rochelle took charge, sketching out how the app would look and function. I then suggested she take the next step and design it using Figma 🖥️. I wanted her to get feedback from her friends and their parents, curious to see if they would pay for such an app 💡.

I created a Figma account for her, and then I headed off to the gym 🏋️‍♂️, giving her the space to experiment on her own. What happened next truly floored me 🤯. After just 1.5 hours, I returned to find that Rochelle had completely designed the app on Figma, and it looked fantastic! What’s incredible is that I didn’t even teach her how to use Figma—she explored the tool on her own, figured out all the controls, and took to it like a duck to water 🦆💦.

Polishing the Design

Once Rochelle finished her designs, I offered some guidance on refining them 💬. I introduced her to advanced concepts like components, instances, and prototyping (linking screens) without ever touching the mouse or keyboard. That’s one of my principles when teaching your kids o code or work with computers, never take over the controls. I firmly believe that learning by doing is the best way for skills to stick 👌.

And I must say, Rochelle did an amazing job 👏. The app is designed with a user-friendly interface, perfect for kids. It’s colorful, intuitive, and fun—just what a creativity canvas for children should be 🖌️✨.

…without ever touching the mouse or keyboard. That’s one of my principles when teaching your kids to code or work with computers, never take over the controls..

One thing I emphasize a lot with my kids is the importance of communication. It doesn’t matter how brilliant your idea is if you can’t present it well. Whether it’s app design, coding, or any other project, being able to clearly articulate your ideas is key to success. So, after every project, we always make time for a formal presentation. Presentation skills are as crucial as the technical ones. Rochelle not only designed this app but also put together a full pitch—explaining her thought process, the user experience, and why kids would love it 🗣️. She’s learning that creativity and communication go hand in hand, and I couldn’t be prouder of how she’s developing these skills 🤗.

We even made a trailer to showcase her work! 🎥 I used iMovie to edit the video (though if anyone knows of a better or easier software for video editing, I’m all ears). Watch the full video.

This experience has truly been an eye-opener. The future is indeed bright 🌟, especially with kids like Rochelle who are so creative and capable at such a young age 💫. Watching her dive into the world of app design with such enthusiasm and skill has made me incredibly proud 💖. Who knows, maybe this app will be the next big thing for kids!

Stay tuned for more updates on Rochelle’s journey. I can’t wait to see where her creativity takes her next! 🚀

Profile-Guided Optimization: A Hands-On Guide to Reducing Computational Wastage

Tue, 27 Aug 2024 23:01:35 +0000

Over the past few months, I’ve had numerous discussions with practitioners and colleagues on the benefits of Profile-Guided Optimization (PGO). While it’s a topic that generates significant interest, many find it challenging to get started or simply lack the time to explore it fully. As a Product Manager in the continuous profiling domain, my curiosity drove me to delve deeper into this subject. After studying various academic papers and articles, I decided to implement PGO myself, benchmarking its impact to assess its true value.

My primary goal was to understand the challenges hindering PGO adoption and to identify key questions that could reveal customers’ real pain points. Additionally, I aimed to explore the business value for end users. Specifically, I wanted to quantify how PGO impacts critical business KPIs such as conversion rates, latency, and even SLOs and SLAs.

This blog summarizes my initial findings.

Key Takeaways:

Efficient software is both cheaper and greener 🌿.
PGO can boost your code’s efficiency by up to 14% for free—without requiring any code changes.
A practical guide to implementing PGO is presented, including insights on measuring compute and end-user performance gains using inlining output, binary size, go-wrk, and flamegraphs.
In the example Go code provided, my analysis revealed a notable performance gain of ~ 6.92% in compute efficiency—an impressive result considering it’s based on a small JSON unmarshalling task. The potential savings in a production environment could be even more substantial.
Many developers and SREs are missing out on potential cost savings by not leveraging PGO. You can learn from Cloudflare’s experience in reducing costs through PGO.
Continuous profiling in production is essential to unlock the benefits of PGO fully.

PGO: Your Code’s Personal Coach for Peak Performance

Imagine you’re an athlete training for the Olympics. A generic training regimen might get you started, but it won’t fully optimize your performance. This is why most athletes hire a personal coach. A personal coach observes your training, identifies areas for improvement, and adjusts your regimen accordingly. This personalized approach leads to better results in a shorter time.

PGO is like that coach for your code. It analyzes your code’s performance, identifies frequently executed code paths, and feeds that data to the compiler to refine its optimization decisions. By informing the compiler of your code’s hotspots, PGO helps your code achieve peak performance, just as a coach helps an athlete win an Olympic gold medal.

To connect the dots, the way a coach observes an athlete’s training regimen and physiological components is similar to profiling in this context. Just as a coach scouts their athletes, continuous profiling analyzes your code’s behavior on an ongoing basis. Additionally, your gold, as a developer or SRE, could be promotion, a reduction in carbon footprint and/or lowering cloud spend.

PGO is an advanced optimization technique that leverages profiling data to guide the compiler in making more informed optimization decisions. By using PGO, the compiler can optimize hot code paths, potentially leading to significant performance improvements.

Further, PGOs can improve your code’s resource utilization efficiency by up to 14%. That’s a lot of cost savings in most medium - to large organizations without making any code changes, that’s free money on the table!

As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%. We expect performance gains to generally increase over time as additional optimizations take advantage of PGO in future versions of Go. (source: PGO Overview, Google)

Cloudflare recently shared a blog post detailing how they significantly reduced cloud spending by implementing PGO.

This indicates that following the release, we’re using ~97 cores fewer than before the release, a ~3.5% reduction. (source:Colin Douch, Cloudflare)

Comparing the before and after flamegraphs of PGO. See details in the results section below

Compilers and the One-Size-Fits-All Problem

As described in the athlete analogy above (inspired by the recent 2024 Paris Olympics), one of the compiler’s primary tasks is to make optimal decisions about your code during compilation. Compilers come equipped with heuristics that guide various optimization techniques—such as dead code elimination, register allocation, constant folding, and function inlining. These strategies are designed to streamline your code and enhance its efficiency.

However, the heuristics that guide the compiler’s decisions have the same problem inherent in most one-size-fits-all designs. There are limits to what a compiler can do on its own. For example, while inlining functions can reduce the overhead of function calls, the compiler can’t inline everything. Inlining too much can lead to bloated binaries, increased cache usage, and ultimately, performance degradation.

This is where PGO comes in. By providing the compiler with profiling data—information about how your code actually runs in real-world scenarios—the compiler can make more informed decisions. It can identify the “hot” functions that are frequently called and optimize them more aggressively, such as by inlining these critical functions or better allocating registers. This targeted optimization helps reduce the overhead of function calls and can lead to significant performance gains. PGO allows the compiler to go beyond generic optimizations, tailoring the final binary to the specific needs of your program based on actual usage.

How does PGO work?

Several other languages support PGO. See Laguages that Support PGO section. In this example, let’s consider a simple Go application that unmarshals JSON data containing personal bios. The application reads data from a file, processes it, and then serves it over HTTP. Here’s the code https://github.com/iogbole/go-pgo

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	_ "net/http/pprof" // Import pprof for profiling
	"os"
)

type BioWrapper struct {
	Bio Bio `json:"bio"`
}

//truncated 

// extractNames is a leaf function that processes course names.
func extractNames(data []string) []string {
	names := make([]string, 0, len(data))
	for _, name := range data {
		names = append(names, name)
	}
	return names
}

func main() {
	// Set up the HTTP server with pprof enabled
	http.HandleFunc("/", processor)
	log.Println("Starting server on :8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

// processor handles incoming HTTP requests, processes the JSON, and returns the JSON data.
func processor(w http.ResponseWriter, r *http.Request) {
	// Read the JSON file
	file, err := os.ReadFile("./bio.json")
	if err != nil {
		http.Error(w, "Error reading bio.json", http.StatusInternalServerError)
		return
	}

	// Unmarshal the JSON into the BioWrapper struct
	var bioWrapper BioWrapper
	err = json.Unmarshal(file, &bioWrapper)
	if err != nil {
		http.Error(w, "Error decoding JSON", http.StatusBadRequest)
		return
	}

	// Access the Bio data
	bio := bioWrapper.Bio

	// Process the Bio data (e.g., print to console)
	fmt.Println("Name:", bio.PersonalInfo.Name)
	fmt.Println("University:", bio.Education.University.SchoolName)
	fmt.Println("Current Job:", bio.WorkExperience.Job2.Role)

	// Use the leaf function to extract course names
	courseNames := extractNames(bio.Education.University.Courses)
	fmt.Println("Courses:", courseNames)

	// Set the response header to indicate JSON content
	w.Header().Set("Content-Type", "application/json")

	// Using json.MarshalIndent for pretty-printing
	responseData, err := json.MarshalIndent(bioWrapper, "", "    ")
	if err != nil {
		http.Error(w, "Error encoding JSON", http.StatusInternalServerError)
		return
	}

	w.Write(responseData)
}

Step 1: Compile and Run Without PGO

First, let’s compile the program without PGO and see how it performs under load.

go build -o pre_pgo -gcflags="-m"  main.go

./pre_pgo

Step 2: Load Testing with go-wrk

To simulate real-world usage, we can use go-wrk, a Go-based benchmarking tool, to send concurrent HTTP requests to the service. Install it from here.

go-wrk -d 20 http://localhost:8080

This command runs the load test for 60 seconds, generating HTTP requests to the server. The results will give us a baseline of how the program performs without PGO.

Step 3: Generate Profiling Data

While that’s running, let’s profile the code for 30 seconds.

curl -o default.pgo "http://localhost:8080/debug/pprof/profile?seconds=30"

Note that default.pgo is the pprof output. An alternative approach is to use the pprof tool.

go tool pprof -output=cpu.prof http://localhost:8080/debug/pprof/profile?seconds=60

Step 4: Compile and Run post PGO

Now, let’s compile the code with PGO enabled using the profiling data we collected:

go build -o post_pgo -pgo=auto -gcflags="-m"  main.go

./post_pgo

Step 5: Add load and Profile

Similar to step 3, generate load and profile - but this time, change the output of the curl to post_pgo.pprof.

curl -o post_pgo.pprof  "http://localhost:8080/debug/pprof/profile?seconds=30"

The Results!!

This was my first experience with PGO, and I was eager to see if it was worth the effort. I’ve saved all the raw benchmark files in the GitHub repository for future comparisons.

Let’s begin!

Inlining Optimization Analysis

Inlining is a compiler optimization technique that directly inserts the body of a function into the calling code, eliminating function call overhead and potentially enabling further optimizations.

Binary Size: Inlining Has a Small Storage Cost

A quick way to check for better inlining is to examine the binary size. While inlining can improve performance, it may increase the binary size due to code duplication.

Let’s check it out:

$ ls -atrl

-rwxr-xr-x 1 israel staff 8162050 Aug 23  18:45 pre_pgo
-rwxr-xr-x 1 israel staff 8244658 Aug 23  21:52 post_pgo

The difference in file sizes between the pre_pgo and post_pgo binaries can be summarized as follows:

File	Size	Difference	Analysis
`pre_pgo`	8,162,050 bytes	N/A	This is the size of the binary before applying PGO.
`post_pgo`	8,244,658 bytes	82,608 bytes larger i.e ~ 1.01% larger	The binary size increased slightly after PGO, likely due to additional inlining and optimizations.

Analysis of the compiler output

For the record, I nearly jumped out of my chair when I realized that the processor function was inlined. It makes all the difference, especially as it’s a leaf function.

Let’s dig a bit deeper by comparing the build outputs:

Before PGO

Refer to the repo for the full build output

Now let’s compare the compiler output for PGO.

After PGO

Refer to the repo for the full build output

The inlining optimization looks way better! Here’s a table comparing the key differences between the compiler output with and without PGO, based on the provided gcflags="-m" output:

Metric	Post PGO	Pre PGO	Difference	Analysis
Inlining	`extractNames`, `fmt.Println`, `http.ListenAndServe`, etc.	`extractNames`, `processor`, `http.(*response).Write`, etc.	More functions inlined, including `processor` and HTTP response writes	More extensive inlining in PGO due to better optimization hints
Parameter Leaks	Multiple parameters escape to the heap	Similar, with some optimizations such as better handling in `extractNames`	Slight improvement in handling heap allocation in PGO	PGO provides better guidance for optimizing heap allocations
Moved to Heap	Variables like `bios` moved to heap	Variables like `bioWrapper` moved to heap	Slight change in what gets moved to heap	Heap allocations are slightly optimized with PGO, but not eliminated
PGO Devirtualization	Not applicable	PGO devirtualizes interface calls	Devirtualization is applied	PGO enables devirtualization, improving performance by optimizing interface calls
Leaking Param	`data`, `w`, etc. escape to heap	Similar parameters leak but with better optimizations such as `extractNames`	Slight reduction in the number of heap escapes	PGO guides more efficient memory management, but some leaks persist

Flamegrapsh analysis

Using https://www.speedscope.app, I uploaded the before and after pprof files for quick analysis. Since Speedscope lacks a differential flamegraph feature, I compared both flamegraphs side-by-side.

But first, let’s identify the top functions and verify the weight of the callers and callees.

The sandwitch view - showing callers and callees

The sandwich view shows that we need to focus on the main.processor function, which is somewhat expected. Let’s verify this hypothesis with a flamegraph.

Differential flamegraphs

The table below summarizes the analysis of the before and after PGO flamegraphs

Metric	Pre PGO (Left)	Post PGO (Right)	Time Gained/Lost	Percentage Improvement
`main.processor` Total Time	48.25 seconds	44.91 seconds	3.34 seconds gained	6.92% improvement
`main.processor` Self Time	10.00ms	0.00ns	10.00ms gained	100% improvement
HTTP Handler Functions Total Time	Higher	Lower	Not Quantified	Slight improvement (qualitative)
Syscall Execution Time	Higher	Lower	Not Quantified	Slight improvement (qualitative)

Looking a bit further, the 6.92% improvement is most likely derived by inlining the main.processor function.

End-user/business impact analysis

The flamegraph analysis revealed a 6.92% improvement in overall code efficiency. While this is a substantial gain, especially considering cloud costs, it’s crucial to understand how it translates into real-world benefits for end users. To quantify the impact, I turned to the go-wrk benchmarking data, located here. By analyzing these metrics, I can assess the practical performance improvements that users will experience. Let’s delve into the details and see how these optimizations translate into tangible benefits

Baseline Pre_PGO output

and the comparison post pgo go-wrk result is:

Let’s delve into the details and see how these optimizations could translate to business KPIs.

Metric	Pre-PGO	Post-PGO	Difference	Percentage Gain/Loss
Total Requests	1,557,674	1,626,603	+68,929 requests	+4.43%
Total Data Transferred	1.47 GB	1.54 GB	+0.07 GB	+4.76%
Requests/sec	26,100.68	27,257.19	+1,156.51 requests/sec	+4.43%
Transfer/sec	25.29 MB	26.41 MB	+1.12 MB/sec	+4.43%
Overall Requests/sec	25,959.83	27,108.41	+1,148.58 requests/sec	+4.42%
Overall Transfer/sec	25.15 MB	26.27 MB	+1.12 MB/sec	+4.45%
Fastest Request	65 µs	66 µs	+1 µs	-1.54%
Average Request Time	382 µs	366 µs	-16 µs	+4.19%
Slowest Request	56.609 ms	57.569 ms	+0.960 ms	-1.70%
Standard Deviation	651 µs	655 µs	+4 µs	-0.61%

Percentile Response Times

Percentile	Pre-PGO	Post-PGO	Difference	Percentage Gain/Loss
10%	82 µs	80 µs	-2 µs	+2.44%
50%	90 µs	88 µs	-2 µs	+2.22%
75%	94 µs	90 µs	-4 µs	+4.26%
99%	96 µs	92 µs	-4 µs	+4.17%
99.9%	97 µs	92 µs	-5 µs	+5.15%
99.9999%	97 µs	92 µs	-5 µs	+5.15%
99.99999%	97 µs	92 µs	-5 µs	+5.15%

In summary:

Requests/sec and Total Data Transferred saw a +4.43% and +4.76% improvement, respectively.
Average Request Time decreased by 16 µs, showing a +4.19% improvement.
Slowest Request and Fastest Request showed minor changes with slight percentage losses.
Percentile Response Times improved across the board, with the most significant gains in the higher percentiles, up to +5.15%.

These results highlight the performance improvements achieved with PGO, reflected in better throughput and reduced latency.

What’s not to like?

Laguages that support PGO

PGO is supported by many popular programming languages and compilers. Key examples include:

C/C++: Supported by GCC, Clang/LLVM, and MSVC with options like -fprofile-generate, -fprofile-use, and /LTCG:PGOptimize.
Go: Available from Go 1.20, using the pprof tool for profiling.
Rust: Supported via rustc with -Cprofile-generate and -Cprofile-use.
Java: Optimized by Oracle HotSpot JVM and GraalVM through JIT compilation.
.NET: The .NET Runtime uses PGO to optimize JIT compilation.
Swift: Leverages LLVM’s PGO infrastructure.
Python: CPython can be compiled with PGO using ./configure --enable-optimizations.
Fortran: Supported in GFortran as part of GCC with the same options as C/C++. Does anyone still write Fortran code?

Conclusion

This blog post explored Profile-Guided Optimization (PGO) and its potential benefits for improving code performance.

We used a simple Go application as an example, demonstrating the steps involved in applying PGO:

Compile and Run Without PGO: Establish a baseline performance metric.
Load Testing: Simulate real-world usage with tools like go-wrk.
Generate Profiling Data: Capture runtime behavior using pprof.
Compile and Run with PGO: Leverage profiling data for optimizations.
Analyze Results: Compare performance improvements and assess optimizations through techniques like flamegraph analysis.

The example showcased potential performance gains through PGO, including reduced execution time and optimized memory management. It’s important to note that specific results may vary depending on the application and the profiling data.

Why aren’t more developers leveraging the power of PGO? Please share your thoughts in the comments.

References

Stack Walking Without Frame Pointers Was a Pain – But Not Anymore!

Fri, 26 Jul 2024 03:01:35 +0000

The omission of frame pointers complicated stack walking, but .eh_frame data now allows accurate stack unwinding without debug symbols. With Fedora and Ubuntu re-enabling frame pointers, profiling and debugging have become much easier and more efficient.

The Importance of Frame Pointers in Sampling Profilers

A sampling profiler periodically instructs the operating system to collect data about the running processes, typically multiple times per second per CPU core. Each time the profiler gets control, it performs a “stack walk.” This means it analyzes the call stack from the current location where the CPU is executing, tracing back through the function calls to determine the sequence of code that led to that point.

This stack walking helps answer crucial questions: “Where are we currently spending our time?” and “How did we get here?” Historically, performing such stack walks was straightforward, but modern compiler optimizations and architectural complexities have made it much more challenging.

Stack Walking

In the good ol’ days, x86 compilers used a predictable set of assembly instructions (the “standard function prologue”) to manage the stack and frame pointers. This made it a breeze to walk the stack and understand the call sequence. Think of it like having a clear map with marked landmarks.

Here’s an example of that prologue:

push ebp
mov ebp, esp
sub esp, sizeof(local variables)

The frame pointer (EBP) used to be a critical register that pointed to the base of the stack frame, allowing easy retrieval of the function call sequence. This method ensured that we could trace the execution path accurately by following these pointers. However, the advent of more advanced compiler optimizations has complicated this process.

The Problem of Omitted Frame Pointers

The x86 architecture had a limited number of general-purpose registers. To improve performance, compiler engineers realized they could free up the frame pointer for general computations by omitting its use and keeping track of the stack pointer (ESP) instead. This optimization, known as frame pointer omission, provided a significant performance boost but at the cost of complicating stack unwinding for profilers.

The rationale behind this trade-off was that developers needing to debug production binaries would use debug symbols, which provide additional data to track the stack pointer’s position. However, this assumption doesn’t hold well for profiling, where simple stack walking without debug symbols is essential.

The Impact on Profiling

The omission of frame pointers breaks the ability of sampling profilers to unwind native stacks through third-party libraries. Even if your code is compiled with frame pointers, any call into a library without them (e.g., libc for memory allocation) disrupts the profiling, rendering the profiler unable to trace the call sequence accurately.

This issue is pervasive as most major Linux distributions ship libraries compiled with frame pointer omission, breaking simple stack walking for profiling. The common workarounds—recompiling libraries without frame pointer omission or deploying debug symbols—are impractical for many organizations due to the complexity and overhead involved.

.eh_frames to the rescue

The challenge lies in finding a way to profile systems effectively without the need for debug symbols or recompilation. Our approach, with Elastic Universal Profiling, leverages C++ exception handling data embedded in executables. This data, present due to the need for stack unwinding during exception handling, is available even in non-C++ projects to ensure interoperability.

By using this data, we can perform full stack unwinding without relying on debug symbols. This involves reading the .eh_frame section of executables, which contains the necessary unwinding information. Our solution uses eBPF (Extended Berkeley Packet Filter) to manage this process efficiently. When our eBPF code encounters an executable compiled without frame pointers, it signals our userspace agent to read the .eh_frame section and convert it into a more manageable data structure. This data is then provided back to the kernel code, enabling rapid stack unwinding.

Framepointers are back

The good news is, framepointer are back again by default.

Following Fedora deciding to enable frame pointers by default for their package builds to help debugging and profiling with the stock packages, Ubuntu Linux is now going to be doing the same.

*source phoronix

Conclusion

The omission of frame pointers, while beneficial for performance, poses significant challenges for profiling. Our innovative use of C++ exception handling data provides a seamless solution for in-production profiling, ensuring accurate stack unwinding without the need for debug symbols or recompilation. This approach simplifies the profiling process, making it more accessible and effective for developers and engineers working to optimize their systems.

The return of framepointers will make life significantly easier for debugging system issues.

Symbolisation: Cracking the code with ELF and DWARF

Fri, 28 Jun 2024 12:01:35 +0000

Understanding DWARF and ELF

Before diving into symbolisation, let’s clarify two fundamental concepts: DWARF and ELF.

DWARF: This is a standard for storing debugging information within executable files. It provides a detailed blueprint of the program’s structure, including variable types, function names, and line numbers. Essentially, it’s a map from machine code back to the original source code.
ELF: The Executable and Linkable Format (ELF) is the file format commonly used for executables, object files, and shared libraries. It encapsulates the machine code, along with metadata such as section headers and symbol tables. ELF provides the container for the DWARF information.

The Art of Symbolisation

Symbolisation is the process of converting raw memory addresses (often hexadecimal) into human-readable symbols. These symbols represent function names, file paths, and line numbers. This transformation is crucial for understanding the behavior of a program, especially when analyzing performance metrics or debugging issues.

Example of an Unsymbolised Stack Trace

These are raw memory addresses that need to be translated into meaningful symbols.

Example of a Symbolised Stack Trace

A symbol in profiling is a combination of a file name, function name, and line number, uniquely identifying a line of code. A stack trace is a list of these symbols, representing a sequence of function calls ending in the currently executing function. Each line in this list is a stack frame.

`(fileA, function1, 10) ->
(fileB, function2, 20) ->
(fileC, function3, 30) ->
(fileD, function4, 40)

This stack trace indicates that function4 in fileD at line 40 called function3 in fileC at line 30, and so on.

Why Symbolisation Matters

Performance Profiling: By symbolising stack traces, developers can pinpoint which functions are consuming the most CPU or memory resources. This information is invaluable for optimization.
Debugging: When a program crashes or behaves unexpectedly, symbolisation helps identify the exact code location causing the issue.
Incident Analysis: In production environments, symbolising crash dumps or error logs aids in understanding the root cause of problems.

The Challenge of Symbol-Less Profiling

Traditionally, profiling required deploying applications with debug symbols, which can increase binary size and potentially impact performance. However, modern profiling tools, such as the Elastic Universal Profiling, have overcome this limitation by enabling symbolisation after the fact.

How Does it Work?

Data Collection: Profiling data is collected without symbols. This usually involves capturing stack traces and performance metrics.
Storage: The collected data is sent to a central repository or analysis platform.
Symbol Matching: The platform attempts to match the memory addresses in the stack traces with symbols from available debug information.
Symbolisation: Once a match is found, the addresses are replaced with human-readable symbols.
Analysis: The symbolised data can now be analyzed to identify performance bottlenecks or other issues.

Practical Example: Profiling a Go Application

Imagine profiling a Go application in production without debug symbols. The profiler collects performance data, including stack traces with memory addresses. This data is then uploaded to a profiling platform. The platform attempts to match the addresses against the symbols of the deployed binary or from a symbol server. Once symbolised, the profiler can generate reports showing which Go functions are consuming the most CPU time or memory.

Profilers collect these unsymbolised stack traces from native code programs and then use backend processes to associate the correct symbols, a process known as “symbolisation.”

Additionally, tools like addr2line are often used to perform this task manually. Here’s a basic example:

addr2line -e my_program 345223

This command would attempt to translate the address 345223 into a symbol using the information in the executable file my_program. The output might look something like:


my_program.c:123

Indicating that the address corresponds to line 123 in the file my_program.c.

Conclusion

Understanding DWARF, ELF, and the process of symbolisation is crucial for developers aiming to optimize application performance and troubleshoot issues efficiently. By leveraging these tools and techniques, one can bridge the gap between machine-level execution and human-readable code, enabling more effective debugging and performance analysis.

MBA Graduation: With Distinction, Israel Ogbole

Fri, 28 Jun 2024 12:01:35 +0000

I am super stoked. I attended my graduation with family and friends, and I am especially proud that my kids were present. The moment was surreal, surrounded by the people who supported me throughout this journey.

Needless to say, doing an MBA whilst working full-time and a young family wasn’t exactly easy. Balancing work responsibilities, family time, and the demands of the MBA program required a lot of dedication and sacrifice. Late nights, early mornings, and countless weekends were spent studying and completing assignments. But then again, nothing good comes easy in life.

Despite the challenges, the journey was incredibly rewarding. I learned so much, not only from the courses but also from the diverse experiences and perspectives of my classmates. The skills and knowledge I gained have already started to pay off in my professional life, and I am excited about the new opportunities that lie ahead.

Here’s a glimpse of the ceremony, a moment I will cherish forever:

It was worth it!

KubeCon Paris: Highlights and Key Trends

Thu, 28 Mar 2024 12:01:35 +0000

Fantastic week at #KubeCon Paris.

Key trends:

The user of AI/GPU demoninated the keynotes
OpenTelemetry - The adoption of continuous profiling. I am super excited about the role I played in making this happen
WebAssembly - it’s potential to enhance application portability
Environmental Sustainability, and
eBPF for observability and security use cases.

It was amazing to reconnect with industry peers, meet new people, and dive deep into cloud-native advancements.

Even more exciting was sharing my passion for continuous profiling, its adoption in OpenTelemetry, and Elastic’s contribution to this progress.

My favorite talk was by John Fastabend and Liz Rice on eBPF’s limitations. While familiar with eBPF’s capabilities, I, like many other attendees, was particularly curious to hear their thoughts on its limitations. After all, who better to critique eBPF than the experts who practically wrote it?

Their talk exceeded expectations, including a crash course on Turing completeness and a live demo of Conway’s Game of Life – using eBPF!

I had to take a selfie; my daughter would never forgive me otherwise! “Dad, are you watching Liz’s coding video again?” 😂

Join the conversation on LinkedIn

In pictures

That’s it!

BLUF: A Key to Effective Communication for Product Managers

Fri, 23 Feb 2024 12:01:35 +0000

BLUF: For product managers, adopting the Bottom Line Up Front (BLUF) communication technique can drastically improve clarity and efficiency in messaging, ensuring critical information is immediately clear and actionable.

In the fast-paced world of product management, clear and concise communication is not just a skill—it’s an imperative. One method stands out for its ability to cut through noise and drive home the core message right from the start: BLUF, or Bottom Line Up Front. This approach, with its origins in the military, offers a blueprint for communication that product managers can leverage to enhance clarity, decision-making, and efficiency in their teams. Let’s dive into what BLUF is, its military roots, why it’s invaluable for product managers, and how it differs from the commonly known TL;DR.

What is BLUF?

BLUF is a communication technique that presents the conclusion or the most important information at the beginning of the message. Instead of building up to the main point through background information or a narrative, BLUF demands that you start with the main message. This method ensures that the recipient understands the essence of the message immediately, enabling faster comprehension and action.

The essence of BLUF lies in its simplicity and directness. It challenges the communicator to distil their message to its core, thereby respecting the recipient’s time and cognitive load. In doing so, BLUF not only facilitates faster comprehension but also encourages a more focused and action-oriented dialogue.

Example: Product Launch Briefing

Without BLUF:

A meeting that starts with a review of the product development process, a detailed description of the product features, and gradually leads to the launch strategy and expected outcomes.

With BLUF:

” Bottom Line Up Front: Our new product will launch on March 15, targeting the young adult demographic with its innovative features designed to enhance their digital lifestyle. Our strategy focuses on digital marketing campaigns and strategic partnerships to maximize reach and engagement.

Now, let’s dive into how we got here, the product’s unique selling points, and the detailed launch plan to ensure we’re all aligned and ready to make this launch a success.”

In this briefing, the BLUF method ensures that the team immediately understands the what, when, and who of the product launch, setting the stage for a more detailed discussion on strategy and execution.

Origin in the Military

The BLUF technique has its roots in the military, where clear, concise communication can be the difference between success and failure, or even life and death. In this context, the ability to convey critical information swiftly and unmistakably is paramount. The military environment, characterized by the need for rapid decision-making under pressure, has thus championed the BLUF approach to ensure that essential information is never buried or overlooked.

Why Product Managers Should Use BLUF

In the world of product management, time is a precious commodity, and ambiguity is the enemy. Here’s why the BLUF approach is particularly well-suited for product managers:

Enhances Clarity: By stating the conclusion or main point upfront, BLUF eliminates the risk of key details getting lost in the shuffle. This clarity is crucial when communicating with stakeholders, team members, and customers who rely on precise information to make informed decisions.
Saves Time: In an era where everyone is battling information overload, BLUF respects the recipient’s time. It allows readers to quickly grasp the message’s essence without sifting through extraneous details.
Facilitates Decision Making: Product managers often need to drive or facilitate decision-making processes. BLUF communication ensures that the decision-makers have the critical information they need from the get-go, enabling more efficient and effective decisions.
Improves Email Efficiency: In a digital age where emails are a primary mode of communication, adopting a BLUF approach can significantly increase the efficiency of email correspondence, ensuring that the most important messages are seen and understood quickly.

BLUF vs. TL;DR

While BLUF and TL;DR (Too Long; Didn’t Read) both aim to summarize content, they serve different purposes and contexts. TL;DR is typically used at the end of a lengthy text to offer a brief summary for those who may not have the time or inclination to read the full text. In contrast, BLUF places the summary or conclusion at the beginning, ensuring that the reader is immediately aware of the main point before delving into any further details.

This distinction makes BLUF particularly effective for professional and strategic communications where immediate comprehension is critical, whereas TL;DR serves as a courtesy to provide a snapshot of lengthy digital content for casual readers.

In Conclusion

For product managers looking to streamline communication and enhance decision-making within their teams and projects, adopting the BLUF approach can be a game-changer. By bringing the bottom line up front, you not only respect your audience’s time but also ensure that your message’s core is never missed. In the demanding field of product management, mastering the art of BLUF communication could very well be the key to efficiency, clarity, and success.

As the digital workspace continues to evolve, embracing techniques like BLUF will be essential for professionals looking to maintain a competitive edge through superior communication strategies. So, the next time you draft an email or prepare a presentation, remember the power of starting with the bottom line up front. Your team, stakeholders, and inbox will thank you.

References

BLUF: The Military Standard That Can Make Your Writing More Powerful - Ref
BLUF Meaning: Learn How to Write Better with This Technique Ref

israelo.io

Coming for Cancer with Quantum-Accelerated Pipelines

Coming for Cancer With Quantum Power

References

The Hidden Curse of Love

User Journeys Are Dead. Enter the Agentic Journey Era

Why This Hit Me

What Really Is the Agentic Journey?

Product Metrics Must Change Too

Token Economics and Efficiency

What You Must Now Do

Closing Thought

Code & Communication: The Keys My 8-Year-Old Uses to Design Apps

Wireframing Her Vision

Polishing the Design

Sharing the Experience

Profile-Guided Optimization: A Hands-On Guide to Reducing Computational Wastage

PGO: Your Code’s Personal Coach for Peak Performance

Compilers and the One-Size-Fits-All Problem

How does PGO work?

Step 1: Compile and Run Without PGO

Step 2: Load Testing with go-wrk

Step 3: Generate Profiling Data

Step 4: Compile and Run post PGO

Step 5: Add load and Profile

The Results!!

Inlining Optimization Analysis

Binary Size: Inlining Has a Small Storage Cost

Analysis of the compiler output

Flamegrapsh analysis

End-user/business impact analysis

Percentile Response Times

Laguages that support PGO

Conclusion

References

Stack Walking Without Frame Pointers Was a Pain – But Not Anymore!

The Importance of Frame Pointers in Sampling Profilers

Stack Walking

The Problem of Omitted Frame Pointers

The Impact on Profiling

.eh_frames to the rescue

Framepointers are back

Conclusion

Symbolisation: Cracking the code with ELF and DWARF

Understanding DWARF and ELF

The Art of Symbolisation

Example of an Unsymbolised Stack Trace

Example of a Symbolised Stack Trace

Why Symbolisation Matters

The Challenge of Symbol-Less Profiling

How Does it Work?

Practical Example: Profiling a Go Application

Conclusion

MBA Graduation: With Distinction, Israel Ogbole

KubeCon Paris: Highlights and Key Trends

In pictures

BLUF: A Key to Effective Communication for Product Managers

What is BLUF?

Example: Product Launch Briefing

Origin in the Military

Why Product Managers Should Use BLUF

BLUF vs. TL;DR

In Conclusion

References