Navid Vafaei

You are in a system design interview at Google for the L5 Senior Engineer role, and the interviewer leans in and asks: “Why does Spotify keep playing when I drive into a tunnel with no signal, but YouTube Music often stops or buffers? If you were designing a music streaming system, what different design choices would lead to these two behaviors?” Here is how you break it down. Btw, if you’re preparing for system design/coding interviews, check out our mock interview tool. You can use it for free here: https://lnkd.in/gpCn7t2T We’ve added new features as well:  -Company-specific interviews -In-built interview scheduler  -performance insights & trends — Both apps look like simple music players. Under the hood, they are optimized for very different priorities. [1] Spotify style – Cache first, stream second Idea: The client behaves like a smart offline player that happens to stream. The backend is built to support aggressive prefetching. What happens when you hit play - Client requests the track from a CDN or edge node. - Instead of tiny chunks, it downloads a big buffer ahead of the playhead. - In parallel, it starts pre downloading the next few tracks in the queue. - Data is written to an encrypted cache on disk, not just memory. - Playback reads from that local cache, not directly from the network socket. What this means for tunnels and bad networks - When the network drops, the player already has tens of seconds or entire tracks cached. - Because the next one or two songs are already downloaded, you can be offline for a while and never notice. - Cold start cost is a bit higher. First play might take slightly longer, but then everything feels smooth. - It burns more local storage and possibly more data, because not every prefetched song will be listened to fully. In a design answer, you can mention - Local disk cache with eviction policies (LRU per user, per device). - Background prefetch of N upcoming tracks based on queue. - Download manager that adapts how aggressively it prefetches based on network quality and user settings. - CDN tuned for larger object delivery and range requests. - Explicit offline mode that pins playlists into cache. [2] YouTube Music style – Stream first, cache is minimal Idea: Treat audio like video streams. Cost and bandwidth are optimized first. When you hit play - Player requests audio (and sometimes video) via HLS or DASH-style chunks. - Each chunk is only a few seconds long. - Client keeps a small rolling buffer in memory, not a large queue on disk. - Prefetch of future songs is limited, because video tracks are large and expensive to fetch speculatively. What this means for tunnels and bad networks - If your connection dies, the player only has a few seconds of buffered data. - As soon as those chunks are consumed, playback stalls. - Startup can feel snappy and data usage is controlled, especially for casual listeners. - Works very well on stable networks, feels fragile in spotty coverage.

1,221

38 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

Anyone notice that the Nvidia GPU architecture is converging closer to the Google TPU over time? Well, not exactly 1:1, but some principles. I wonder if the focus on matmul performance is driving the design decisions. For example, the Hopper WGMMA lets all of the tensor cores in an SM be “used together” in lock-step unlocking a larger unit tile churned. And, in Blackwell, you can smash together all the tensor compute from 2 adjacent SMs together, for a UMMA, compounded by the reduced operand tile movement from the distributed SMEM. The CUDA/PTX level abstracts away the hardware nuance. But, this clearly looks like some “virtual systolic array” designed for some serious arithmetic intensity (flops/byte). You take this to the extreme, and you end up with a single large systolic array backed by a giant on-chip SRAM, i.e., the MXU and VMEM of the TPU. I wonder if leaving all that configurability and “generalization” in there (especially for backwards compatibility in terms of the programming model) is costing them (Nvidia) something. A design with a laser focus on LLM inference could look entirely different.

179

6 Comments

The story of hybrid architectures is honestly fascinating! I've been diving deep into why Transformers became the default choice, and looking at new model architectures. It's not because "attention is all you need" (though catchy!) It's because they exploited GPU parallelism so beautifully. But attention scales quadratically with context. With only attention and nothing else, it gets inefficient as we max out our context windows with the latest context engineering best practices - and try to run LLMs everywhere, on phones, laptops, desktops. Anyways, there's really cool work taking the same principle of fitting the model architecture to the hardware for efficiency + scale. You want a private real-time audio LLM on your phone? Yep. You want an LLM with 1000x higher rate limit on your laptop, so you can run an obscene number of agents in parallel? Yep. OK, so that brings me to the latest and possibly the greatest in this space! I recently learned more about Liquid AI's work. Their new foundation model LFM2 just dropped and they got this highly-coveted-in-AI top left triangle diagram comparing other LMs that run efficiently on CPU (see diagram). They used an evolutionary algo based on a research approach called STAR to discover new hybrid attention+X model architectures for CPUs to use in laptops and desktops, including AMD Ryzen. So instead of all attention blocks, the algo composed a gated convolution block to mix in. They found that 10 of those convs + 6 attention blocks were best for one of their CPU hardware targets. And the conv block looks like this: ```py def lfm2_conv(x): B, C, x = linear(x)      # input projection x = B*x                        # gating (gate depends on input) x = conv(x)                 # short conv x = C*x                        # gating x = linear(x) return x ``` I find it super cool that an algorithm composed this. Basically, let the algorithm discover what works, rather than forcing our preconceptions. Kind of a core principle of deep learning: just backprop to your dreams. The deeper insight: This is doubling down on Transformer's big win and maybeee moving from "one architecture to rule them all" to "let the hardware guide the design." Kinda like having specialized brain regions that each speak their native computational language. The challenge: There have been an absurdly huuuge number of optimizations (over)fit for attention and autoregressive models, from the super optimized kernels up to the GUIs. Makes sense, but this means that feeling these improvements as an end user can be difficult. And don't get me wrong, we still need attention, it's just not all we need :)

378

24 Comments

“There's Plenty of Room at the Bottom – By Richard Feynman” Introducing BitNet v2, the first native 4-bit activation for 1-bit LLMs (and for any LLMs). It represents a significant technical advancement in 1-bit LLMs, following the introduction of BitNet v1 in October 2023 and BitNet b1.58 in February 2024. Activation outliers present significant challenges to LLM efficiency for low-bit LLMs, particularly in cloud and batch environments. We tackled this issue by proposing an online Hadamard transformation before activation quantization. This technique smooths sharp distributions into more Gaussian-like forms, making them amenable to low-bit representation. Experimental results demonstrate that BitNet v2 trained from scratch achieves performance comparable to BitNet b1.58 at 8-bit activations. More importantly, training with native 4-bit activations yields minimal performance loss, offering substantial reductions in memory footprint and computational cost for batched inference. BitNet v2 is a local optimum for low-bit LLMs – 1.58-bit model weights and 4-bit activations. Following the introduction of BitNet b1.58 2B4T, we anticipate the development of more and larger native 1-bit LLMs. To truly revolutionize AI efficiency, we reiterate our call for custom systems and hardware specifically optimized for these models, advocating for the co-design and co-evolution of model architecture, systems, and hardware. To conclude, echoing Richard Feynman's famous and profound quote "There's Plenty of Room at the Bottom," which envisioned a future of manipulating matter at incredibly small scales, a similar principle may hold true as well for 1-bit LLMs. https://lnkd.in/gy59Wtrd

178

3 Comments