Large Language Models (LLMs)

Projects

QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

ICLR 2026

QeRL is a framework that accelerates LLM training by combining NVFP4 quantization with LoRA. It reduces memory overhead, allows 32B model training on a single H100 GPU, and improves performance by leveraging quantization noise to

Fast-dLLM v2: Efficient Block-Diffusion Large Language Model

ICLR 2026

(

)

Our approach introduces a novel training recipe that combines a block diffusion mechanism with a complementary attention mask, enabling blockwise bidirectional context modeling without sacrificing AR training objectives.

Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

ICLR 2026

(

)

We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.

Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter

ASPLOS 2026

(

)

TLT is a lossless acceleration framework for reasoning-oriented LLM RL training, introducing adaptive speculative decoding to eliminate long-tail generation bottlenecks. It achieves over 1.7× end-to-end speedup while fully preserving model quality and producing a high-quality draft model for efficient deployment.

Long-RL: Scaling RL to Long Videos

NeurIPS 2025

(

)

Long-RL enables RL on hour-long videos on a single A100; LongVILA-R1-7B supports 8,192 frames and scores 71.1% on VideoMME with subtitles.

StreamingVLM: Real-Time Understanding for Infinite Video Streams

arXiv 2025

(

)

StreamingVLM enables real-time understanding of infinite videos with low, stable latency. By aligning training on overlapped video chunks with an efficient KV cache, it runs at 8 FPS on a single H100. It achieves a 66.18% win rate vs. GPT-4o mini on a new benchmark with videos averaging over 2 hours long.

Blog Posts

Why Stacking Sliding Windows Can't See Very Far

August 25, 2025

A mathematical explanation of why sliding window attention's effective receptive field is O(W) rather than the theoretical O(LW), regardless of depth, due to information dilution and exponential decay from residual connections.

Statistics behind Block Sparse Attention

August 22, 2025

A statistical model revealing how block sparse attention achieves efficiency and accuracy through learned similarity gaps.

How Attention Sinks Keep Language Models Stable

August 7, 2025

We discovered why language models catastrophically fail on long conversations: when old tokens are removed to save memory, models produce complete gibberish. We found models dump massive attention onto the first few tokens as "attention sinks"—places to park unused attention since softmax requires weights to sum to 1. Our solution, StreamingLLM, simply keeps these first 4 tokens permanently while sliding the window for everything else, enabling stable processing of 4 million+ tokens instead of just thousands. This mechanism is now in HuggingFace, NVIDIA TensorRT-LLM, and OpenAI's latest models.

TinyChat 2.0: Accelerating Edge AI with Efficient LLM and VLM Deployment

December 12, 2024

Explore the latest advancement in TinyChat – the 2.0 version with significant advancements in prefilling speed of Edge LLMs and VLMs. Apart from the 3-4x decoding speedups achieved with AWQ quantization, TinyChat 2.0 now delivers state-of-the-art Time-To-First-Token, which is 1.5-1.7x faster than the legacy version of TinyChat.

Block Sparse Attention

October 10, 2024

We introduce Block Sparse Attention, a library of sparse attention kernels that supports various sparse patterns, including streaming attention with token granularity, streaming attention with block granularity, and block-sparse attention. By incorporating these patterns, Block Sparse Attention can significantly reduce the computational costs of LLMs, thereby enhancing their efficiency and scalability. We release the implementation of Block Sparse Attention, which is modified based on FlashAttention 2.4.2.