Izzy Ekpo

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1

Ruokai Yin, our summer intern from Yale to Microsoft Azure, worked with Xuan Zuo, @Sattwik Deb Mishra, Hokchhay T. and Preyas Shah and me. We published a paper based on his project: https://lnkd.in/eHuG3BSt Summary of paper: Finding optimal sharding is hard especially for large LLMs. A large number of devices have to be divided into TP, PP, EP dimensions. For TP, we have to decide the tensor dimension e.g. head, hidden size to shard for every op. Additionally, TP can be one-dimensional/two-dimensional etc. This makes for a large sharding space that is hard to cover exhaustively. SoTA methods use heuristics to select sharding e.g. GSPMD, Sequence Parallelism, Helix etc. There are significant time gaps, months or years, between the discovery of each of these methods. We want to fast-forward the process of discovery by 1) using a formal representation of the sharding space, and, 2) using reinforcement learning to search the sharding space.    This paper demonstrates that our method is 3.5x better than random search, and on par with hand-tuned heuristics when looking only at 1D sharding and without allowing sequence parallelism. We aim to extend the work to multi-dimensional sharding as well as sequence parallelism. Beyond that, we wish to combine the problem of sharding with fusion, tiling and collectives as well.

58

🚨 Data streaming is due for its Iceberg moment. In this new post, our CEO, Sijie Guo, and Principal Sales Engineer, David Kjerrumgaard, explore why tightly coupled streaming architectures like Apache Kafka are hitting their limits—and what the future looks like. 👉https://hubs.ly/Q03tR4QS0 Drawing lessons from the lakehouse revolution (e.g., Apache Iceberg), they propose a three-layer architecture for streaming:
 🔹 Data – Durable, scalable object storage
 🔹 Metadata – Decoupled and authoritative
 🔹 Protocol – Stateless, multi-interface routing Streaming should be as modular, cloud-native, and vendor-neutral as modern analytics: 📉 Cut costs by up to 90%
 ⚙️ Evolve each layer independently 🔓 Enable open, multi-tool access Projects like #ApachePulsar and #StreamNativeUrsa are already paving the way!🌟 #StreamingData #Kafka #Iceberg #Lakehouse

6

Learn how you can minimize latency and LLM costs at scale: Partha Sarkar's new guide unpacks the nuances of validation-aware, multi-tier caching.

8

If your RAG is underperforming, the model you’re working with might not be the problem. It might be your chunking strategy. Effective chunking keeps LLMs accurate, efficient, and relevant by segmenting data into semantically meaningful, retrieval-friendly pieces. But scaling RAG requires more than naive splitting. Learn more from Redis AI Product Marketing Manager, Rini Vasan, here: https://bit.ly/46oeSRB

22

🔥Starburst Galaxy Kafka Streaming ingestion is on fire! 📊 New third-party Iceberg Kafka streaming ingestion benchmark results are out, and they provide strong independent validation that Starburst Galaxy data ingestion is highly performant and cost-effective compared to competitors. Here’s the breakdown: 1️⃣ In the benchmark, Concurrency Labs’ Ernesto Marquez compared Starburst Galaxy Data Ingestion with Amazon Web Services (AWS) AI Data Firehose and Confluent Tableflow for streaming ingestion from Kafka into Apache Iceberg using Amazon S3. 2️⃣ The tests used a standard TPC-DS dataset and evaluated ingestion from both Amazon MSK and Confluent Cloud under identical conditions. 3️⃣ The results show Starburst Galaxy resulted in an approximately 7x higher record ingestion rate compared to AWS Data Firehose or Confluent Tableflow. 4️⃣ On cost, Concurrency Labs estimates Starburst ingestion to be 87% lower than AWS Data Firehose and 81% lower than Confluent Tableflow at the tested throughput, while also producing significantly smaller Iceberg tables due to higher compression. 👉 Bottom line: If you are building Kafka to Iceberg pipelines and care about performance, cost, and open lakehouse architectures, this third-party analysis is worth reading and makes the case that Starburst Galaxy is the best possible way to implement Kafka streaming into Iceberg. Read the full benchmark results: https://okt.to/XhTSf7 #StarburstGalaxy #ApacheIceberg #Kafka #StreamingIngestion #Lakehouse #DataEngineering #Benchmarking #CostOptimization

79

5 Comments

As LLMs push toward 100‑trillion‑parameter models and agentic AI workloads explode, compute performance is advancing ~20X every two years, while memory bandwidth improves only 2–3X. If you’re working on AI silicon, HPC, or data‑center architectures, this is a must‑read on how HBM4 is breaking down the 'memory wall'. https://lnkd.in/gxUAeiQ3

63

1 Comment

OpenAI breaks down the PostgreSQL architecture behind its large-scale AI systems. A practical look at the design choices, scaling techniques, and lessons learned while running PostgreSQL in production at serious scale. 🔗 https://lnkd.in/dqWg6mRw #OpenAI #PostgreSQL #DatabaseScaling #AIInfrastructure #SystemDesign #DevOps #CloudComputing #SoftwareEngineering #AI #DB

4

Amazon Web Services (AWS) and Cerebras unveil a disaggregated AI inference architecture combining Trainium and CS-3 systems to accelerate LLM workloads on Amazon Bedrock. Read the Latest Full News – https://lnkd.in/dWjbDvAX #AIInfrastructure #GenerativeAI #LLMInference #CloudComputing #AIChips #MachineLearning #AIAcceleration #DataCenterTechnology #EnterpriseAI #AIInnovation #TechEdgeAI #TechEdge

1