Sachin Joglekar

The story of hybrid architectures is honestly fascinating! I've been diving deep into why Transformers became the default choice, and looking at new model architectures. It's not because "attention is all you need" (though catchy!) It's because they exploited GPU parallelism so beautifully. But attention scales quadratically with context. With only attention and nothing else, it gets inefficient as we max out our context windows with the latest context engineering best practices - and try to run LLMs everywhere, on phones, laptops, desktops. Anyways, there's really cool work taking the same principle of fitting the model architecture to the hardware for efficiency + scale. You want a private real-time audio LLM on your phone? Yep. You want an LLM with 1000x higher rate limit on your laptop, so you can run an obscene number of agents in parallel? Yep. OK, so that brings me to the latest and possibly the greatest in this space! I recently learned more about Liquid AI's work. Their new foundation model LFM2 just dropped and they got this highly-coveted-in-AI top left triangle diagram comparing other LMs that run efficiently on CPU (see diagram). They used an evolutionary algo based on a research approach called STAR to discover new hybrid attention+X model architectures for CPUs to use in laptops and desktops, including AMD Ryzen. So instead of all attention blocks, the algo composed a gated convolution block to mix in. They found that 10 of those convs + 6 attention blocks were best for one of their CPU hardware targets. And the conv block looks like this: ```py def lfm2_conv(x): B, C, x = linear(x)      # input projection x = B*x                        # gating (gate depends on input) x = conv(x)                 # short conv x = C*x                        # gating x = linear(x) return x ``` I find it super cool that an algorithm composed this. Basically, let the algorithm discover what works, rather than forcing our preconceptions. Kind of a core principle of deep learning: just backprop to your dreams. The deeper insight: This is doubling down on Transformer's big win and maybeee moving from "one architecture to rule them all" to "let the hardware guide the design." Kinda like having specialized brain regions that each speak their native computational language. The challenge: There have been an absurdly huuuge number of optimizations (over)fit for attention and autoregressive models, from the super optimized kernels up to the GUIs. Makes sense, but this means that feeling these improvements as an end user can be difficult. And don't get me wrong, we still need attention, it's just not all we need :)

377

24 Comments

📢 New Post: Meet AXIOM : The Architecture That Outpaced OpenAI by 5000× The secret? A new, active inference based paradigm by VERSES that- -) Segments the world into objects (no labels, no CNNs) -) Assigns identities, learns type-conditioned physics -) Models cause-effect from scratch -) Simulates futures in real time -) Grows its structure when surprised -) Prunes redundant rules as it learns -) Plans actions by minimizing both regret and ignorance All of this happens online, from pixels, in minutes. No gradients. No backprop. And with an unmatched compute efficiency.  In the code-breaking challenge, VERSES Genius proved to be 140 times faster and over 5,260 times cheaper than OpenAI’s o1-preview. Against DeepSeek R1, Genius demonstrated far greater reliability by achieving a 100% success rate compared to DeepSeek’s 45%, while also being nearly 300 times faster and 800 times more cost-effective. To see how Verses accomplishes their elite performance, read the full breakdown here- https://lnkd.in/dCCnGgeT

115

18 Comments

Not all large language models are created equal. The architecture behind an LLM determines how fast it runs, how well it scales, and where it shines. Here are 6 architectures every AI builder should know: ➤ 1. Encoder-Only → Focus: Deep understanding, embeddings → Trained on: Masked tokens → Ideal for: Search, classification Examples: BERT, RoBERTa ➤ 2. Decoder-Only → Focus: Next-token prediction → Trained on: Left-to-right context → Ideal for: Chat, code, agents Examples: GPT-4, Claude, LLaMA ➤ 3. Encoder-Decoder (Seq2Seq) → Focus: Input → Output mapping → Trained on: Paired data → Ideal for: Translation, summarization Examples: T5, BART ➤ 4. Mixture of Experts (MoE) → Focus: Sparse routing for scale → Trained on: Expert selection → Ideal for: High performance, lower cost Examples: GLaM, DeepSeek-V2 ➤ 5. State Space Models (SSMs) → Focus: Long context, fast compute → Trained on: State transitions → Ideal for: Large documents, low latency Example: Mamba ➤ 6. Hybrid Models → Focus: Best of all worlds → Trained on: Mixed paradigms → Ideal for: Balanced performance Examples: Jamba, Grok 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources)  channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

221

24 Comments

Memory is becoming the new infrastructure layer of AI. Every week, new frameworks emerge promising to fix the same fundamental problem: LLMs forget too quickly, and retrieval systems can’t keep up with the complexity of real work. Let’s break down a few of the most influential approaches — and where they still fall short: 🔹 REFRAG (Meta + NUS) – Instead of stuffing long documents directly into context, it compresses retrieval chunks into embeddings, reducing latency and context bloat. Smart for efficiency, but still struggles with knowledge integration: how do you connect ideas across multiple documents, not just compress them? 🔹 KG-RAG (Adobe Research) – Builds structured knowledge graphs out of enterprise docs. Great for reducing hallucination and noise, but graph construction is brittle and costly. Updating ontologies fast enough for dynamic teams remains unsolved. 🔹 Human-Inspired Frameworks (SALM, CAIM, PREMem) – Bring in concepts like episodic, semantic, and procedural memory. These are closer to how humans think, shifting some reasoning into memory itself. But they’re mostly research-grade, not production-ready: integration into daily team workflows is minimal. 🔹 System-Level OS Approaches (MemOS, MemoryOS) – Treat memory like an operating system, with lifecycle management, scheduling, and migration across “short-term,” “mid-term,” and “long-term” layers. Strong architectural vision, but too abstract for non-research teams who just want answers today, not a full OS overhaul. 🔹 Social & Group Memory (Social-RAG) – Retrieves not just facts but social context from group interactions. Crucial for collaboration use cases, but limited to niche scenarios right now. Across all of these, two unsolved challenges keep coming up: Integration vs. Efficiency – Compressing memory helps scale, but often destroys the relational context teams actually need. Lifecycle & Evolution – Few frameworks handle the reality that knowledge is constantly being updated, contradicted, and reshaped. 💡 Why Tanka takes a different path Instead of treating memory as just a retrieval hack, Tanka treats it as the native infrastructure of work. Every message, doc, and decision is captured once, then structured into living memory — automatically. Agents don’t just retrieve snippets; they act on evolving context, update it, and carry it across workflows. Memory isn’t abstracted away into embeddings or graphs nobody sees — it’s visible, searchable, and agent-actionable. For teams drowning in information every day — switching between Slack, Notion, Google Docs, and email — this difference is existential. Most frameworks solve part of the puzzle. Tanka solves the team problem: how do we stop losing context every time someone leaves, a tool changes, or a project shifts? The answer: memory-native collaboration. Because the future of work isn’t just about smarter AI — it’s about AI that never forgets.

61

7 Comments