LLM Financial Applications

If you’re building anything with LLMs, your system architecture matters more than your prompts. Most people stop at “call the model, get the output.” But LLM-native systems need workflows, blueprints that define how multiple LLM calls interact, how routing, evaluation, memory, tools, or chaining come into play. Here’s a breakdown of 6 core LLM workflows I see in production: 🧠 LLM Augmentation Classic RAG + tools setup. The model augments its own capabilities using: → Retrieval (e.g., from vector DBs) → Tool use (e.g., calculators, APIs) → Memory (short-term or long-term context) 🔗 Prompt Chaining Workflow Sequential reasoning across steps. Each output is validated (pass/fail) → passed to the next model. Great for multi-stage tasks like reasoning, summarizing, translating, and evaluating. 🛣 LLM Routing Workflow Input routed to different models (or prompts) based on the type of task. Example: classification → Q&A → summarization all handled by different call paths. 📊 LLM Parallelization Workflow (Aggregator) Run multiple models/tasks in parallel → aggregate the outputs. Useful for ensembling or sourcing multiple perspectives. 🎼 LLM Parallelization Workflow (Synthesizer) A more orchestrated version with a control layer. Think: multi-agent systems with a conductor + synthesizer to harmonize responses. 🧪 Evaluator–Optimizer Workflow The most underrated architecture. One LLM generates. Another evaluates (pass/fail + feedback). This loop continues until quality thresholds are met. If you’re an AI engineer, don’t just build for single-shot inference. Design workflows that scale, self-correct, and adapt. 📌 Save this visual for your next project architecture review. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

🌟 This editorial from Radiology by Merel Huisman, MD, PhD, Felipe Kitamura, MD, PhD, Tessa Cook, Keith Hentel, Jonathan Elias, George Shih, and Linda Moy discusses the benefits and challenges of using large language models (LLMs) in clinical radiology, specifically focusing on clinical decision support, society guidelines and best practices, accuracy monitoring, academic administrative support, open-source and commercial LLMs, and agentic workflows. 🔍 It explores the potential of LLMs to enhance radiologists' work, highlighting their capabilities in generating reports, improving diagnostic accuracy, and providing patient-centered information. 🚨 The authors warn against overreliance on LLMs and the need for continuous monitoring, emphasizing the importance of maintaining accuracy and addressing biases. They advocate for combining quantitative metrics with qualitative user feedback. 💯 🤖 The article also explores the development of open-source LLMs, a potential solution to avoid overdependence on commercial LLMs, and the emerging field of agentic workflows, where LLMs can perform tasks and make decisions autonomously. 👍🏼 It's refreshing to see radiology finally discuss agentic AI. Overall, this editorial provides excellent insight into LLMs in radiology, highlighting not only their potential but also their pitfalls and limitations. It's a recommended read for anyone interested in LLMs in radiology. 🔗 to the editorial is in the first comment. 👇🏼 #Radiology #ArtificialIntelligence #LLMs #GenAI #AgenticAI Radiological Society of North America (RSNA) #RadiologyAI

Small variations in prompts can lead to very different LLM responses. Research that measures LLM prompt sensitivity uncovers what matters, and the strategies to get the best outcomes. A new framework for prompt sensitivity, ProSA, shows that response robustness increases with factors including higher model confidence, few-shot examples, and larger model size. Some strategies you should consider given these findings: 💡 Understand Prompt Sensitivity and Test Variability: LLMs can produce different responses with minor rephrasings of the same prompt. Testing multiple prompt versions is essential, as even small wording adjustments can significantly impact the outcome. Organizations may benefit from creating a library of proven prompts, noting which styles perform best for different types of queries. 🧩 Integrate Few-Shot Examples for Consistency: Including few-shot examples (demonstrative samples within prompts) enhances the stability of responses, especially in larger models. For complex or high-priority tasks, adding a few-shot structure can reduce prompt sensitivity. Standardizing few-shot examples in key prompts across the organization helps ensure consistent output. 🧠 Match Prompt Style to Task Complexity: Different tasks benefit from different prompt strategies. Knowledge-based tasks like basic Q&A are generally less sensitive to prompt variations than complex, reasoning-heavy tasks, such as coding or creative requests. For these complex tasks, using structured, example-rich prompts can improve response reliability. 📈 Use Decoding Confidence as a Quality Check: High decoding confidence—the model’s level of certainty in its responses—indicates robustness against prompt variations. Organizations can track confidence scores to flag low-confidence responses and identify prompts that might need adjustment, enhancing the overall quality of outputs. 📜 Standardize Prompt Templates for Reliability: Simple, standardized templates reduce prompt sensitivity across users and tasks. For frequent or critical applications, well-designed, straightforward prompt templates minimize variability in responses. Organizations should consider a “best-practices” prompt set that can be shared across teams to ensure reliable outcomes. 🔄 Regularly Review and Optimize Prompts: As LLMs evolve, so may prompt performance. Routine prompt evaluations help organizations adapt to model changes and maintain high-quality, reliable responses over time. Regularly revisiting and refining key prompts ensures they stay aligned with the latest LLM behavior. Link to paper in comments.

Clayton Christensen announced it — product managers are underestimating the disruption caused by Large Language Models (LLMs) for the reasons described in The Innovator's Dilemma. Incumbent organizations often focus on what new technologies CANNOT do, highlighting their limitations and risks instead of embracing the low-cost and scalability benefits that are emerging. Every profession has an implicit Return On Investment (ROI). If you're rejecting LLMs because they can only accomplish tasks with 80% quality, you're missing the point. A machine that can accomplish 80% of a task (= return) with merely 1% of the effort (= investment) offers a much much much better ROI than a human everything manually. Adding to this, there exists an absurd subconscious belief among some product managers that their lack of adoption will somehow slow down the inevitable tsunami of disruption. Combined with natural organizational inertia, this mindset results in a profession that clings to internal debates—such as the distinction between a product manager and a product owner—when it should be focusing on learning how to surf this lava-wave. Product managers should be obsessed with: 1. Breaking down their jobs into huge lists of tiny tasks; 2. Exploring how each task could be done slightly more rapidly thanks to LLMs; 3. Figuring out what new investments or habits need to happen to accelerate the tango — starting by abandoning ChatGPT and hopping onto LLMs that tap into private databases, your most important asset moving forward. Here's the beautiful part: LLMs are an amazing piece of technology, but the actual products remain to be invented on top of it. What's holding you back?

Many projects aimed at integrating LLMs into consumer-facing product features focus on generating content, which I think is mistaken. The power of LLMs for consumer-facing tasks is often better invoked through systems that rank, sort, classify, etc. at scale according to well-defined expert evals. Most large consumer platforms don't need *more* content; they need to better rank their existing content for some purpose. A blog post by Netflix serves as an excellent example of this: the company built an LLM-as-a-Judge system for evaluating content synopses. Netflix had already written these synopses, and in many cases, they had multiple per piece of content. The company wanted to rate each synposis in this bank according to four metrics: Precision, Tone, Factuality, and Clarity. To do that, it developed a rubric for each metric and asked creative experts to score a subset of 1,000 synopses according to it. Consensus was low, so they ran eight "calibration" rounds across the experts with 50 synopses each, updating the criteria at each round until the experts reached 80% consensus. Then, the experts produced a golden dataset of synopses with consensus labels that was used as a benchmark for Automatic Prompt Optimization (APO). The team iterated on the prompt using a dev subset of the golden labels, evaluating it against some accuracy objective, which allowed them to optimize separate prompts for each rubric metric. With the rubric and metric-specific prompts honed, each synopsis is evaluated with a binary score across all four metrics. Different levels of reasoning were induced through prompting (although no actual reasoning model was used), with some metrics using multiple LLM calls (Tone and Clarity) and the Factuality metric being scored by several agents that tested for specific information. The zero-shot CoT output (condensed through prompting to 1-2 sentences) was attached to metric scores for auditing, with those scores used to monitor synopsis quality at scale. But notably, the authors found that the resulting scores correlated with user engagement metrics; this suggests that LLM scores derived from expert-defined quality signals can serve as useful proxies for consumer behavior. What's noteworthy about the approach is that it uses off-the-shelf tools, with the expert evals contributing to APO rather than being used for fine-tuning. Further, the synopses themselves remain expert-crafted; the LLMs don't produce consumer-facing output but rather evaluate existing content against quality standards. LLM output isn't a differentiator or an asset; it's easy to detect and consumers mostly reject it. But cheap, commodity models can provide helpful reasoning traces and are valuable in generating content scores that can be utilized in innumerable consumer-facing product contexts. Netflix blog post linked below.

One of the most interesting debates I heard today at HIMSS... “Is it time to trust LLMs over interoperability standards?” For decades, healthcare IT has tried to solve interoperability the same way: Build structured standards → translate data → move it between systems of record. FHIR. HL7. Interfaces. Mapping tables. Endless implementation cycles. But the panel posed a provocative idea: What if LLMs flip that model entirely? Instead of forcing every piece of healthcare data into rigid structure… 👉 Let LLMs work directly with the messy, unstructured reality of healthcare. #RaihanReacts to some themes that stood out: 1. LLMs make massive EHR outputs digestible One panelist joked that they “spent yesterday afternoon in Claude Code building their own apps.” That comment actually reflects a real shift happening across healthcare IT. Clinicians and operators are suddenly able to take: • thousands of lines of chart data • referral notes • discharge summaries • scanned PDFs • messy clinical documentation …and have LLMs summarize, extract, and explain it in seconds. The result: 👉 EHR data becomes understandable and actionable, not just stored. 2. Unstructured data may actually be healthcare’s natural format Healthcare has always struggled to structure everything. But maybe that was the wrong goal. Clinical care is inherently messy: • free-text notes • patient narratives • imaging reports • external records • faxes (still…) LLMs are the first tech that actually thrives in this environment. Instead of forcing the system to change…the tech adapts to healthcare’s complexity. 3. Accuracy and governance are the real challenge Of course, trusting LLMs introduces a critical question: How do we verify the outputs? Several panelists talked about the need for: • monitoring frameworks • validation layers • auditability of summaries • “toggle switches” to turn LLM workflows on or off 👉 This is going to be one of the major operational challenges of AI adoption in healthcare. Not whether models can summarize data. But whether orgs can safely trust those summaries at scale. 4. Early operational wins are already emerging A few examples discussed: • Faster prescription workflows • Referral turnaround improvements • Automated chart summarization for intake and review These are small changes individually. But across a health system, they add up to thousands of hours of operational efficiency. My take: - Interoperability standards are not going away. - Systems of record will still matter. - But LLMs may become the universal interface layer that sits on top of them. HIMSS Global Health Conference & Exhibition

LLMs force a shift from performing work to specifying work. Because the machine only responds to what you articulate, you end up writing constantly: intent, constraints, context, expected output. The act is functional rather than literary. You strip away tone, polish and performance. What remains is the core of knowledge work: clarifying the problem, defining the goal, summoning the factual context. The immediacy of feedback is captivating. You see instantly how a clearer description yields a better result. Over time this trains the fundamental skills of high-leverage white-collar work: documenting ideas, setting objectives, defining deliverables, and enriching the brief with the right context. A generation now growing up with this muscle memory will expect work to run this way. Their default workflow will be: capture the idea in writing, specify the task, let the machine execute or assist. Meetings without agendas, projects without specs, assignments without context will feel amateurish. The norms of today’s office—already bloated by vague communication and performative collaboration—will look as dated as the mid-century rituals of secretaries typing dictated letters and executives spending afternoons in performative chatter over drinks. The shift is not that AI replaces office work; it rewires the discipline of doing it.

I’ve analyzed 20+ studies and spoken with a dozen experts to understand what makes content LLM‑friendly. Some of the most interesting takeaways: - Regularly update your content - one of the most underrated things with a huge impact. - Write like you speak - ChatGPT is trained on conversational data. - Structure articles around real audience questions - answer them clearly and right away. - Include concise definitions - LLMs answer lots of “What is…” queries, so use a dictionary‑style intro (e.g., “SEO is…”). - Offer multiple perspectives - LLMs love citing threads with varied experiences and problem‑solving approaches. - Publish proprietary data - content with original stats and research sees 30–40% higher visibility in LLM responses. I’ve put all my findings into an in‑depth set of guidelines - the exact ones my clients use. There’s even a checklist at the end that writers follow to make sure their content is LLM‑friendly. Want access? Comment “Share the guidelines” and I’ll send you the link. Let’s make our content visible beyond Google.

I’ve been building and managing data systems at Amazon for the last 8 years. Now that AI is everywhere, the way we work as data engineers is changing fast. Here are 5 real ways I (and many in the industry) use LLMs to work smarter every day as a Senior Data Engineer: 1. Code Review and Refactoring LLMs help break down complex pull requests into simple summaries, making it easier to review changes across big codebases. They can also identify anti-patterns in PySpark, SQL, and Airflow code, helping you catch bugs or risky logic before it lands in prod. If you’re refactoring old code, LLMs can point out where your abstractions are weak or naming is inconsistent, so your codebase stays cleaner as it grows. 2. Debugging Data Pipelines When Spark jobs fail or SQL breaks in production, LLMs help translate ugly error logs into plain English. They can suggest troubleshooting steps or highlight what part of the pipeline to inspect next, helping you zero in on root causes faster. If you’re stuck on a recurring error, LLMs can propose code-level changes or optimizations you might have missed. 3. Documentation and Knowledge Sharing Turning notebooks, scripts, or undocumented DAGs into clear internal docs is much easier with LLMs. They can help structure your explanations, highlight the “why” behind key design choices, and make onboarding or handover notes quick to produce. Keeping platform wikis and technical documentation up to date becomes much less of a chore. 4. Data Modeling and Architecture Decisions When you’re designing schemas, deciding on partitioning, or picking between technologies (like Delta, Iceberg, or Hudi), LLMs can offer quick pros/cons, highlight trade-offs, and provide code samples. If you need to visualize a pipeline or architecture, LLMs can help you draft Mermaid or PlantUML diagrams for clearer communication with stakeholders. 5. Cross-Team Communication When collaborating with PMs, analytics, or infra teams, LLMs help you draft clear, focused updates, whether it’s a Slack message, an email, or a JIRA comment. They’re useful for summarizing complex issues, outlining next steps, or translating technical decisions into language that business partners understand. LLMs won’t replace data engineers, but they’re rapidly raising the bar for what you can deliver each week. Start by picking one recurring pain point in your workflow, then see how an LLM can speed it up. This is the new table stakes for staying sharp as a data engineer.

LLMs are transforming how data teams tackle everyday tasks. Building ETLs, running analyses, statistical inference, and training ML models. The results are genuinely impressive. Some see this as a threat to their roles, and that concern is valid. It is still too early to tell exactly what all of this means for data teams. But I think that all of these systems still need a skilled professional at the center guiding them. We are simply just removing the painful, repetitive grunt work that used to define a data person's day-to-day work. It feels like data work is actually becoming what it was billed as ten years ago when I first got into it: the "sexiest job of the 21st century." Now professionals can finally focus on what actually matters: strategy, insights, and real business impact instead of wrestling with Airflow configs and debugging SQL for hours. What this means in practice is that data teams have dramatically increased their velocity. They're shipping analyses, dashboards, and insights at the speed the business actually moves. No more waiting weeks for a simple report. When a business question comes in, the data team can answer it now, not next sprint. That's a fundamental shift in how data teams operate and how valuable they actually are to the business.