Abhishek Sundararajan

Books 2026

2026-01-11T00:00:00+00:00

Tracking all that I read in 2026</p>

January</h1>

The Switch, Elmore Leonard</h2>
Not too nasty, which was unexpected. Only the deserving die in this one. Altogether, a serviceable Leonard.</p>

52 Pickup, Elmore Leonard</h2>
Nasty little crime thriller. SA makes me uncomfortable and the hints of it in this one was more than enough to make me want to never re-visit this again. But it's Elmore Leonard. Of course I devoured it as soon as I read the first line.</p>

Lord of the Rings, JRR Tolkein</h2>
Read this in a delirium, in 6 days, before the work week starts. That’s a close repeat of my experience with LOTR the first time I read it, 20 years ago. This time, I even shed some tears when King Théoden died, can you believe it. Then, two decades ago, it was Aragorn that captured my imagination. Now it’s Samwise Gamgee. Maybe that tells you what matters as you age: friendship, loyalty, good food. You know, the simple virtues.   > ‘Come, Mr. Frodo!’ he cried. ‘I can’t carry it for you, but I can carry you and it as well. So up you get! Come on, Mr. Frodo dear! Sam will give you a ride. Just tell him where to go, and he’ll go.’</p>
A fellowship is formed to see the end of the ring. Then the fellowship is broken involuntarily. And we follow some of the members as they go on their separate adventures. Aragorn, Gimli, and Legolas are always together. As are Frodo and Sam. Merry and Pippin for a brief while and then they are taken apart. We follow each. But not Gandalf. Or Boromir. In fact the only interiority we observe are the little people - Gimli (though not a Hobbit, still small in stature), Merry, Pippin, Frodo, and Sam. I wonder why that is. Maybe it’s because the Halflings are our surrogates as they get to know Middle-earth? That’s probably it. As the story stands, and given his deeply Catholic worldview, it’s just as likely that Tolkien’s point is that history is made by the humble. I’m tempted to say the meek inherit Middle-earth, but the story is too melancholy to allow that conclusion. They save it, but don’t get to keep it. And the world is passes into a lesser age.</p>

Mahaviri, Nityananda Misra</h2>

Some software, 2025

2025-12-23T00:00:00+00:00

One of things that make me happy are deep conversations. Happy is the appropriate word, I would think, if I were to describe that experience later to others. At the moment of connection, it’s more accurate to say my attention is narrow and judgement suspended, and I am completely engaged. I am in a flow state. Such conversations are few and far between, and thankfully there are many other opportunities in life to enter into that zone. Like using new tools, getting fluent with them, building with them. By tool, I mean both an instrument that allows one to get some job done, but also in this context, the job it self. Lego. Ikea furniture. Woodworking. Video games. Meditation. Seeing people use their tools with a feeling for craft, that does something to me. Take a look at master watchmakers or knife makers</a>. Tell me it doesn’t make you want to immediately build something. Good software makes me feel that way. In 2025, I found three: Ghostty</a>, Helix</a>, and Claude Code</a>.</p>

If I spent my career using one set of tools, working more or less the same way, there was a tsunami called Claude Code this year that wiped all that out. I am hard pressed to think of another product that has decisively changed the way I work. At the beginning it was not even clear that I was more productive with it than doing things by hand. But I found it enthralling to use, finding ways to work around the tasteless code it produces, figuring out ways to create guard rails. I make liberal use of plan mode, and I find that the result is still valuable even if I throw away any code that Claude Code produces: I have clarified my thinking about something. The more I work with it, the less I trust it to be right in many cases, but in every case it's useful. There's no going back now. No one in my org codes by hand anymore. Well, there are a few holdouts, and we don’t force people to use these tools. But they will give in eventually. It’s via Claude Code that I learned to enjoy living on the terminal. I now judge other terminal agents by the UX standards that Claude Code has set. Do you have a plan mode? What about subagents? Skills? No? Pass.</p>

About a couple of times a week, Claude Code on my iTerm terminal would get into an infinite scrolling mode and become unresponsive to key presses. Giving up on Claude Code was not an option. So I shopped around for other terminals and picked Ghostty because someone recommended that as a solution on slack. It also helped that Mitchell Hashimoto’s writing resonated with me, in particular, his blog post on vibe coding a UI feature</a>.</p>

The UX on Ghostty blew my mind. It was not as feature rich as iTerm, but I never used any of the fancy features anyway. What I want from a terminal are reasonable shortcuts for opening window panes and tabs, navigating between them, and closing them. Ghostty uses GPU for rendering, and on MacOS this done using Metal. Ghostty seems to cache rendered characters into a texture atlas on the GPU. When a character needs to be displayed, it's simply copied from the cache to the screen instantly. Ghostty uses a custom-built terminal core written in Zig. It interacts as directly as possible with the operating system's windowing primitives. All of this is to say, Ghostty owns the entire stack from the window creation to the PTY (pseudoterminal) communication. The end result is that all this design choices shaves milliseconds off input latency from the moment a key is pressed, to a character appearing on screen. Who knew the effect would be so mesmerizing? The phrase buttery smooth comes to mind. I haven’t found myself going back to iTerm or even exploring other terminal emulators at all.</p>

Ghostty gave me such a good experience that I wanted to do other things on the terminal alone. And what do I do on my computer most of the time? Aside from browsing, it’s reading and writing code. That led me to Helix.</p>

Helix is a modal editor. I had been unhappy with VSCode for a while, especially its performance on large code bases or opening large files. I played around with Zed and while I admit it was performant, it still left me with an empty feeling. It would do, however. But then along came Ghostty and it opened up my mind to the possibilities of TUIs. Around the same time, I read Julia Evans’s post about moving to Helix and I decided to check it out. You know, just some fooling around. I just never went back to VSCode and Zed. Learning modal editors was like discovering a game, a new way of seeing and navigating a world I thought I was already familiar with. Helix has sane defaults, key bindings that make sense. They were easy to memorize because they mapped to semantic understanding of what navigating a code base meant. Who knew that navigating a file with Alt+n</code> and Alt+p</code> and Alt+i</code> - moving up and down the syntax tree - would be faster not just because there's no mouse scrolling, but because it fills in your mental model of the code as you go? I am not sure I even made a conscious decision to memorize them, but here we are. Now it’s second nature. Those early days of exploration, I sometimes even found myself dreaming about Helix.</p>

So why these three, out of the many softwares that come and go? Part of it is that the friction to get deadly with them was just enough - not too much, certainly not too little. But the larger part is that these are made by people who clearly use them for their daily work, and that shows in UX choices that feel inevitable. Good tools make it easy to form habits, moves you reach for without thinking. That's the same thing that happens in a good conversation: the boundary between you and the thing you're engaged with starts to blur. Flow, again.</p>



Tale as old as time
2025-12-06T00:00:00+00:00
The little menace woke us up a couple of nights ago with a scream. I saw her sitting, looking into the distance, shaking and saying "I want to go home." The mother, panicking for a brief moment, tried to soothe her. A nightmare. This was a first. As mommy held her I whispered, "it's okay kanna, naan irukken. We will protect you." That, to my surprise, seemingly calmed her down. She snuggled next to me, resting her head on mine. Then a minute or two later, she said "I love you appa." It took her a while to fall asleep. And when it happened, her little hands were holding my finger.</p>
With my sleep ruined, and it being close to my waking time anyway, I got up soon after, and spent some time noodling around. I had ICLR review deadlines to meet. Some time around 7 am, I could hear sounds from the bedroom, and I knew the little menace was up. This was my second favourite part of the day, to greet the boss in the morning. The first place goes to the evenings, when I pick her up from day care, where she would unfailingly greet me with a smile and a bear hug the moment she spies me.</p>
As I walk into the bedroom, she looks at me groggily, and says, "Go away, I want my amma."</p>


Helix aide-mémoire
2025-11-23T00:00:00+00:00
You should be in NOR mode about 90% of the time. With coding agents, this is likely to be 99%, or even a 100%.
The trick with modal editors is to make it a habit to switch to NOR mode from INS often. Be liberal with that ESC press.</p>
You will want to use Helix exclusively to get fluent with it, so make it the default Unix editor. Drop VScode, Zed, TextEdit and any other WYSIWYG editors.</p>
Navigation</h2>
option+p</code>(p == previous), option+n</code> (n == next) = select previous block and next block. You gotta select a work or character first, which you will via w</code>.</p>
option+i</code> = go to beginning of selection, but in NOR mode (in comparison to just i)
This uses the LSP to navigate the code blocks.</p>
To navigate via paragraphs or functions, ]f</code> or ]p</code> to go to the next paragraph, and [f</code> or [p</code> for the previous paragraph.</p>
Then there is h,j,k,l</code>.
Do 10k</code> to jump 10 lines up.
120j</code> to go down the page 120 lines.</p>
Use w</code>, e</code>, b</code> to navigate word by word, forward and backwards.</p>
To navigate through all the changes you have introduced, use ]g</code> and [g</code>.</p>
gg</code> and ge</code>: Go to beginning and end of the file.</p>
g10g</code> or g41g</code> and so on: go to line 10, line 41 etc.</p>
gw</code> will show labels at each word, and you can type the label to select that word.</p>
gs</code> and gl</code>: Go to the first non white space character of the current line, and the end of the line.</p>
To cycle through buffers - files that you have opened - use gn</code> and gp</code>.</p>
Use gd</code> to navigate to a function definition. gr</code> to go to all the references.</p>
Splits</h2>
To create a vertical split, go to command mode and type "vs".
In file picker mode, select a file and do ctrl+v</code> and it will open that file in a new split.
ctrl+w+q</code> closes a split.</p>
To navigate between splits, ctrl+w+w</code>.</p>
Selection</h2>
Go up and down the syntax tree via option+o</code> and option+i</code>.
Select the entire file/buffer: %</code>.</p>
Click v</code> to go into selection mode, and now when you navigate via w</code> or b</code> or our friends h,j,k,l</code>, the selection should stay. Press ESC or semi-colon exit this mode.</p>
To select a line, press x</code>.</p>
To select all text between a pair of quotes, you would do mi"</code>. To select all text between brackets, mi(</code> or mi{</code>, or mi[</code>.
To surround a selection with some text, such as double quotes, you would make the selection, and then do ms"</code>. Or with parentheses, do ms(</code>.</p>
To select a word, with your cursor on some character in the word, you would do miw</code>. Lowercase w</code> selects a sequence of letters, numbers and underscores.
If you do miW</code>, the uppercase W</code> ensures all contiguous non-whitespace characters are selected.</p>
Combining the above information, if you want surround a word with quotes, you would hit these series of characters: miWms"</code>.</p>
Misc</h2>
To insert a new line above or below the current line, you will use [+space</code> and ]+space</code>.</p>
space+y</code> to copy any selection to the system clipboard. cmd+v</code> to paste from system clipboard.</p>
Once you have a selection, to make edits at the end, press a</code>. To make edits at the beginning of the selection, press i</code>.</p>
To delete a selection, press d</code>. To delete a selection and go into INS mode, press c</code>.
To undo, u</code>. To redo, U</code>.</p>
space+f</code> for fuzzy searching files, space+/</code> for fuzzy searching text through the project. Both are great - fast and miles ahead of VSCode.</p>


The Delhi Sultanate
2025-09-13T00:00:00+00:00
Book: The Delhi Sultanate: A Political and Military History</strong>  

Author: Peter Jackson</strong></p>
1.</h1>
In 1200 CE, the North Indian landscape was dominated by warring Hindu kingdoms, the successor states to the Gurjara-Pratihara polity that had ruled the region with their capital at Kanauj, today in modern-day Uttar Pradesh. Despite not presenting a united front, these kingdoms offered significant resistance to Muhammad bin Sam of Ghori as he expanded the Ghurid empire eastwards. Gradually though, the Ghurids won. A case could be made that the Afghans had a military advantage through their use of Turks. That is, they had an abundance of horses and a lifetime of practice in warring. They secured major victories, such as at Tarain (1192 CE) against Prithviraja and Govindaraja, and against Jayachandra of the Gahadavalas (1194 CE). The native Indian kingdoms did not just roll over however, and direct Muslim rule was only patchy. For example, the Rajput hill fortresses kept reverting to Hindu rulers, requiring recapture by subsequent Muslim rulers, as it was with Gwalior or Ranthambor.</p>
But the military advantage was real. This was reinforced by ideology. The Ghurids and the broader Islamic world's expansionist framework made it easier for them to incorporate fresh troops of steppe warriors into their ranks.</p>
2.</h1>

  
  Qutub Minar</figcaption>
  
</figure>
The first Turkish adventurers and marauders came to India following the campaigns of Mahmud of Ghazni around 1000 CE. They had entered Persia and other Islamic lands as free men during large-scale migrations of recently converted nomadic tribal groups from Central Asia. Turkic tribes were prized as slaves because of their martial prowess. Their children were enslaved, converted to Islam, and trained as soldiers. "Slave" in Arabic is ghulam, a word that has been borrowed into Hindustani with the meaning more or less intact. The ambitious empire of Ghor, based out of modern day Afghanistan, had many such ghulams in its service. Muhammad Ghori used them first as lieutenants, later as governors, in Punjab. From these positions of power, they eventually formed their own kingdom in India, known to posterity as the Slave/Ghulam dynasty, or Delhi Sultanate. It should be evident by now that ghulam</em> was a special type of servitude that was not seen as degrading. The descendants of these slaves formed a new elite within the aristocracy. Peter Jackson traces Delhi Sultanate's history from this foundation in 1210 CE under Iltutmish (d. 1236 CE), the ghulam of Qutb ud-Din Aibak. It was the first Islamic state to be established in India.</p>
The major dynasties that formed the Delhi Sultanate till 1400 CE are the Shamsids (Iltutmish and family), the Ghiyathids (Balaban and family), the Khaljis (Jalal al-Din Firuz and family), and the Tughluqs (Ghiyath al-Din Tughluq and family). While Iltutmish and Balaban were at one time ghulams, the majority of the rulers, and their respective descendants, were not.</p>
Aibak's chosen heir, Aram Shah (d. 1210-11), was murdered by Iltutmish before he ascended the throne. Balaban murdered his father-in-law, the last Shamsid sultan Nasir al-Din Mahmud Shah (d. 1266). Balaban also murdered his cousins, and Shamsi maliks, potential rivals for the throne. Of ten sultans in the 62 years after Iltutmish's death, only Balaban died a natural death.</p>
Alauddin Khalji's (d. 1316) reign began in 1296, and he was a nephew of the preceding sultan, who he murdered. Ibn Batuta</a> suggests that he in turn was poisoned by his chosen heir, Khidr Khan. Khidr Khan and his brother were blinded and imprisoned by Alauddin Khalji's slave, Malik Kafur (d. 1316), who was soon enough assassinated by Qutb al-Din Mubarak Shah, another of Alauddin's sons. Qutb al-Din Mubarak Shah then had his now-blinded brothers killed. Qutb al-Din Mubarak Shah (d. 1320) in due course met his end at the hands of his lover, the Indian slave Hasan, who ascended the throne as Nasir al-Din Khusraw Shah.</p>
Ghiyath al-Din Tughluq Shah, the first of the Tughluqs, marched to Delhi and overthrew Khusraw Shah. He died when a newly constructed building collapsed on him at Afghanpur in 1324 CE. Contemporary chroniclers strongly suspected that his eldest son and designated heir, Muhammad bin Tughluq, had contrived his death. What is certain is that Muhammad bin Tughluq (d. 1351) killed his brother.</p>
This book's narrative ends with the fall of Delhi to Timur in 1398, when the last Tughluq sultan, Mahmud Shah, fled Delhi after the rout of their forces.</p>
All our sultans, with the one Indian exception noted above, were of Turco-Mongol stock.</p>
3.</h1>
Lahore served as the capital of Muhammad Ghori's Indian province and the centre of Islam in India when Qutb ud-Din Aibak established himself as ruler there in 1206. Aibak was the Ghurid sultan's slave. The capital gradually shifted to Delhi due to frontier pressures - first from the expanding Khwarazm empire, then from the Mongols.</p>
While Delhi became an accidental capital, it proved an ideal location for raids into Hindu territory. Daring campaigns were conducted against Deogiri, Warangal, and Madurai. Such raids were highly lucrative, and the Sultanate gradually expanded its ambition from plunder to permanent suzerainty. However, Delhi's control over territories beyond their core northern region always remained weak. Governors and local chiefs routinely established independent rule. Bengal was incorporated into the Sultanate by Iltutmish, and would be an exemplar of this persistent insubordination. Gujarat was initially incorporated by Aladdin Khilji in 1310 CE, would follow a similar pattern of revolt and reconquest. By Tughluq's death in 1351 CE, "Bengal and every tract south of the Vindhyas had declared their independence, and none of these provinces was ever recovered." The Deccan remained under Delhi for less than thirty years.</p>
Maps show a uniform shading to define the extent of Sultanate rule, though a series of dots would indicate with greater realism the extent of actual power. Each dot would be a fortified city or garrison, where the provincial Muslim governor had chosen to recognize the monarch. Even within these areas, the authority of the governor waned with distance, depending on local military operations or distance from Delhi.</p>
Ibn Batuta, who traversed these lands between 1333 CE and 1347 CE, noted how India was different from other Muslim polities. Elsewhere in the Islamic world it made sense to talk about Dar al-Islam (abode of Islam) and Dar al-Hard (the “war-zone” or pagan territory), but not in India. Here, there was no clear physical boundary between Muslim and non-Muslim lands.</p>
4.</h1>

  
  The Execution of Jalal-al-Din Firuzshah II by Alauddin Khalji. Via an encyclopedia from the Mongol Ilkhanate, Jami' al-tawarikh.</figcaption>
  
</figure>
Iltutmish's initial encounter with Mongols were diplomatic and indirect. Genghis Khan was in the process of swallowing up the Khwarazmian empire, and chased the son of that Shah, Jalal al-Din all the way to the Indus. Genghis Khan contemplated returning to Mongolia through the Himalayan foothills and sent envoys to Iltutmish to request permission to pass through his dominion. Iltutmish sent the envoys away under guard, with a no. He was lucky that Genghis was superstitious enough to take another way that one time. The destruction of the Khwarazmian empire inadvertently ensured Muslim India was not part of the Khwarazmian sphere of influence or direct control. The Mongols ravaged the regions of Multan and Lahore, and, under Ogodei, these were subject to the Mongols and considered part of their border territory.</p>
Mongol incursions devastated these border regions, with Punjab suffering annual attacks. The Mongols frequently penetrated deep into the Delhi Sultanate's territory, even reaching the capital itself on multiple occasions. This threat along the Indus frontier forced successive Sultanate rulers into aggressive militarization and massive resource allocation for defense. For example, Sultan Balaban (d. 1287 CE) established a separate army specifically to combat the Mongols. Alauddin Khalji implemented drastic fiscal and administrative measures to support large armies while keeping prices low in the capital to fund his troops. Muhammad bin Tughluq had a large Khurasan force, intended for an offensive against the Mongols, with an estimated 470,000 men.</p>
This frontier pressure also created a major flow of human capital into Delhi. Iltutmish deliberately positioned himself as a refuge for those fleeing the Mongols, attracting Persian scholars, ulemas, and soldiers of fortune. There was another source of human capital. Victorious campaigns against Hindu kingdoms provided slaves for the Sultanate's ambitious construction projects. A historian of the court, Barani, reports that Alauddin Khalji employed 70,000 Hindu laborers on his Alai Darwaza and Jamait Khana mosque. Indian slaves and their paiks also served in Muslim military campaigns.</p>
The resource drain of constant frontier warfare ultimately came at the expense of internal expansion. Muhammad bin Tughluq's heavy taxation in the Doab, imposed to fund his Mongol offensive, provoked widespread revolt among cultivators, leading them to burn crops and flee to the forests. Successive rulers eventually abandoned distant campaigns, focusing instead on internal administration. This conscious reduction in long-range military capabilities left the Sultanate unprepared to face Timur in 1398, who destroyed Delhi and massacred its population.</p>
After Timur's devastation, the Sultanate reverted to being merely one of several competing northern powers. New or renascent Hindu principalities (Mewar, Alwar, Gwalior) and rival Muslim kingdoms (Gujarat, Bengal, Malwa) eroded its territorial control. The Delhi Sultanate itself lacked a ruler for three years following Timur's departure.</p>
5.</h1>
The Ghurid historian Siraj al-Din Juzjani served in the court of Sultan Nasir ud din Mahmud Shah, son-in-law to Sultan Balaban. In his Tabaqat-i Nasiri, he frequently refers to Muhammad Ghori as Sultan-i-Ghazi, the holy warrior sultan, a pillar of Islam. Juzjani likens his triumphs over the Indian infidel to the victories of Muhammad Ghori's contemporary Saladin over the Christian Franks of Syria and Palestine. It's been suggested that the primary motivation was booty.</p>
Qadi Nasir al-Din 'Umar accompanied Timur on his invasion of 1398 to Delhi. He produced a work, now lost, Ruz-Nama-yi Futuhdat-i Hindustan. Abridged versions of it have survived. In turn, they have been incorporated in other works, including the possibly fabricated memoir Malfuzat-I Timuri</a>. Timur's biographers say his motivation to invade Delhi was the merit he would gain as a ghazi by taking part in a religiously mandated holy war (jihad) against non-believers. Timur is said to have declared: "By the order of God and the Prophet it is incumbent upon me to make war upon these infidels and polytheists". It's been suggested that a crucial motivation was booty.</p>
Babur, a Chagathay Turk, claimed direct descent from Timur through his father. His mother traced her ancestry to the Mongol Genghis Khan. Babur's perspectives are included in his memoirs, Baburnama. For Islam's sake, he says, he wandered pagan lands, battled Hindus and destroyed idols, and thus became a ghazi. He also reveals more concrete evidence for motivations other than holy war or protection of the faith: he was displaced from his ancestral lands - Samarkand - and he wanted territory of his own. He decided this territory would be the claims of his forbears, that is Timur, to sovereignty east of the Indus. Babur meticulously recorded the financial potential of territories, making estimates of the tax yield of Sultan Ibrahim Lodi's north India in monetary terms.</p>
Delhi fell to Babur in April 1526.</p>
6.</h1>

  
  </figcaption>
  
</figure>


  
  The hill fortresses kept reverting to Hindu rulers, requiring recapture by subsequent Muslim rulers, as it was with Gwalior or Ranthambor.</figcaption>
  
</figure>
</p>
One primary source for life under the Delhi Sultanate, and the ambitions of its rulers, is the works left behind by its court historians: Barani, 'Isami, Amir Khusraw, and Shams-i Siraj. For external accounts, there's Ibn Batuta's Rehla, and records compiled under the Mamluks of Egypt and Mongol Persia. Ghurid historian Juzjani's Tabaqat-i Nasiri has sections on the early Delhi Sultanate. Finally, there are administrative texts such as Hajji 'Abd al-Hamid Ghaznawi's Dastur al-Albab, a treatise, among other things, on the subject of taxation.</p>
There are no contemporary Hindu narrative sources, except some inscriptions in stone or copper plate grants, although the references there to Mlecchas, Turushkas and Yavanas are vague.</p>
These sources reveal a society where violence and instability pervaded even elite circles, with purges one bad night or sultan away. Wealth offered no security when political favor could shift overnight. The Delhi Sultanate was a society heavily reliant on and shaped by slavery, and political elimination was a standard tool of governance.</p>
For military and administration, the Sultanate relied on Turkish slave guards, the ghulams. They were highly valued for their martial skills, received religious instruction, and rigorous military training. Slaves, both Turkish and Indian, were promoted to key administrative and military positions.</p>
Hindus were extensively enslaved during military campaigns, with many thousands brought back as booty and used for labor or sold. While the wealthy and powerful enjoyed more comforts and opportunities, their lives were far from secure. Muslim chroniclers, such as Hasan-i Nizami, proudly recorded the demolition of Hindu temples and the reuse of their stone for constructing mosques, as seen in Delhi and Ajmer. Fakhr-i Mudabbir enthusiastically claimed that "Infidel towns have become cities of Islam. In place of images, they worship the Most High. Idol temples have become mosques". Sultan Firuz Shah, in particular, claimed to have destroyed newly built Hindu temples and replaced them with mosques, even repopulating a township with Muslim settlers, although this policy was mostly enforced in the vicinity of Delhi. Firuz Shah's edict, promising release from the jizya (poll-tax) to Hindus who converted to Islam, indicating a direct economic incentive and pressure to abandon native faiths. Amir Khusraw notes that the Turks could seize and buy and sell Hindus.</p>
A recurring theme in Barani's writings is that infidels must not be allowed to live in ease and affluence. He recounts a hypothetical conversation at Iltutmish's court where the ulama argued that Hindus should be given only the choice between death and Islam, as they were not "People of the Book". While the wazir deemed this impolitic due to the small Muslim population, the ulama insisted that the sultan should at least refrain from honoring Hindus or permitting idolatry.</p>
From the perspective of the Sultanate's chroniclers, Muslims constituted a common people, and Hindus were rarely seen as inherently interesting, but rather "only as converts, as capitation tax-payers, or as corpses".</p>
Indian Muslims and immigrant Muslims were generally treated differently, especially within the elite and court circles, with a clear preference for the latter, though Indian Muslims could rise in status. Ibn Battuta explicitly states that Muhammad bin Thughlaq preferred foreigners to the indigenous aristocracy and that the Indians in turn loathed the immigrant 'Khurasani' nobles. Barani often stigmatized Muhammad's Indian servitors as lowborn.</p>
And yet, the Hindus and Jain and non-Turks did survive and remained a significant majority of the population, often in positions vital to the functioning of the state.</p>
In rural areas, forests, and hills, remained "the domain of the infidel". Even in fortified towns, such as Gwalior, Hindus formed the majority of inhabitants. Muslim victory did not always entail the displacement of Hindu rulers. Often, Hindu chiefs, the rais and the ranas, were allowed to retain their thrones in return for paying tribute.</p>
The Turko-Persian nobility accumulated enormous debts to Hindu bankers and brokers, known as Multanis, who remained among the sultan's wealthiest subjects. Hindu clerks were needed to staff the administration even if under Muslim supervision.</p>
The majority of rulers within the Sultanate extended the status of 'protected peoples' (dhimmis) to Hindus, to the fury of the ulema. This status, in return for acceptance of Muslim rule and payment of the jizya (poll-tax), theoretically offered protection of life and property. This was a pragmatic adaptation as in the Islamic lands outside of India, dhimmi status was applied only to monotheistic people of the Book.</p>
There were some periods of accommodation and royal favor, based on any one Sultan's personal disposition. Muhammad bin Thughlaq was often interested in Hindu practices, attending the Holi festival and frequenting jogis, though that made him an apostate in the eyes of our historians, 'Isami and Barani.</p>
Muslim rulers had adopted distinctly Indian practices, such as riding elephants and consulting Hindu astrologers. Donations of tax-free land to Brahmans, Jains, jogis, and temples, recorded extensively in the later Mughal era, often refer to renewals or extensions of grants made by earlier Muslim rulers, indicating continuity in Hindu religious and landholding institutions even through periods of Muslim rule.</p>
7.</h1>
The Delhi Sultanate established a historical precedent: the first and sole bastion of Muslim power in the Indian subcontinent for a hundred years. If you take seriously the argument that they prevented the Mongols from invading deeper into the country, one consequence of this pressure from the Mongols was administrative reforms and centralization of a sort not seen in India before.</p>
The Turks and their situation - smaller ruling elites in a sea of hostile natives - accelerated the process of urbanization across much of northern India. Small market towns and large garrison cities grew in number throughout the Ganges basin. The need to maintain expensive horses and well-trained soldiers required the regular availability of cash to pay for them. The new urban centers expanded opportunities to trade. These changes fostered the development of a money economy and an expansion in craft production.</p>
Iltutmish's consolidation of power allowed him to launch a new coinage system based on the pure silver tanga, something that fostered economic integration within the expanded Sultanate. One major innovation was the iqta' system, a transferable revenue assignment in lieu of salary for military service. This became hereditary, allowing positions, titles, and emoluments to pass from father to son. It would influence later feudal systems and the rise of powerful regional families, and eventually the Mansabdari system of the Mughals, and in turn the Zamindari system across the country via the East India Company. Alauddin Khalji's harsh, roughshod introduction of a highly centralized and rigorous revenue administration based on measurement and cash payments, marked a fundamental departure from earlier decentralized, agrarian systems in North India. Muhammad bin Tughluq's administration directly colonized the northern Deccan with North Indian immigrant-settlers between the Narmada and Krishna Rivers and assimilated indigenous chieftains as iqta'dars.</p>
The Sultanate's internal conflicts, Mongol invasions, and policies of expansion to direct rule led to the secession of major provinces (e.g., Bengal, Deccan, Madurai) very quickly. Deccan insubordination would prove enduring. The Deccan would always remain another frontier to Delhi, including for the Mughals. The Mughal Empire would eventually unravel trying to get its Deccan holdings under central control. Native imperial authority would pass from a Deccan power, the Marathas, a confederacy no less, to another Deccan power, the East India Company, operating from Bombay and Madras. Another Deccan power, the Hyderabad State, was a significant regional power and its Nizam would at one point be the richest man in the world.</p>
The Delhi Sultanate, it is fair to say, established a template for Muslim rule in India. Military elites would be drawn from Persia or Central Asia. The Sultan, while the</em> defender of Islam, would accommodate his Hindu majorities. And there would be persistent attempts at taming the Deccan, and this would devolve into depending on household estates for control and revenue. That is, dependence on the Zamindars, and the title holders such as Patedars and Deshmukhs. This pattern of ambitious central authority ultimately constrained by local realities is certainly a feature of Indian governance, then and now.</p>
After trudging through this book, there's a claim I feel bold enough to make. The Delhi Sultanate's cycles of conquest, revolt, and reconquest initiated a sustained and contentious dynamic between a powerful northern imperial authority and diverse Indian regional polities. This period saw the imposition of tribute and the adoption of some external administrative or cultural forms, but frequently provoked strong resistance. The Mughal Empire inherited and deepened this cycle with its tributaries and successor states and conditioned a lot of northern and deccani Indians to view Delhi as a symbolic authority of imperial power. This was leveraged by the Marathas, the British East India Company, and the modern Indian state for their own legitimization and internal consolidation.</p>


System Design 2
2025-09-01T00:00:00+00:00
Design a Generic Batch Processing System</h1>
Designing a batch processing system involves handling large volumes of work items with a focus on reliability, fault tolerance, and efficient resource utilization. This system enables users to submit collections of work items for distributed processing with comprehensive progress tracking and failure recovery.</p>
Requirements</h2>
Functional Requirements</h3>

Users should be able to submit batch jobs with multiple work items.</strong> Each batch job contains a collection of items to be processed by the same task type with different parameters.</li>
The system should distribute work items across multiple workers for parallel processing.</strong> Large batches should be split into smaller work units for efficient distribution.</li>
The system should track progress at both batch and item levels.</strong> Users need visibility into overall batch completion and individual item status.</li>
The system should handle partial failures gracefully.</strong> When some items in a batch fail, the system should retry failed items while continuing to process successful ones.</li>
Users should be able to monitor batch execution status and retrieve results.</strong> This includes real-time progress updates and final result aggregation.</li>
</ul>
Non-Functional Requirements</h3>

Scalability</strong>: Handle up to 50,000 concurrent batches with 10 million total work items processing simultaneously</li>
Durability</strong>: No work items should be lost due to system failures; all progress must be recoverable</li>
At-least-once processing</strong>: Each work item is guaranteed to be processed at least once, with idempotency handling</li>
Fault tolerance</strong>: System continues operating with individual component failures</li>
Cost efficiency</strong>: Auto-scaling capabilities to minimize resource costs during low utilization</li>
</ul>
Capacity Estimation</h3>

Peak load</strong>: 50,000 batches/hour, average 200 items per batch = 10M items/hour</li>
Processing rate</strong>: ~2,800 items/second sustained</li>
Storage</strong>: ~100GB metadata, 10TB temporary work data</li>
Worker fleet</strong>: 1,000-5,000 workers depending on item complexity</li>
</ul>
Core Entities</h2>

BatchJob</strong>: A user-submitted collection of work items sharing the same task type</li>
WorkItem</strong>: An individual unit of work within a batch, containing input parameters and tracking state</li>
Task</strong>: A reusable processing template that defines how to execute work items</li>
BatchExecution</strong>: Runtime instance tracking the overall progress and status of a batch</li>
WorkerAssignment</strong>: Tracks which worker is processing which work items for failure recovery</li>
</ul>
System Interface</h2>
Batch Management API</h3>
POST /batches
</span>{
</span>  "task_id": "image_resize",
</span>  "batch_name": "user_avatars_q4_2024",
</span>  "work_items": [
</span>    {"item_id": "img_1", "input_url": "s3://...", "params": {...}},
</span>    {"item_id": "img_2", "input_url": "s3://...", "params": {...}}
</span>  ],
</span>  "batch_config": {
</span>    "failure_threshold": 0.05,  // Allow 5% failure rate
</span>    "retry_limit": 3,
</span>    "timeout_seconds": 300
</span>  }
</span>}
</span>Response: {"batch_id": "batch_123", "estimated_completion": "2024-01-15T10:30:00Z"}
</span></code></pre>
GET /batches/{batch_id}/status
</span>Response: {
</span>  "batch_id": "batch_123",
</span>  "status": "RUNNING",
</span>  "progress": {
</span>    "total_items": 1000,
</span>    "completed": 750,
</span>    "failed": 25,
</span>    "retrying": 10,
</span>    "pending": 215
</span>  },
</span>  "estimated_completion": "2024-01-15T10:25:00Z"
</span>}
</span></code></pre>
GET /batches/{batch_id}/items/{item_id}
</span>Response: {
</span>  "item_id": "img_1",
</span>  "status": "COMPLETED",
</span>  "worker_id": "worker_456",
</span>  "start_time": "2024-01-15T10:00:00Z",
</span>  "end_time": "2024-01-15T10:02:00Z",
</span>  "output_location": "s3://results/img_1_thumb.jpg"
</span>}
</span></code></pre>
Data Flow</h2>

Batch Submission</strong>: User submits batch job through API Gateway → Batch Management Service</li>
Work Item Creation</strong>: Batch Management Service creates individual WorkItem records and queues them</li>
Work Distribution</strong>: Workers pull work items from queue and claim ownership</li>
Progress Tracking</strong>: Workers update item status throughout processing lifecycle</li>
Completion Detection</strong>: Batch Coordinator monitors progress and determines batch completion</li>
Result Aggregation</strong>: Completed batch results are aggregated and made available to users</li>
</ol>
High-Level Design</h2>
Core Components</h3>


API Gateway</strong>: Entry point handling authentication, rate limiting, and request routing</p>
</li>

Batch Management Service</strong>:</p>

Creates and validates batch jobs</li>
Splits batches into individual work items</li>
Manages batch lifecycle and configuration</li>
</ul>
</li>

Work Item Database</strong>:</p>

Choice</strong>: PostgreSQL with read replicas for metadata consistency</li>
Schema</strong>:

batches</code>: batch_id, task_id, status, config, created_at, completed_at</li>
work_items</code>: item_id, batch_id, status, worker_id, input_data, output_location, retry_count, last_error</li>
worker_assignments</code>: assignment_id, worker_id, item_id, claimed_at, heartbeat_at</li>
</ul>
</li>
</ul>
</li>

Work Queue</strong>:</p>

Choice</strong>: AWS SQS for reliability and auto-scaling</li>
Separate queues for new work, retries, and dead letters</li>
Message contains: item_id, batch_id, task_id, input_parameters</li>
</ul>
</li>

Worker Pool</strong>:</p>

Choice</strong>: ECS with auto-scaling based on queue depth</li>
Workers are stateless and task-agnostic</li>
Implement heartbeat mechanism for failure detection</li>
</ul>
</li>

Batch Coordinator Service</strong>:</p>

Monitors batch progress and determines completion</li>
Handles retry logic for failed items</li>
Manages batch-level timeouts and failure thresholds</li>
</ul>
</li>

Data Storage</strong>:</p>

Input/Output</strong>: S3 for large data files with presigned URLs</li>
Results</strong>: Structured results in PostgreSQL, large outputs in S3</li>
</ul>
</li>

Monitoring & Alerting</strong>:</p>

CloudWatch for infrastructure metrics</li>
Custom dashboards for batch processing KPIs</li>
Alerts for stuck batches, high failure rates, queue backlogs</li>
</ul>
</li>
</ol>
Workflow</h3>


Batch Creation</strong>:</p>

Batch Management Service validates request and creates batch record</li>
Individual work items are created in database with PENDING status</li>
Work items are queued for processing</li>
</ul>
</li>

Work Processing</strong>:</p>

Workers poll queue and claim work items (update worker_id and status to CLAIMED)</li>
Workers send periodic heartbeats during processing</li>
Upon completion, workers update status and output location</li>
</ul>
</li>

Progress Monitoring</strong>:</p>

Batch Coordinator periodically scans active batches</li>
Calculates completion percentage and estimates completion time</li>
Detects stuck workers via missed heartbeats and reassigns work</li>
</ul>
</li>

Failure Handling</strong>:</p>

Failed items are retried up to configured limit</li>
Items exceeding retry limit are marked FAILED</li>
Batch completion determined by failure threshold configuration</li>
</ul>
</li>
</ol>
Deep Dives</h2>
1. Work Distribution and Load Balancing</h3>
Challenge</strong>: Efficiently distribute work items to prevent worker starvation and handle varying processing times.</p>
Solution</strong>:</p>

Use SQS FIFO queues partitioned by batch_id to maintain order for dependent items</li>
Implement worker prefetch limits to prevent hoarding</li>
Dynamic batching: workers can claim multiple small items or single large items based on estimated processing time</li>
</ul>
Work Claiming Protocol</strong>:</p>
-- Atomic work claiming
</span>UPDATE work_items
</span>SET status = </span>'</span>CLAIMED</span>'</span>, worker_id = $</span>1</span>, claimed_at = NOW()
</span>WHERE item_id IN (
</span>  SELECT item_id FROM work_items
</span>  WHERE status = </span>'</span>PENDING</span>' </span>AND batch_id = $</span>2
</span>  LIMIT $</span>3</span> FOR UPDATE SKIP LOCKED
</span>)
</span>RETURNING item_id, input_data;
</span></code></pre>
2. Failure Detection and Recovery</h3>
Worker Failure Detection</strong>:</p>

Workers send heartbeats every 30 seconds while processing</li>
Batch Coordinator detects missed heartbeats and marks items as ABANDONED</li>
Abandoned items are re-queued after updating retry_count</li>
</ul>
Partial Failure Handling</strong>:</p>

Individual item failures don't affect other items in the batch</li>
Failed items are retried with exponential backoff</li>
Persistent failures are sent to dead letter queue for manual inspection</li>
</ul>
Recovery Scenarios</strong>:</p>

Worker crash</strong>: Heartbeat timeout triggers work reassignment</li>
Network partition</strong>: Worker continues processing; coordinator detects timeout and may create duplicate work (handled by idempotency)</li>
Database failure</strong>: Work queue persists items; processing continues when database recovers</li>
</ul>
3. Batch Completion Logic</h3>
Completion Criteria</strong>:</p>

All items completed successfully, OR</li>
Failure rate exceeds configured threshold (batch marked FAILED), OR</li>
Batch timeout exceeded</li>
</ul>
Completion Detection</strong>:</p>
-- Efficient batch status calculation
</span>SELECT
</span>  </span>b</span>.</span>batch_id</span>,
</span>  COUNT(</span>*</span>) as total_items,
</span>  COUNT(CASE WHEN </span>wi</span>.</span>status </span>= </span>'</span>COMPLETED</span>' </span>THEN </span>1 </span>END) as completed,
</span>  COUNT(CASE WHEN </span>wi</span>.</span>status </span>= </span>'</span>FAILED</span>' </span>THEN </span>1 </span>END) as failed
</span>FROM batches b
</span>JOIN work_items wi ON </span>b</span>.</span>batch_id </span>= </span>wi</span>.</span>batch_id
</span>WHERE </span>b</span>.</span>status </span>= </span>'</span>RUNNING</span>'
</span>GROUP BY </span>b</span>.</span>batch_id
</span>HAVING COUNT(</span>*</span>) = COUNT(CASE WHEN </span>wi</span>.</span>status </span>IN (</span>'</span>COMPLETED</span>'</span>, </span>'</span>FAILED</span>'</span>) THEN </span>1 </span>END);
</span></code></pre>
4. Scaling and Performance</h3>
Database Scaling</strong>:</p>

Read replicas for status queries</li>
Partition work_items table by batch_id for large batches</li>
Connection pooling and prepared statements</li>
</ul>
Queue Scaling</strong>:</p>

SQS auto-scales to handle message volume</li>
Use multiple queues for different priority levels</li>
Implement message batching for high-throughput scenarios</li>
</ul>
Worker Auto-Scaling</strong>:</p>

Scale workers based on queue depth and message age</li>
Use spot instances for cost optimization</li>
Implement graceful shutdown for worker termination</li>
</ul>
Performance Optimizations</strong>:</p>

Batch database updates for progress tracking</li>
Use Redis for high-frequency heartbeat storage</li>
Implement result caching for repeated status queries</li>
</ul>
5. Monitoring and Observability</h3>
Key Metrics</strong>:</p>

Batch throughput (batches/hour, items/second)</li>
Processing latency (time from submission to completion)</li>
Failure rates by task type and error category</li>
Worker utilization and queue depth</li>
Cost per processed item</li>
</ul>
Alerting Thresholds</strong>:</p>

Queue depth > 10,000 items (scale up workers)</li>
Batch failure rate > 10% (investigate task issues)</li>
Worker heartbeat gaps > 5 minutes (check worker health)</li>
Database connection pool exhaustion</li>
</ul>
Dashboards</strong>:</p>

Real-time batch processing overview</li>
Historical performance trends</li>
Cost analysis and optimization opportunities</li>
Error analysis and retry patterns</li>
</ul>
This architecture provides a robust foundation for batch processing that handles the complexities of work distribution, failure recovery, and progress tracking at scale while maintaining cost efficiency and operational simplicity.</p>
ChatGPT with Inference Batching</h1>
Understanding the Problem</h2>
What are we building?</strong> A ChatGPT-like system that efficiently utilizes a pool of GPU servers for text inference. Each GPU server has a fixed latency for processing batches of 1-100 strings, making request aggregation the key optimization strategy.</p>
Functional Requirements</h3>
Core Requirements</strong></p>

Users should be able to submit text prompts and receive generated responses</li>
The system should aggregate individual requests into batches for GPU processing</li>
Responses should be correlated back to original users accurately</li>
The system should handle variable request arrival patterns efficiently</li>
</ol>
Below the line (out of scope)</strong></p>

Real-time streaming responses (token-by-token generation)</li>
Conversation history and context management</li>
User authentication and authorization</li>
Advanced prompt engineering or content filtering</li>
</ul>
Non-Functional Requirements</h3>
Core Requirements</strong></p>

High GPU Utilization</strong>: Maximize batch sizes (target: 90+ requests per batch) to optimize cost efficiency</li>
Acceptable User Latency</strong>: Balance batching delays with user experience (target: <500ms total response time)</li>
High Throughput</strong>: Handle thousands of concurrent requests efficiently</li>
Scalability</strong>: System should scale horizontally as demand increases</li>
Fault Tolerance</strong>: Basic resilience acceptable; memory-based components are fine per problem constraints</li>
</ol>
Below the line (out of scope)</strong></p>

Strong consistency guarantees</li>
Persistent storage requirements</li>
Multi-region deployment</li>
Advanced security features</li>
</ul>
The Set Up</h2>
Planning the Approach</h3>
This system's core challenge is request aggregation</strong> - efficiently batching individual user requests to maximize GPU utilization while maintaining acceptable response times. The fixed GPU latency characteristic means sending 1 request takes the same time as sending 100 requests.</p>
Defining the Core Entities</h3>
Request</strong>: An individual user prompt with associated metadata (requestId, timestamp, client connection info)</p>
Batch</strong>: A collection of 1-100 requests sent to GPU servers as a single inference call</p>
Response</strong>: Generated text corresponding to each request in a batch</p>
System Interface</h3>
Simple HTTP REST API for request-response pattern:</p>
POST /generate
</span>{
</span>  "prompt": "What is the capital of France?",
</span>  "max_wait_ms": 200  // Optional: client timeout preference
</span>}
</span>
</span>Response:
</span>{
</span>  "response": "The capital of France is Paris.",
</span>  "batch_size": 87,
</span>  "processing_time_ms": 245
</span>}
</span></code></pre>
High-Level Design</h2>
Core Architecture</h3>
[Clients] --HTTP--> [Load Balancer] ---> [Frontend Services] ---> [GPU Pool Manager] ---> [GPU Servers]
</span>                                              |                                              ^
</span>                                              v                                              |
</span>                                       [Batch Accumulator]                                  |
</span>                                       (In-Memory Queue)                                    |
</span>                                              |                                              |
</span>                                              +----------------------------------------------+
</span></code></pre>
Component Overview</h3>
Load Balancer</strong>: Distributes incoming HTTP requests across Frontend Service instances using round-robin or least-connections</p>
Frontend Service</strong>: Stateless HTTP servers that handle request batching and response correlation</p>
Batch Accumulator</strong>: In-memory data structure within each Frontend Service for collecting requests</p>
GPU Pool Manager</strong>: Simple load balancer that routes batches to available GPU servers</p>
GPU Servers</strong>: External inference servers with the fixed-latency API</p>
Data Flow</h2>
Request Processing Flow</h3>

Request Arrival</strong>: Client sends HTTP POST to load balancer with text prompt</li>
Load Distribution</strong>: Load balancer routes request to available Frontend Service instance</li>
Batch Accumulation</strong>: Frontend Service adds request to in-memory batch accumulator</li>
Batch Triggering</strong>: Batch is sent to GPU Pool Manager when either:

100 requests accumulated (size trigger)</li>
Timeout reached (75-100ms default)</li>
</ul>
</li>
GPU Routing</strong>: GPU Pool Manager selects least-loaded GPU server</li>
Inference Processing</strong>: GPU server processes entire batch with fixed latency</li>
Response Disaggregation</strong>: GPU Pool Manager returns batch response to Frontend Service</li>
Response Correlation</strong>: Frontend Service maps individual responses back to waiting HTTP requests</li>
Client Response</strong>: Each client receives their specific response over original HTTP connection</li>
</ol>
Key Data Transformations</h3>
Individual Request → Batch Request</strong>:</p>

Input: Single prompt string per HTTP request</li>
Transform: Collect into list of 1-100 prompt strings</li>
Output: Batch API call to GPU with list of strings</li>
</ul>
Batch Response → Individual Responses</strong>:</p>

Input: List of response strings from GPU</li>
Transform: Map each response to original request using stored correlation</li>
Output: Individual HTTP responses to waiting clients</li>
</ul>
Potential Deep Dives</h2>
1) How do we optimize the batching timeout strategy?</h3>
The timeout value creates a fundamental trade-off between user latency and GPU utilization. This is the most critical optimization lever in the system.</p>
Dynamic Timeout Adjustment</strong>:
During high traffic periods, shorter timeouts (20-30ms) work well because batches fill quickly. During low traffic, longer timeouts (100ms) are necessary to collect reasonable batch sizes. The system monitors request arrival rates and adjusts timeouts accordingly.</p>
Traffic Pattern Analysis</strong>:</p>

Track requests per second over rolling windows</li>
High traffic (>50 req/sec): Use 25ms timeout</li>
Medium traffic (10-50 req/sec): Use 50ms timeout</li>
Low traffic (<10 req/sec): Use 100ms timeout</li>
</ul>
User-Specified Preferences</strong>:
Allow clients to specify maximum wait time in their requests. Premium users might get priority treatment with faster timeouts, while batch processing jobs can accept longer delays.</p>
2) How do we handle uneven load distribution across Frontend Services?</h3>
Each Frontend Service maintains its own batch accumulator, which can lead to suboptimal batching if traffic is unevenly distributed.</p>
Load Balancer Strategies</strong>:</p>

Least-connections routing ensures requests go to Frontend Services with fewer active batches</li>
Health checks prevent routing to overloaded instances</li>
Sticky sessions are not needed since each request is independent</li>
</ul>
Cross-Service Batch Sharing</strong>:
Frontend Services could share accumulated requests through a lightweight coordination mechanism. When one service has a partial batch approaching timeout, it could request additional items from other services' accumulators.</p>
Capacity Planning</strong>:
Monitor batch utilization metrics across all Frontend Services. If average batch sizes drop below targets, this indicates either uneven load distribution or insufficient traffic density requiring more aggressive timeout tuning.</p>
3) How do we ensure GPU pool efficiency and handle failures?</h3>
GPU Load Balancing</strong>:
The GPU Pool Manager tracks pending requests per GPU server and routes new batches to the least loaded server. This prevents GPU hotspots while maintaining utilization.</p>
Failure Handling</strong>:
When a GPU server fails or times out, the batch is immediately retried on a different server. The correlation mapping is preserved, so clients don't experience duplicate responses. Circuit breaker patterns prevent sending traffic to consistently failing GPUs.</p>
Health Monitoring</strong>:
Track GPU response times, error rates, and processing capacity. Servers showing degraded performance are temporarily removed from rotation. Automatic re-inclusion occurs after a recovery period with successful health checks.</p>
4) How do we scale the system horizontally?</h3>
Frontend Service Scaling</strong>:
Frontend Services are stateless and scale independently. Each instance maintains only its local batch accumulator and correlation mappings. Adding instances increases both batch accumulation capacity and HTTP request handling capacity.</p>
Memory Management</strong>:
Each Frontend Service has bounded memory usage since batch accumulators have maximum sizes and correlation mappings are short-lived. Monitor memory utilization and set limits to prevent resource exhaustion.</p>
GPU Pool Scaling</strong>:
Adding GPU servers increases overall processing capacity. The GPU Pool Manager automatically includes new servers in routing decisions. Removal requires draining existing batches before taking servers offline.</p>
5) How do we handle different request priorities and patterns?</h3>
Priority Tiers</strong>:
Implement separate batch accumulators for different priority levels. High-priority requests get smaller batch sizes and shorter timeouts, while standard requests wait for full batches. This balances responsiveness for critical users with efficiency for regular traffic.</p>
Predictive Batching</strong>:
Analyze historical traffic patterns to predict peak periods. Pre-warm batch accumulators during expected traffic surges and adjust timeout strategies based on time-of-day patterns.</p>
Burst Handling</strong>:
During traffic spikes, the system can temporarily reduce batch size requirements to maintain responsiveness. Monitoring request queue depths helps detect when burst mode should activate.</p>
6) How do we monitor and optimize system performance?</h3>
Key Metrics</strong>:</p>

Average batch utilization percentage</li>
Request latency distribution (P50, P95, P99)</li>
GPU server utilization and response times</li>
Timeout frequency vs size-triggered batches</li>
Request correlation accuracy</li>
</ul>
Performance Optimization</strong>:
Continuously analyze the relationship between timeout values and batch sizes. Optimal timeout tuning maximizes the product of batch efficiency and user satisfaction. A/B testing different timeout strategies helps identify optimal configurations.</p>
Alerting Strategy</strong>:
Monitor for degraded batch utilization, increased latency, or GPU failures. Set thresholds that trigger alerts before user experience degrades significantly.</p>
Design ChatGPT Playground</h1>
Understanding the Problem</h2>
🎮 What is a ChatGPT Playground?</strong></p>
A ChatGPT Playground is an interactive environment where users can experiment with Large Language Models (LLMs) by sending prompts and receiving responses. Unlike standard chat interfaces, playgrounds provide advanced configuration options like adjusting model parameters (temperature, max tokens, top-p), switching between different models, and managing multiple conversation sessions for different experiments.</p>
Functional Requirements</h3>
Core Requirements</strong></p>

Users should be able to send text prompts to an LLM and receive streaming responses in real-time.</li>
Users should be able to view and persist conversation history within playground sessions.</li>
Users should be able to configure LLM parameters (temperature, max tokens, model selection, system prompts).</li>
Users should be able to create, name, and switch between multiple playground sessions.</li>
</ol>
Below the line (out of scope)</strong></p>

Sharing playground sessions with other users</li>
Advanced prompt templating or batch processing</li>
Custom model fine-tuning or training</li>
Integration with external APIs or data sources</li>
</ul>
Non-Functional Requirements</h3>
Core Requirements</strong></p>

The system should provide low latency responses with first token arriving within 500ms of prompt submission.</li>
The system should scale to support 50,000 concurrent users with 5,000 prompts per second at peak.</li>
The system should be highly available (99.9% uptime) with eventual consistency acceptable for conversation history.</li>
The system should guarantee data durability - no conversation data should be lost due to system failures.</li>
The system should handle WebSocket connections gracefully with automatic reconnection and session recovery.</li>
</ol>
Below the line (out of scope)</strong></p>

Strong consistency guarantees across all operations</li>
Sub-100ms response times</li>
Cross-region data replication</li>
</ul>
Here's how it might look on your whiteboard:</p>
Playground Requirements</strong></p>
The Set Up</h2>
Planning the Approach</h3>
For this problem, we'll follow the delivery framework focusing on real-time streaming capabilities and session management. The core challenge is handling persistent WebSocket connections while maintaining conversation state and integrating with external LLM providers reliably.</p>
Defining the Core Entities</h3>
User</strong>: Represents an individual using the playground platform, with authentication and usage tracking.</p>
PlaygroundSession</strong>: A named workspace containing LLM configuration and associated conversation history. Users can create multiple sessions for different experiments.</p>
Message</strong>: Individual prompts (user) or responses (assistant) within a session, with support for streaming token metadata.</p>
LLMConfiguration</strong>: Parameters controlling LLM behavior (model, temperature, max_tokens, system_prompt) associated with each session.</p>
StreamingState</strong>: Transient state tracking active streaming responses, token buffers, and connection status.</p>
Core Entities</p>
The API</h3>
We'll use REST for session management and WebSockets for real-time prompt/response interaction:</p>
Session Management (REST)</strong></p>
POST /sessions
</span>{
</span>  "name": "My Experiment",
</span>  "config": {
</span>    "model": "gpt-4",
</span>    "temperature": 0.7,
</span>    "max_tokens": 1000
</span>  }
</span>}
</span>
</span>GET /sessions/{sessionId}
</span>Response: {
</span>  "session_id": "uuid",
</span>  "name": "My Experiment",
</span>  "config": {...},
</span>  "messages": [...] // paginated
</span>}
</span>
</span>PUT /sessions/{sessionId}/config
</span>{
</span>  "temperature": 0.9,
</span>  "max_tokens": 500
</span>}
</span></code></pre>
Real-time Interaction (WebSocket)</strong></p>
// Connection
</span>ws://api.playground.com/sessions/{sessionId}/chat
</span>
</span>// Client → Server
</span>{
</span>  "type": "prompt",
</span>  "content": "Explain quantum computing",
</span>  "message_id": "uuid"
</span>}
</span>
</span>// Server → Client (streaming)
</span>{
</span>  "type": "token",
</span>  "content": "Quantum",
</span>  "message_id": "uuid",
</span>  "position": 0
</span>}
</span>{
</span>  "type": "token",
</span>  "content": " computing",
</span>  "message_id": "uuid",
</span>  "position": 1
</span>}
</span>{
</span>  "type": "complete",
</span>  "message_id": "uuid",
</span>  "full_response_id": "uuid"
</span>}
</span></code></pre>
High-Level Design</h2>
1) Users should be able to send prompts and receive streaming responses</h3>
The core challenge is maintaining persistent WebSocket connections while orchestrating calls to external LLM providers and streaming responses back to clients.</p>
Pattern: Real-time Updates</h6>
Streaming LLM responses represent a classic real-time updates</strong> pattern where servers must push data to clients as it becomes available. This requires persistent connections and careful handling of connection lifecycle, backpressure, and failure recovery.</p>
Our architecture centers around a WebSocket Gateway that manages persistent connections and a Chat Service that orchestrates LLM interactions:</p>
Basic Streaming Architecture</p>
WebSocket Gateway</strong>: Manages persistent connections, handles authentication, and routes messages between clients and backend services. Maintains connection state in memory with Redis backing for connection recovery.</p>
Chat Service</strong>: Stateless service that processes prompts, calls the LLM Provider, and streams responses back through the WebSocket Gateway. Handles message persistence and session state management.</p>
LLM Provider</strong>: External service (OpenAI, Anthropic, etc.) that processes prompts and returns streaming responses.</p>
Flow for sending a prompt:</strong></p>

Client sends prompt via WebSocket connection to Gateway</li>
Gateway forwards to Chat Service with connection identifier</li>
Chat Service validates session, retrieves LLM config from database</li>
Chat Service calls LLM Provider with streaming enabled</li>
As tokens arrive from LLM Provider, Chat Service immediately forwards to Gateway</li>
Gateway streams tokens to client via WebSocket</li>
Chat Service persists complete message to database once streaming finishes</li>
</ol>
2) Users should be able to view and persist conversation history</h3>
Conversation persistence requires careful coordination between real-time streaming and durable storage, with support for session recovery and message replay.</p>
We'll use PostgreSQL with a conversation-optimized schema:</p>
CREATE TABLE </span>playground_sessions</span> (
</span>  session_id UUID </span>PRIMARY KEY</span>,
</span>  user_id UUID NOT NULL,
</span>  name </span>VARCHAR</span>(</span>255</span>) NOT NULL,
</span>  config JSONB NOT NULL,
</span>  created_at </span>TIMESTAMP DEFAULT</span> NOW(),
</span>  updated_at </span>TIMESTAMP DEFAULT</span> NOW()
</span>);
</span>
</span>CREATE TABLE </span>messages</span> (
</span>  message_id UUID </span>PRIMARY KEY</span>,
</span>  session_id UUID NOT NULL,
</span>  role </span>VARCHAR</span>(</span>20</span>) NOT NULL, </span>-- 'user' or 'assistant'
</span>  content </span>TEXT </span>NOT NULL,
</span>  tokens JSONB, </span>-- streaming metadata, token timing
</span>  sequence_number </span>INTEGER </span>NOT NULL,
</span>  created_at </span>TIMESTAMP DEFAULT</span> NOW(),
</span>  UNIQUE(session_id, sequence_number)
</span>) PARTITION BY HASH(session_id);
</span>
</span>-- Index for efficient conversation retrieval
</span>CREATE INDEX </span>idx_messages_session_sequence
</span>ON messages (session_id, sequence_number);
</span></code></pre>
Message Persistence Strategy:</strong></p>

Optimistic Writing</strong>: Start persisting user prompts immediately when received</li>
Streaming Buffer</strong>: Accumulate assistant response tokens in memory during streaming</li>
Atomic Completion</strong>: Persist complete assistant response as single transaction when streaming finishes</li>
Sequence Guarantees</strong>: Use incrementing sequence numbers per session to ensure message ordering</li>
</ol>
3) Users should be able to configure LLM parameters</h3>
LLM configuration management requires handling parameter validation, hot-swapping configurations mid-session, and maintaining compatibility across different model providers.</p>
Session Configuration</p>
Configuration Service</strong>: Dedicated service managing LLM parameters with validation rules, default values, and provider-specific constraints.</p>
Configuration validation examples:</strong></p>

Temperature: 0.0-1.0 for OpenAI models, 0.0-2.0 for Anthropic</li>
Max tokens: Provider-specific limits (4096 for GPT-3.5, 100k for Claude)</li>
Model availability: Check provider APIs for supported models</li>
</ul>
Hot Configuration Updates:</strong></p>
-- Track configuration versions for rollback
</span>CREATE TABLE </span>config_versions</span> (
</span>  version_id UUID </span>PRIMARY KEY</span>,
</span>  session_id UUID NOT NULL,
</span>  config JSONB NOT NULL,
</span>  created_at </span>TIMESTAMP DEFAULT</span> NOW()
</span>);
</span></code></pre>
When users update configurations mid-conversation, the Chat Service validates parameters against provider constraints and applies changes to subsequent prompts without affecting in-flight requests.</p>
4) Users should be able to create and switch between multiple sessions</h3>
Session management requires efficient session listing, context switching, and maintaining session state across WebSocket reconnections.</p>
Session Management Architecture</p>
Session Router</strong>: Routes WebSocket connections to appropriate Chat Service instances based on session ID, enabling sticky session behavior for connection management.</p>
Session State Recovery</strong>: When clients reconnect or switch sessions:</p>

Authenticate WebSocket connection with session ID</li>
Retrieve last N messages from database for context</li>
Check for any incomplete streaming responses</li>
Resume or restart as appropriate</li>
</ol>
Efficient session switching:</strong></p>
// Client-side session switching
</span>websocket</span>.send({
</span>  type: </span>"</span>switch_session</span>"</span>,
</span>  new_session_id: </span>"</span>uuid</span>"</span>,
</span>  resume_from_message: </span>"</span>last_message_id</span>"</span>,
</span>});
</span></code></pre>
Potential Deep Dives</h2>
1) How do we handle WebSocket connection reliability and session recovery?</h3>
WebSocket connections are inherently fragile - networks drop, browsers close, mobile apps background. Our system needs robust connection management and seamless recovery.</p>
Approach: Heartbeat and Reconnection</h6>
Implement client-server heartbeating with exponential backoff reconnection. Client sends ping every 30 seconds, server responds with pong. If either side misses 3 consecutive heartbeats, assume connection is dead.</p>
// Client-side connection management
</span>class </span>PlaygroundWebSocket </span>{
</span>  </span>connect</span>(</span>sessionId</span>) {
</span>    </span>this</span>.</span>ws </span>= new </span>WebSocket</span>(</span>`/sessions/${</span>sessionId</span>}/chat`</span>);
</span>    </span>this</span>.</span>startHeartbeat</span>();
</span>    </span>this</span>.</span>setupReconnection</span>();
</span>  }
</span>
</span>  </span>startHeartbeat</span>() {
</span>    </span>this</span>.</span>heartbeat </span>= setInterval(() </span>=> </span>{
</span>      </span>if </span>(</span>this</span>.</span>ws</span>.readyState === </span>WebSocket</span>.</span>OPEN</span>) {
</span>        </span>this</span>.</span>ws</span>.send(JSON.stringify({ type: </span>"</span>ping</span>" </span>}));
</span>      }
</span>    }, </span>30000</span>);
</span>  }
</span>
</span>  </span>setupReconnection</span>() {
</span>    </span>this</span>.</span>ws</span>.</span>onclose </span>= () </span>=> </span>{
</span>      </span>this</span>.</span>reconnectWithBackoff</span>();
</span>    };
</span>  }
</span>}
</span></code></pre>
Challenges</h6>
Heartbeating adds bandwidth overhead and complexity. Mobile clients may not handle backgrounding gracefully, leading to unnecessary reconnections.</p>
Approach: Stateless Recovery with Message Replay</h6>
Store connection state in Redis with TTL. When clients reconnect, they provide their last received message ID, and the server replays any missed messages.</p>
Connection State Storage:</strong></p>
SET connection:{connectionId} {
</span>  "session_id": "uuid",
</span>  "user_id": "uuid",
</span>  "last_message_sequence": 42,
</span>  "streaming_message_id": "uuid" // if streaming was interrupted
</span>} EX 3600
</span></code></pre>
Recovery Protocol:</strong></p>

Client reconnects with last known message sequence</li>
Server queries Redis for connection state</li>
Server replays messages from database where sequence > last_known</li>
If streaming was interrupted, restart from LLM provider or serve cached partial response</li>
</ol>
Challenges</h6>
Redis becomes a single point of failure. Message replay can be expensive for long-lived sessions with extensive history.</p>
Approach: Connection Pooling with Sticky Sessions</h6>
Use consistent hashing to ensure users consistently connect to the same WebSocket Gateway instance. This enables in-memory connection state without Redis dependency.</p>
Each Gateway instance maintains:</p>
// In-memory connection registry
</span>const </span>connections </span>= new </span>Map</span>(); </span>// connectionId -> WebSocketConnection
</span>const </span>sessionConnections </span>= new </span>Map</span>(); </span>// sessionId -> Set<connectionId>
</span></code></pre>
Load balancer uses consistent hashing on session ID to route WebSocket upgrade requests to the same Gateway instance.</p>
Benefits</strong>: No external dependencies, faster connection recovery, simpler debugging.</p>
Trade-offs</strong>: Gateway failures lose all connection state, requiring clients to reconnect and replay from database.</p>
For our scale (50k concurrent users), consistent hashing with 10-20 Gateway instances provides good balance of reliability and simplicity.</p>
2) How do we handle LLM provider failures and rate limiting?</h3>
External LLM providers have API limits, experience outages, and may return errors. Our system needs graceful degradation and retry strategies.</p>
Pattern: External Service Integration</h6>
LLM provider integration demonstrates classic external service integration</strong> challenges including rate limiting, circuit breaking, and graceful degradation when dependencies fail.</p>
Circuit Breaker Pattern:</strong></p>
class </span>LLMProviderClient</span>:
</span>    </span>def </span>__init__</span>(</span>self</span>):
</span>        </span>self</span>.circuit_breaker = </span>CircuitBreaker</span>(
</span>            </span>failure_threshold</span>=</span>5</span>,  </span># failures before opening
</span>            </span>recovery_timeout</span>=</span>60</span>,  </span># seconds before trying again
</span>            </span>expected_exception</span>=LLMProviderError
</span>        )
</span>
</span>    @</span>circuit_breaker
</span>    </span>def </span>stream_completion</span>(</span>self</span>, </span>prompt</span>, </span>config</span>):
</span>        </span># Call external LLM provider
</span>        response = requests.</span>post</span>(
</span>            </span>f</span>"</span>{provider_url}</span>/completions</span>"</span>,
</span>            </span>json</span>={</span>"</span>prompt</span>"</span>: prompt, **config},
</span>            </span>stream</span>=True,
</span>            </span>timeout</span>=</span>30
</span>        )
</span>
</span>        </span>for </span>chunk </span>in </span>response.</span>iter_content</span>():
</span>            </span>yield </span>parse_token</span>(chunk)
</span></code></pre>
Provider Rate Limiting:</strong></p>
Implement token bucket rate limiting per API key:</p>
class </span>RateLimiter</span>:
</span>    </span>def </span>__init__</span>(</span>self</span>, </span>requests_per_minute</span>=</span>1000</span>):
</span>        </span>self</span>.bucket = </span>TokenBucket</span>(
</span>            </span>capacity</span>=requests_per_minute,
</span>            </span>refill_rate</span>=requests_per_minute / </span>60
</span>        )
</span>
</span>    </span>async def </span>acquire</span>(</span>self</span>):
</span>        </span>if </span>not </span>self</span>.bucket.</span>consume</span>(</span>1</span>):
</span>            </span>raise </span>RateLimitExceeded</span>(</span>"</span>Provider rate limit exceeded</span>"</span>)
</span></code></pre>
Graceful Degradation Strategies:</strong></p>

Cached Responses</strong>: For identical prompts with deterministic settings (temperature=0), serve cached responses</li>
Fallback Providers</strong>: Route to secondary LLM provider when primary fails</li>
Queue with SLA</strong>: Buffer requests for up to 10 seconds during provider issues</li>
User Communication</strong>: Send clear error messages explaining delays</li>
</ol>
3) How do we scale to handle 50,000 concurrent WebSocket connections?</h3>
WebSocket connections are stateful and memory-intensive. Scaling requires careful resource management and connection distribution.</p>
Connection Capacity Planning:</strong></p>
Each WebSocket connection consumes approximately:</p>

8KB TCP buffers (send/receive)</li>
2KB application state (user ID, session ID, last message tracking)</li>
1KB average message buffer for streaming</li>
</ul>
Total: ~11KB per connection × 50,000 = 550MB per Gateway instance.</p>
With modern servers having 32GB+ RAM, a single Gateway instance can handle 50k connections, but we'll use multiple instances for fault tolerance.</p>
Scaling Architecture:</strong></p>
Gateway Scaling</p>
WebSocket Gateway Scaling:</strong></p>

Deploy 5 Gateway instances behind L4 load balancer</li>
Each handles 10k connections (plenty of headroom)</li>
Use consistent hashing on session ID for sticky routing</li>
Auto-scale based on connection count metrics</li>
</ul>
Chat Service Scaling:</strong></p>

Stateless horizontal scaling based on CPU/memory usage</li>
Queue depth monitoring for LLM provider calls</li>
Circuit breaker integration prevents cascade failures</li>
</ul>
Database Connection Pooling:</strong></p>
With 5k prompts/second peak, database connections become critical:</p>
# Connection pool configuration
</span>DATABASE_POOL </span>= {
</span>    </span>"</span>min_connections</span>"</span>: </span>10</span>,
</span>    </span>"</span>max_connections</span>"</span>: </span>100</span>,
</span>    </span>"</span>connection_timeout</span>"</span>: </span>30</span>,
</span>    </span>"</span>idle_timeout</span>"</span>: </span>300
</span>}
</span></code></pre>
PostgreSQL Scaling Strategy:</strong></p>

Primary for writes (session creation, message persistence)</li>
Read replicas for conversation history retrieval</li>
Partition messages table by session_id hash for write distribution</li>
</ul>
4) How do we ensure data consistency and handle partial failures?</h3>
Distributed systems create opportunities for partial failures where some operations succeed while others fail, potentially leaving the system in inconsistent state.</p>
Message Persistence Consistency:</strong></p>
The critical consistency challenge occurs during streaming: what happens if we successfully start streaming to the client but fail to persist the complete response?</p>
Write-Ahead Logging with Compensation:</strong> Before starting LLM call, write message intent to database with PENDING status. Stream to client while buffering complete response. Then atomically update intent to COMPLETED and insert complete message in single transaction. Background cleanup job identifies stale PENDING intents and handles recovery.</p>
Idempotency for Duplicate Prompts:</strong> Clients may retry prompts during network issues. Implement idempotency using client-generated message IDs. Check for existing messages with same ID before processing and return cached responses for duplicates.</p>
Session State Consistency:</strong> Configuration updates during active streaming create race conditions. Use optimistic locking with version numbers on session configuration. Update operations include version checks and fail if concurrent modifications occurred.</p>
Distribute ML models in a Data Center</h1>
P2P</h2>

So it seems like one machine should download the model and distribute it to other machines.</li>
This looks suspiciously like p2p torrenting so I am going to approach it similarly.</li>
Machine 1 has to download the ML model fully. Then it should chunk it in let's say 4 mb chunks. Each chunk is SHA1 hashed and this machine keeps tracks of the chunks and the hashes associated with it. Let's call this the seed machine.</li>
The seed machine communicates with other machines telling there's a ML model to be downloaded. Since these machines are in the same DC, they know each other's IPs. We will have a dedicated gRPC server and client on each machine for the purpose of downloading ML models. So the seed machine starts by distributing the manifest to each of the machines.</li>
The other machines then start downloading the chunks. The very first sets of chunks are downloaded from the seed machine.</li>
Each machine periodically gossips with a few random peers to learn who has what chunks. When a machine needs a specific chunk, it picks a peer that has it (from its gossip knowledge) and downloads directly via gRPC.</li>
Eventually at some time, all the machines have all the chunks, and they can validate they indeed have that because each has a copy of the manifest file.</li>
</ol>
The trade off between gossip versus having some central tracker:</p>

network overhead in gossip, while with a central tracker, one can make efficient queries.</li>
gossip protocol is self-healing, while a central tracker is a single point of failure.</li>
</ol>
Pros: potentially fast due to parallelism. inherent fault tolerance: multiple sources for each chunk. scalable, work from 10 to 10_000 machines.</p>
Cons: Complex coordination (gossip, central tracker), unpredictable timing.</p>
Tree</h2>
If we know there are exactly N machines, then scaling is not important and maybe controlled behaviour is.
Each machine can pass the model to exactly one other machine in each round. So:</p>

Machine 1 -> Machine 2</li>
Machine 1 -> Machine 3, Machine 2 -> Machine 4</li>
Machine 1 -> Machine 5, Machine 2 -> Machine 6, Machine 3 -> Machine 7, Machine 4 -> Machine 8</li>
</ol>
For N machines, the number of rounds we will have is log2(N). For 100, that's at least 7 rounds.
Pros: There's no complex discovery here, and bandwith usage is optimal - each round, for a machine, it's committed to 100% upload or 100% download. Predictable timing and progress tracking.</p>
Cons: half the machines are idle in each round. If one node fails, sub tree fails. Load imbalance: Early nodes (closer to root) work harder than leaf nodes.</p>
Hybrid</h2>
Tree approach for first few nodes, then P2P file transfer for stragglers or failures.</p>
Spotify Top-K</h1>
Functional Requirements</p>

Real time Top K songs for all users.</li>
</ol>
Non-functional</p>

Real time - low latency.</li>
Availability over consistency over all</li>
</ul>
Data flow</h2>
Naive:</p>

Cassadandra for write heavy system, to store analytics.

Cassandra has CDC that writes to a commit log, files on disk.</li>
</ul>
</li>
CDC into OLAP (a dataware house, maybe clickhouse).

You must build a service that parses the commit logs and push changes to OLAP.</li>
Could even be Kafka.</li>
</ul>
</li>
Followed by hourly map reduce workers to compute states.</li>
</ul>
It's naive because it's not realtime. At least you diffeentiated between OLTP and OLAP.</p>
Better:</p>

User plays song -> Analytics service collects info. -> pushes this into Kafka.

Save the song id, the user id, the timestamp.</li>
</ul>
</li>
Stream processor like flink or Spark Streams consumes it in windows.

Pull the above data from kafka.</li>
Aggregate the counts every minute or 5 minutes.</li>
</ul>
</li>
</li>
</ul>
FB Live comments</h1>
High Level Data Flow</p>

User has a post id. He uses that to post a comment.</li>
The comment is processed by a CommentService, which
makes an entry in a db. The db records the user id,
comment id, post id, and timestamp.</li>
This comment is then made available to all users via
an SSE connection established by the clients

The client establishes the SSE GET request. The
service keeps the connection open and continuously
sends data to the client.</li>
</ul>
</li>
The service keeps track of a few things:

The user id/session id and the specific TCP connection object.</li>
The user id and the post id the user is listening to.</li>
Comments for a post id.</li>
</ul>
</li>
So when a comment comes in, a service should publish this comment
to everyone who is subscribing to a post. So a pub/sub is the means
of routing comments.</li>
We can use a Redis pub/sub (since Redis will push, fire once and forget)
and is fast. When a comment is posted, it's published to Redis, which
maps a post id to the user id/connection. An SSE service manages the SSE
connections, and listens to Redis channels.

The service subscribes to various channels based on the what the user
connections it's managing are interested in. It receives corresponding
messages, iterates though its in-memory list of active connections
and sends messages to those users.</li>
</ul>
</li>
</ol>


System Design 1
2025-08-31T00:00:00+00:00
Design Stripe</h1>
Requirements</h2>
Functional Requirements:</p>

Merchant can request payment</li>
Users should be able to make payment.</li>
Merchant (and user) must be able to view status update - pending, completed etc</li>
</ul>
Non-functional requirements:</p>

highly secure</li>
no data loss, even on failure.</li>
gurantee transaction and hence financial safety.</li>
high, bursty volume (10k qps?)</li>
</ul>
Core Entities</h2>

Merchant</li>
Intent
What's being asked and delivered, how much etc.</li>
Transaction

Record of money movement - charges, refunds, payout. disputes.</li>
</ul>
</li>
</ul>
Merchant -> Intent -> Transaction Record
If transaction failed, merchat will keep trying with the Intent ID.</p>
API</h2>
Merchant creates an intent that will used for payment</p>
POST</span> /payment-intent -> intentId
</span>{
</span> </span>"</span>amount</span>"</span>:</span> 1234, </span>"</span>currency</span>"</span>: </span>"</span>dollar</span>"</span>, </span>"</span>description</span>"</span>: </span>"</span>shoes</span>"
</span>}
</span></code></pre>
POST</span> transaction/{intentId}
</span>{
</span> </span>"</span>name</span>"</span>: </span>"</span>John Doe</span>"
</span> </span>"</span>card</span>"</span>: </span>"</span>1234..</span>"
</span>}
</span></code></pre>
(instead of send these over REST, we will first create a jwt token - that's for security)</p>
Now check status of transaction</p>
GET</span> /payment-intent/{intentID}
</span>{
</span>    </span>"</span>status</span>"</span>: </span>"</span>PENDING</span>"
</span>}
</span></code></pre>
System Design</h2>
Merchant makes a request for payment</h3>

Merchant -> API Gateway (load balancing, authentication, rate limiting, routes requests to the right microservice ...) -> paymentIntentService -> metadataDB (create intent record - id, merchand id, customer id, status, amount, product.)</li>
</li>
</ul>
Users pay for product</h3>

Collect payment details from customer.</li>
On the merchant website, the user enters their card details.</li>
The merchant, with intendId, initiates a transaction. Transaction service takes over - dealts with external banking system, while making entries in the DB.</li>
</ul>
Merchant and User status updates</h3>

Transaction service get call, with look up on the same db.</li>
</ul>
System Design Scaling</h2>
So focus areas: security, transaction volume, durability - no loss of records.</p>
Security</h3>

Is the person why they say they are (i.e merchant)</li>
Are we saving and keeping encrypted the customer details?</li>
</ol>
For (1) -> Generate API keys, associated with a merchant when they sign up. Each transaction should pass this API key in headers (like so: Authorization: Bearer {api_key}</code>). This will be the public key. They will be given a secret key that can use used to generate a unique nonce at the time of transaction, along with a signature and timestamp to validate.</p>
//</span> Example request with signature
</span>{
</span>  </span>"</span>method</span>"</span>: </span>"</span>POST</span>"</span>,
</span>  </span>"</span>path</span>"</span>: </span>"</span>/payment-intents/{paymentIntentId}/transactions</span>"</span>,
</span>  </span>"</span>body</span>"</span>: </span>{
</span>    // body here
</span>  },
</span>  </span>"</span>headers</span>"</span>: </span>{
</span>    </span>"</span>Authorization</span>"</span>: </span>"</span>Bearer pk_live_51NzQRtGswQnXYZ8o</span>"</span>, // API Key
</span>    </span>"</span>X-Request-Timestamp</span>"</span>: </span>"</span>2023-10-15T14:22:31Z</span>"</span>, // Timestamp
</span>    </span>"</span>X-Request-Nonce</span>"</span>: </span>"</span>a1b2c3d4-e5f6-7890-abcd-ef1234567890</span>"</span>, // Nonce
</span>    </span>"</span>X-Signature</span>"</span>: </span>"</span>sha256=7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069</span>"</span> // Hash of body
</span>  }
</span>}
</span>
</span></code></pre>
For (2) -> client info should never hit merchant server. Provide a javascript sdk that within an iframe, loads our server domain (stripe's, other providers) where they can enter card details. This is then sent directly to our server. We encrypt this on client machine of course, with a public key. On our servers, with associated private key, we decrypt. HTTPS provides the transport and security layer.</p>
Durability</h3>

We need to track not just what the current state is, but the entire sequence of events that led to that state. When a customer disputes a charge six months later, we must be able to prove exactly what happened: when the payment was initiated, what amount was authorized, when it was captured, and whether any refunds were processed. A single missing record could mean inability to defend against chargebacks, failed compliance audits, or worse—being unable to determine the true state of customer accounts.</p>
</blockquote>
So how?
Separate operation concerns from audit concerns.</p>
Operations -> MetadataDB.
Audit -> CDC on the metadatadb, which publishes data to a Kafka queue. Differnt subscribers will consume event stream and make entries in differnt audit tables.</p>

For durability, Kafka provides configurable replication (typically 3x) across multiple brokers and availability zones. Events are retained for a configurable period (often 7-30 days) on disk, with older events archived to object storage for permanent retention. All events are automatically flushed to S3 for long-term auditability, ensuring we can reconstruct any payment's complete history even years later for compliance audits or dispute resolution. This gives us both hot data for operational use and cold storage for compliance.</p>
</blockquote>
Transaction Volume</h3>
Event log, Kafka:</p>

We are expecting ~10k tps. While Kafka can support millions of messages per second at the cluster level, it's important to note that each individual partition typically handles around 5,000-10,000 messages per second under normal production conditions. To comfortably support our 10k TPS requirement, we'd need multiple partitions—likely 3-5—to ensure both throughput and fault tolerance.</p>
</blockquote>

With 10k TPS for our event stream, we can expect around 10k writes per second to our database as well. This is right on the edge of what a well-optimized PostgreSQL instance can handle this load, especially when combined with read replicas and proper indexing. To better handle the scale, we can shard our database by merchant_id.</p>
</blockquote>

With regards to data growth, we can estimate that each row is ~500 bytes. This means we are storing 10,000 _ 500 bytes = 5mb of data a second, 5mb _ 100,000 (rounded seconds in a day) = 500gb of data a day, and 500gb * 365 = ~180tb of data a year. This is significant storage growth that will require careful planning for data retention and archiving strategies.
For transactions older than a certain period (e.g., 3-6 months), we can move them to cold storage like Amazon S3 or Google Cloud Storage.</p>
</blockquote>

For read scaling, we can implement read replicas of our database. Most payment queries are read operations (checking payment status, generating reports), so having multiple read replicas will distribute this load.</p>
</blockquote>
Design Whatsapp</h1>
Requirements</h2>
Functional:</p>

User should be able to send/receive 1:1 message</li>
User should be able to create groups.</li>
User should be able to send media.</li>
Messages should be able to receive message if they are not online.</li>
</ol>
Non-functional:</p>

Less than 500ms latency wehn receiving a message.</li>
Scale - support a billion users</li>
Gurantee messages are delivered.</li>
</ol>
Out of line:</p>

Security</li>
Spam</li>
</ul>
Entities</h2>

User: Details about the number, name, last online status.</li>
Message: content of message, from and to whom/what</li>
Group: Participant list, admin</li>
Media: associated with a message</li>
</ul>
(hello interivew: users, chat (same as group) , messages, clients (multiple devices))</p>
API</h2>
Sockets: sendMessage</p>
High Level System Design</h2>

</li>
</ul>
Deep Dives</h2>

</li>
</ul>
Design a Local Delivery Service</h1>
Like Uber Eats. Or Rapido.</p>
Requirements</h2>
Function:</p>

Customers can search for items available for delivery within 1 hour</li>
Order by location</li>
Customers can order multiple items</li>
</ol>
Out of scope:</p>

Driver matching</li>
Payments</li>
Cancellations</li>
</ol>
Non-functional:</p>

Availablility: search should be fast</li>
Ordering: Two customers should not place requests on the same product id.</li>
Volume: 10m/day</li>
</ol>
Entities</h2>

Product.</li>
Distribution Center.</li>
Order.</li>
Inventory? where a product is located, physically.</li>
</ol>
APIs</h2>
The user facing apis are really: check availability, then order product.</p>

Check availability</li>
</ol>
GET</span> /availability?lat={}&</span>long</span>=</span>{}&keyword={}&page_size=10&page_num={} </span>-</span>>
</span>{
</span> </span>items: </span>{
</span>    name: NAME,
</span>    quanitity: QUANTITY
</span> }
</span>}
</span></code></pre>

Order</li>
</ol>
POST</span> /order ->
</span>{
</span>    </span>lat:</span> LAT, long: LONG, items: </span>[</span>item1, item2...</span>]
</span>}
</span></code></pre>
High Level System Design</h2>

Customers should be able to query availability of items: a db search. First search for locations in via postgis maybe, then given a list of dcs search for product in those dcs.</li>
Customers order item. Strong consistency here to ensure two people don't order the same item. "By putting both orders and inventory in the same database, we can take advantage of the ACID properties of our Postgres database. Using a singular transaction with isolation level SERIALIZABLE we can ensure that the entire transaction is atomic. This means that if two users try to order the same item at the same time, one of them will be rejected."</li>
</ol>
Deep Dives</h2>

How do we make the look ups fast? PostGIS index. Use time to to estimate traffic, or use an external service.</li>
And what else? Scaling reads: Use cache, use read replicas to distribute load. "Our orders need to be strongly consistent, so those transactions need to be sent to the Postgres leader, but our availability queries can go to our read replicas."</li>
</ol>
Design a Price Tracking Service</h1>
A price tracking service that monitors Amazon product prices over time and alerts users when prices drop below their specified thresholds.</p>
Requirements</h2>


Functional</p>

For an amazon product, users should be able to see it's price history.</li>
Users should be able to subscribe to notifications when prices drop.</li>
(below the line) Search for amazon products.</li>
(below the line) product reviews and ratings.</li>
</ul>
</li>

Non-functional</p>

Availability over consistency: users should always be able to see some price, even if it's not the latest price.</li>
Notification latency: < 1 hour once price drops to notifying the user about it.</li>
Price history query in < 500 ms.</li>
Search amazon catalog of over 500 million items.</li>
</ul>
</li>
</ul>
Entities</h2>

User: interested in products, credit card on file, who has to be notified.</li>
Product: with it's price history.</li>
Subscription: Link users and products.</li>
Price: price history associated with products.</li>
</ul>
APIs</h2>

Query for product price (search is below scope)</li>
</ol>
GET</span> /products/{productID}/price?period=10d&</span>granularity</span>=</span>hourly </span>-</span>> priceHistory</span>[]
</span></code></pre>

Subscribe for notification</li>
</ol>
POST</span> /subscription
</span>{
</span> </span>productID,
</span> </span>priceThreshold,
</span> </span>notificationType
</span>} -> </span>200
</span></code></pre>
High Level System Design</h2>
This is the data flow:</p>

we have a crawler that crawls amazon on a schedule and updates a db with prices.</li>
The user requests the latest price for a product, or it's history, and we show them this information.</li>
For a product, the user will subscribe to be notified on a price drop.</li>
For each product, we have a queue of subscribers with different price points, and we notify them whenever the cralwer brings in a new and price and it's below the desired threshold. We expire this once the price drops for a user.</li>
</ol>
Two tables with two different characteristics to organize the data:</p>

A time series db, append only, for price info.</li>
A db with product info and metadata.</li>
</ol>
When a user subscribes, an entry is made in the db recording the details of the product and the desired price threshold.</p>
The simplest system is a cron that that checks the db every hour and notifies all the users.</p>

  
  Price Tracker</figcaption>
  
</figure>
Deep Dive</h2>
The above the is the simplest system that does the work. There are issues, for example, each notification is preceded by a table scale, which is slow.</p>
1. How do we efficiently discover and track 500 million amazon products?</h3>
Amazon has rate limits - let's say 1 crawl/sec per IP. So crawling all the products is out of question. Solutions are:</p>



Prioritize a few products based on use (most things follow pareto distribution.)</li>
</ol>

Maintain different queues - high, medium, and low, based on product priority. Assign a priority score based on user interest, and factors such as products changing price rapidly.</li>
Primary disadvantage is new product discovery - we can't here before it gets popular.</li>
</ul>
</li>


Make use of existing distribution channel for product discover - the chrome plug in with 1 million users. When they browse product catalogs, track them and these are the prices we track.</li>
</ol>
</li>
</ul>
2. How do we efficiently notify users of price changes?</h3>
Our NFR is letting users know within an hour for price changes. Table scans will not cut it. Some ways to do this more efficiently are:</p>

Instead of tracking changes every few hours via cron job, we can create a notification delivery event as soon as a price changes. Example, on the price db, set up a change data capture service, so it automatically publishes events, which are sent to kafka. "Our notification service subscribes to these events and immediately queries the subscriptions table to find affected users."</li>
Or the crawler writes to the kafka queue immediately, and the notification service proces the product id or drops it.</li>
</ol>
Design a Bookseller Platform</h1>
A service where users can submit book purchase requests with details such as customer information, book info, maximum price willing to pay, and payment information. The service should check with registered book sellers to find books priced at or below the maximum price. It should place a buy request with the seller offering the minimum price within the customer's limit. The design should include APIs for book seller registration and adapt to various seller APIs, supporting up to 10,000 queries per second.</p>
Requirements</h2>
Functional</h3>

Customers submit book purchase requests.</li>
Book sellers can register with service.</li>
Service makes a purchase order satisfying customer's request with seller.</li>
Service supports seller's adaptable APIs.</li>
</ul>
Out of scope:</p>

Invoice, order management.</li>
Delivery.</li>
Reccomendations.</li>
</ol>
Non-functional</h3>

10k qps for customer service request.</li>
availability over consistency - make a purchase reqeust over the best purchase request imediately.</li>
System should respond to request in < 2 seconds.</li>
Security: transaction for payments.</li>
</ol>
Entities/Data Model</h2>

Customer</li>
Seller</li>
Purchase Request</li>
Order</li>
Quotes: different sellers</li>
</ol>
APIs</h2>

make book purchase request</li>
</ol>
POST</span> /v1/purchase -> requestId
</span>{
</span>    </span>customer_id: </span>"</span>xxx</span>"
</span>    </span>"</span>details</span>"</span>: </span>{
</span>        </span>"</span>isbn</span>"</span>: </span>"</span>...</span>"</span>,
</span>        </span>"</span>title</span>"</span>: </span>"</span>...</span>"
</span>    },
</span>    </span>max_price:</span> 20.00
</span>    </span>currency:</span> USD
</span>}
</span></code></pre>

Register book seller</li>
</ol>
POST</span> /v1/register -> </span>200
</span>{
</span>  </span>"</span>seller_id</span>"</span>: </span>"</span>xxx</span>"</span>,
</span>  </span>"</span>api_config</span>"</span>: </span>{
</span>    </span>"</span>base_url</span>"</span>: </span>"</span>...</span>"</span>,
</span>    </span>"</span>auth_type</span>"</span>: </span>"</span>api_key</span>"</span>,
</span>    </span>"</span>endpoints</span>"</span>: {
</span>        </span>"</span>search</span>"</span>: </span>"</span>/search</span>"</span>,
</span>        </span>"</span>purchase</span>"</span>: </span>"</span>/purchase</span>"</span>,
</span>    }
</span>  }
</span>}
</span></code></pre>

Check status?</li>
</ol>
GET</span> /v1/purchase/{requestId}/status
</span></code></pre>
High Level Design</h2>

Customer submits a purchase request.</li>
Our platform queries all the sellers for that books.</li>
Orders the sellers by prices and picks the first one lesser than or at the customer's desired price.</li>
Make a purchase request via the seller's API</li>
system processes payment and confirms order.</li>
Communicates purchase order/request.</li>
</ol>
There are two different API layers:</p>

Customer facing</li>
Seller facing</li>
</ol>
Simplest dataflow is as follows</h3>
When customer makes a request: User -> API Gateway -> purchaseRequestService -> make record in DB -> search seller APIs and aggregate them -> pick the item that matches -> make another db record -> transaction processing.</p>
Seller registers: Seller -> API Gateway -> registerService -> make record in db
The seller is given a key that is used when communicating.</p>
Deep Dives</h2>
1. How to handle scale 10k qps?</h3>
Use Dynamo DB. There's not complex relationships, all look ups based on key, which can be request id, or seller id, or order id. single millisecond response time. There are also no strong consistency requirements - so Postgres is an overkill.</p>
The partition key for dynamo db will be request key, sort key can be customer id</p>
To handle scale, even larger ones, for things like bursty traffic, use SQS queue for seller (and customer) calls. You want to make seller API requests in parallel - hence the queue with a separate worker for each API (or use async within the main service).</p>
Further, we can also cache seller api configuration, seller api responses.</p>
2. How to handle security?</h3>

Customer info never reaches sellers.</li>
Sellers, when they register, create RSA key pair. We have the public key. They have the private key.</li>
When we make an order on behalf of customer, we hold the money. We transfer the money to the seller.</li>
We create a payment token to the seller, encrypted with the public key. The seller uses their private key to decrypt and approve the transaction, confirm the purchase has been made.</li>
</ul>
3. How to ensure 2 second response time with multiple sellers?</h3>

Use async and gather results, with aggresive timeouts. Use connection pooling.</li>
Keep seller priority internally based on historic response times.</li>
</ul>
4. How to handle different seller APIs?</h3>

We want to offer a configuration where sellers can adapt their APIs.</li>
Maintain adapter versions for different sellers, graceful fallback between different versions.</li>
</ul>
Design a Hierarchical Distributed File System Design</h1>
Design a hierarchical distributed file system where customers can store an arbitrary number of files and directories, nested recursively. Files should be immutable once created, while directories must be mutable to allow creating and deleting files and subdirectories. Implement APIs for file retrieval, storage, directory creation, listing, and deletion. The system should support hundreds of petabytes of storage for thousands of Databricks customers, operating within a single region and data center. Existing distributed storage systems cannot be reused, but single machine filesystems, databases, and KV stores can be utilized.</p>
Requirements</h2>
Functional</h3>

Core

Store arbitrary number of files and directories with recursive nesting</li>
Files are immutable once created (write-once, read-many)</li>
Directories are mutable (create/delete files and subdirectories)</li>
Support file retrieval, storage, directory creation, listing, and deletion</li>
Hierarchical namespace with path-based access (e.g., /customer1/project/data/file.txt)</li>
</ul>
</li>
Out of scope

file versioning, history</li>
file compression</li>
</ul>
</li>
</ul>
Non-functional</h3>

Scale to multiple petabytes of storage.</li>
Many hundreds of customers.</li>
High availability and fault tolerance.</li>
Low latency for metadata operations (< 100 ms).</li>
High throughput for data operations ( GS/s per customer).</li>
</ul>
Data Model + API</h2>

Customer</li>
Files</li>
File Metadata</li>
Dir</li>
DataNode</li>
</ul>
File Ops, Dir Ops, List Ops</p>
# Initiate chunked upload
</span>POST</span> /api/v1/customers/{customer_id}/files/upload
</span>{
</span>  </span>"</span>path</span>"</span>: </span>"</span>/project/data/large_file.txt</span>"</span>,
</span>  </span>"</span>size</span>"</span>:</span> 1073741824,
</span>  </span>"</span>chunk_size</span>"</span>:</span> 67108864
</span>}
</span>Response: </span>{
</span>  </span>"</span>upload_id</span>"</span>: </span>"</span>upload_123</span>"</span>,
</span>  </span>"</span>chunk_urls</span>"</span>: </span>[</span>"</span>https://api/upload/chunk/1</span>"</span>, </span>"</span>https://api/upload/chunk/2</span>"</span>, ...</span>]</span>,
</span>  </span>"</span>total_chunks</span>"</span>: 16
</span>}
</span>
</span># Upload individual chunks
</span>PUT</span> /api/v1/upload/chunk/{chunk_id}
</span>Content-Type:</span> application/octet-stream
</span>[raw</span> binary data - 64MB chunk]
</span>
</span># Complete upload
</span>POST</span> /api/v1/customers/{customer_id}/files/upload/complete
</span>{
</span>  </span>"</span>upload_id</span>"</span>: </span>"</span>upload_123</span>"
</span>}
</span>
</span># Read file
</span>GET</span> /api/v1/customers/{customer_id}/files?path=/project/data/large_file.txt
</span>Response: </span>{
</span>  </span>"</span>file_id</span>"</span>: </span>"</span>file_456</span>"</span>,
</span>  </span>"</span>size</span>"</span>: 1073741824,
</span>  </span>"</span>block_locations</span>"</span>: [
</span>    {</span>"</span>block_id</span>"</span>: </span>"</span>block_001</span>"</span>, </span>"</span>nodes</span>"</span>: </span>[</span>"</span>node1</span>"</span>, </span>"</span>node3</span>"</span>, </span>"</span>node7</span>"</span>]</span>},
</span>    {</span>"</span>block_id</span>"</span>: </span>"</span>block_002</span>"</span>, </span>"</span>nodes</span>"</span>: </span>[</span>"</span>node2</span>"</span>, </span>"</span>node4</span>"</span>, </span>"</span>node8</span>"</span>]</span>}
</span>  ]
</span>}
</span>
</span># Delete file
</span>DELETE</span> /api/v1/customers/{customer_id}/files?path=/project/data/large_file.txt
</span></code></pre>
High Level System Design</h2>
File Upload Flow:</h3>

Client requests file upload via REST API to API Gateway</li>
API Gateway returns upload metadata (file_id, chunk_size, upload URLs)</li>
Client uploads file as chunked HTTP streams (64MB chunks) to API Gateway</li>
API Gateway streams chunks directly to Data Nodes via internal gRPC</li>
Data Nodes write chunks directly to HDD as temporary files (/storage/blocks/block_001.tmp)</li>
On successful write, Data Nodes atomically rename to final location (/storage/blocks/block_001)</li>
Metadata Service creates inode mapping file to block locations</li>
Directory Service updates parent directory to include new file entry</li>
</ul>
File Read Flow:</h3>

Client requests file via REST API path lookup</li>
API Gateway queries Directory Service for path → inode mapping</li>
Directory Service returns inode_id for requested file</li>
API Gateway queries Metadata Service for inode → block locations</li>
API Gateway returns block locations to client for parallel HTTP reads</li>
Client reads blocks directly from Data Nodes via HTTP streaming</li>
Client reassembles file from downloaded blocks</li>
</ul>
Directory Operations:</h3>

Client requests directory operation (create/list/delete)</li>
API Gateway routes to Directory Service based on path hash</li>
Directory Service manages namespace operations with distributed locking</li>
Updates replicated across directory metadata nodes using Raft consensus</li>
Results returned to client with directory metadata</li>
</ul>
So:</p>
User -> API Gateway -> UploadService -> Metadata DB</p>
Design FB Post Search</h1>
Suitable for any search feature/product.</p>
Requirements</h2>
Functional</h3>

Users should be able to create posts.</li>
Users should be able to search for posts by content.</li>
Users should be able to order search results by recency, like count etc.</li>
</ol>
Out of scope:</p>

fuzzy matching of terms for search.</li>
search images and media.</li>
</ol>
Non-functional</h3>

Search results < 500 ms.</li>
New posts should be searchable in < 1 minute.</li>
Support high volume of requests.</li>
System should be highly available.</li>
</ol>
1B users, on avg, 1 post/day = 10k posts/second.
Likes = 100k likes/second.</p>
Searches = 10k searches/second.
Volume of posts created (and hence searchable): 1B posts/day * 365 days/year * 10 years = 3.6T posts
Size: 3.6T * 1kb = 3.6 Pb.</p>
Entities/Data Model</h2>

User</li>
Post</li>
Like</li>
</ol>
API</h2>

Search: GET /v1/search?query=QUERY&sort_order=likes</code></li>
Post: POST /post</code></li>
Like: POST /post/{id}/like</code></li>
</ol>
High Level Design</h2>
User -> Post Service, Like Service -> DB
User -> Search Service -> DB</p>
For search, we will need to set up an inverted index. Scaling challenges around which words to index, (ignore common words like conjunctions? etc).</p>
User -> Search Service -> Db (sort results)
Request time sorting is super slow. The number of results could be large - in 10s of millions. So a solution is using multiple indices. One tracks sort by time, another likes. "When a new post is created, we'll add it to both indexes for every keyword it contains. When a like event happens, we'll update the score in our sorted set for the likes index."</p>
Deep Dive</h2>
Focus on satisfying scaling and handling failure modes.</p>


Handle a large volume of requests:
Have a distributed cache. With an eviction policy. TTL of < 1 min. Use CDN for another solution: "Most CDNs operate like a big set of geographically-spread HTTP caches. You configure an "origin" or target for the cache and configure DNS to route through the CDN. If the cache has the item the user is looking for, it can return it faster than almost any alternative option: most CDNs have locations very close to most users. If it doesn't hit the cache, the CDN will proxy the request back to your servers to handle."</p>
</li>

Handle complicated search "Cat in the hat seeing a bat" versus single word:
Index bigrams and shingles. So index "Cat in", "the hat", "hat seeing". This will increase the size of the database. A possible remediation is to only index bigrams that are likely to be searched.</p>
</li>

Address large volume of writes:
Post creation: Fan out creation requests using a queue, partition load. We can also buffer requests so that we can handle bursts of post creation. Finally, we can scale out our index by sharding the indexes by keyword.
Like record: batch likes, so instead of writing 500 times for 500 likes, write once. The information in our db is likely stale though.</p>
</li>
</ol>
Design Google Docs</h1>
Requirements</h2>
Functional</h3>

Users create new docs</li>
Multiple users can edit the same doc.</li>
Users can see the other person's changes in real time.</li>
</ol>
Out of scope:</p>

markdown docs or fancy text formatting.</li>
perimissions, only viewing</li>
doc history, versioning</li>
</ul>
Non-functional</h3>

100 simulatenous users on one doc.</li>
Documents should be eventually consistent - at some point in time, everyone should see the same doc.</li>
Low latency edits. Let's say < 100ms for an edit to appear.</li>
Docs should be durable even over server restarts.</li>
</ol>
Entites + API</h2>

User</li>
Doc</li>
Edit</li>
Cursor: position of the cursor on the doc.</li>
</ol>
API</h3>
We are going to be using websockets because there are going to lots of bidirectional communication between the docs and the users edits the docs.</p>

Create Doc. This is REST API: `POST /v1/doc -> docId</li>
Edit Doc:</li>
</ol>
   WS /v1/doc/{docId}
</span>   SEND {
</span>    type: "insert"
</span>    ...
</span>   }
</span></code></pre>
   WS /v1/doc/{docId}
</span>   SEND {
</span>    type: "delete"
</span>    ...
</span>   }
</span></code></pre>

Get update:</li>
</ol>
  WS /v1/doc/{docId}
</span>    RECV {
</span>      type: "update"
</span>      ...
</span>    }
</span></code></pre>
High Level Design</h2>

Create new docs: Client connects to a server via API Gateway, an entry is made in a db creating a doc.</li>
Multiple users editing concurrently: A central server transforms the edits so that ordering is preserved. We only send edits instead of the whole document to save space and help latency. The edits are writtent to a new db that supports high volume of writes: cassandra.</li>
When a user connects to an existing doc: load the doc, apply the latest Operational Transforms.</li>
</ol>
Deep Dives</h2>
Imagine two users, Alice and Bob, are editing the same sentence: "Hello!". Alice adds ", world" at the end, and at the exact same time, Bob deletes the "!". What specific data does Alice's client send to the server? What about Bob's? And how does your server merge these two conflicting operations to ensure both clients end up with the same, correct document state?</h1>

For saving the doc, we can't save the entire doc each time a user makes a change. That takes up too much bandwidth and will be too slow.</li>
So when a user makes an edits, it's worthwhile to save only the edits.</li>
So we need to have a data model of the how edits should be saved and transmitted.</li>
When multiple edits are happening to the document at the same time, the server is responsible for reconciling the edits and transmitting it to the users.</li>
For Google Docs, we are going to use Operational Transforms for manage the edits.</li>
The data we will store when alice edits is: INSERT(6, ", world"). For bob: DELETE(6).</li>
The server, when it has to route two such edits, on the server side, it updates the parameters of the operation so they are normalkized to the changes that happen before it. In this case, after insert, the delete operation becomes: DELETE(12).</li>
So when an user makes an edit, we first apply the local edit to his application so it's immediate, then on the server side, apply the chain transformation of edits to all users.</li>
On the local side, when a message is recevied, we apply OT locally.</li>
</ol>
The server is now a critical component; it's the single source of truth that establishes the final order of operations. What happens if that single server fails? How do you ensure durability for the edits being processed, and how does the system recover to a consistent state once a new server takes over?</h1>
Yeah, the server is doing a hell of a lot of work here, and in OT, it's a critical component. We need to have multiple servers, each handling a subset of users. It's best if we ensure all the users who are editing a doc are connected to the same server. The server instance also has to be large enough to handle the volume of messages coming. We can assume each sever has maybe 100k connections, and for 10 million concurrent users, we will need 10*1e6/100000 =100 servers.</p>
When a single server fails, we need to manage:</p>

the connections that server was dealing with</li>
The edits flowing through that server - ensure they are not lost, even if connection was immediately terminated for all users.</li>
</ol>
The way we can deal with (1) is by using a coordination service like zookeeper to manage the server instances - scale it up or down, though with socket servers, this might be complicated, and we could do with just N sockets and then bring a new one up manually. The zookeeper services records all the server ids and the sockets assigned to each server id. Each user is given an id and mapped to a socket server via an hash. Once a server fails, we immediately notify to users that server connection has been lost - this can be done at the application layer to each user.</p>
For the changes that were in teh server but not applied yet to all users, these have to be backed up, and backed up often. The way we are going to manage this is each edit will be saved, along with the document version id and time stamp. We can save this in a dynamodb.</p>
When users come back and are connected to a new server, we compare the reconciled document id in the possibly blob store, Then apply the edits since the last change and send it to the user.</p>
So a reconciled, materialized file is saved once in a while, let's say a blob store, and the link to this file is present in our db.</p>
So our db tracks:</p>

The edits from all users,</li>
Document id and the document version id</li>
The materialized file in the blob store.</li>
</ol>
The materialized file can be created once in a while, for example when new users connect.</p>
Design an Online Auction Platform Like eBay</h1>
Resources</h1>

System Design Handbook</a></li>
Hello Interview</a></li>
</ul>


CV
2025-08-11T00:00:00+00:00

Abhishek Sundararajan

Books 2026

January</h1>

The Switch, Elmore Leonard</h2> Not too nasty, which was unexpected. Only the deserving die in this one. Altogether, a serviceable Leonard.</p>

52 Pickup, Elmore Leonard</h2> Nasty little crime thriller. SA makes me uncomfortable and the hints of it in this one was more than enough to make me want to never re-visit this again. But it's Elmore Leonard. Of course I devoured it as soon as I read the first line.</p>

Mahaviri, Nityananda Misra</h2>

Some software, 2025

Tale as old as time

Helix aide-mémoire

The Delhi Sultanate

System Design 2

Requirements</h2>

System Interface</h2>

High-Level Design</h2>

Deep Dives</h2>

ChatGPT with Inference Batching</h1>

Understanding the Problem</h2> What are we building?</strong> A ChatGPT-like system that efficiently utilizes a pool of GPU servers for text inference. Each GPU server has a fixed latency for processing batches of 1-100 strings, making request aggregation the key optimization strategy.</p>

The Set Up</h2>

High-Level Design</h2>

Potential Deep Dives</h2>

High-Level Design</h2>

1) Users should be able to send prompts and receive streaming responses</h3> The core challenge is maintaining persistent WebSocket connections while orchestrating calls to external LLM providers and streaming responses back to clients.</p>

Potential Deep Dives</h2>

1) How do we handle WebSocket connection reliability and session recovery?</h3> WebSocket connections are inherently fragile - networks drop, browsers close, mobile apps background. Our system needs robust connection management and seamless recovery.</p>

Challenges</h6> Heartbeating adds bandwidth overhead and complexity. Mobile clients may not handle backgrounding gracefully, leading to unnecessary reconnections.</p>

Challenges</h6> Redis becomes a single point of failure. Message replay can be expensive for long-lived sessions with extensive history.</p>

Hybrid</h2> Tree approach for first few nodes, then P2P file transfer for stragglers or failures.</p>

System Design 1

Design Stripe</h1>

Requirements</h2> Functional Requirements:</p>

System Design</h2>

System Design Scaling</h2> So focus areas: security, transaction volume, durability - no loss of records.</p>

Design Whatsapp</h1>

API</h2> Sockets: sendMessage</p>

Design a Local Delivery Service</h1> Like Uber Eats. Or Rapido.</p>

Design a Price Tracking Service</h1> A price tracking service that monitors Amazon product prices over time and alerts users when prices drop below their specified thresholds.</p>

Deep Dive</h2> The above the is the simplest system that does the work. There are issues, for example, each notification is preceded by a table scale, which is slow.</p>

Requirements</h2>

Deep Dives</h2>

Requirements</h2>

High Level System Design</h2>

Design FB Post Search</h1> Suitable for any search feature/product.</p>

The Switch, Elmore Leonard</h2>
Not too nasty, which was unexpected. Only the deserving die in this one. Altogether, a serviceable Leonard.</p>

52 Pickup, Elmore Leonard</h2>
Nasty little crime thriller. SA makes me uncomfortable and the hints of it in this one was more than enough to make me want to never re-visit this again. But it's Elmore Leonard. Of course I devoured it as soon as I read the first line.</p>

Understanding the Problem</h2>
What are we building?</strong> A ChatGPT-like system that efficiently utilizes a pool of GPU servers for text inference. Each GPU server has a fixed latency for processing batches of 1-100 strings, making request aggregation the key optimization strategy.</p>

1) Users should be able to send prompts and receive streaming responses</h3>
The core challenge is maintaining persistent WebSocket connections while orchestrating calls to external LLM providers and streaming responses back to clients.</p>

1) How do we handle WebSocket connection reliability and session recovery?</h3>
WebSocket connections are inherently fragile - networks drop, browsers close, mobile apps background. Our system needs robust connection management and seamless recovery.</p>

Challenges</h6>
Heartbeating adds bandwidth overhead and complexity. Mobile clients may not handle backgrounding gracefully, leading to unnecessary reconnections.</p>

Challenges</h6>
Redis becomes a single point of failure. Message replay can be expensive for long-lived sessions with extensive history.</p>

Hybrid</h2>
Tree approach for first few nodes, then P2P file transfer for stragglers or failures.</p>

System Design Scaling</h2>
So focus areas: security, transaction volume, durability - no loss of records.</p>

API</h2>
Sockets: sendMessage</p>

Design a Local Delivery Service</h1>
Like Uber Eats. Or Rapido.</p>

Design a Price Tracking Service</h1>
A price tracking service that monitors Amazon product prices over time and alerts users when prices drop below their specified thresholds.</p>

Deep Dive</h2>
The above the is the simplest system that does the work. There are issues, for example, each notification is preceded by a table scale, which is slow.</p>

Design FB Post Search</h1>
Suitable for any search feature/product.</p>