Local LLMs on potato computers feat. the llm Python CLI and sllm.nvim

2025-12-21T00:00:00+00:00

Foreword</h1>
The year is 2022. We are living in a world where someone, not a machine, probably wrote that article or tutorial you’re reading. Sure, the Dead Internet</a> theory is already gaining traction, but mostly, content online is created by people.
Transformers changed everything. In a 2017 research paper named “Attention Is All You Need”</a>, currently cited 207 000 times 🤯, eight scientists working for Google introduced this revolutionary new technique, and late-2022, ChatGPT (GPT for Generative, Pre-trained and Transformer) was launched. Today, NVIDIA is the most valued company on Earth, firms like OpenAI and Anthropic boast enormous valuations and make even bigger promises, and AI is being forced down our throats at every occasion.
I do not like generative AI the way it is used and pushed by our tech overlords, but I do think the underlying science, machine learning, can be very useful and not wasteful of resources if done right. Machine learning has been a thing for decades at this point: LLMs (for Large Language Models, i.e. AI chatbots) and the current transformers-induced AI boom is just one very specific way it’s used.
Like many, I’ve never used ChatGPT, Claude, Copilot, any of them. The argument “you should or you’ll get left behind” is utter nonsense: don’t take it from me, take it from people that actively use these tools everyday</a>.
The only AI that is of remote interest to me is AI I can run locally, so I’ve been keeping an eye on places like the LocalLLaMA</a> subreddit and trying out small and local LLMs. These are what we call open weight models (not to be confused with truly open source models like Olmo</a>). I want these models running on my laptop, not on a server, even if I control the server: this way, their fingerprint is kept very small (running them is no different than playing a video game).
In this article, I’ll show you how to setup the `llm</code></a> Python CLI, what you can do with it, and how to use it with Neovim using the` `sllm.nvim</code></a> plugin. I’ll share how I’ve been using these LLMs, and while it didn’t change my life, it can be pretty useful, and probably will be more and more useful if small LLMs continue improving the way they have.`
`If you must use AI, I recommend you use it in a similar way. Keep control over your data, do not participate in the AI madness by paying hefty subscriptions or buying/renting monster GPUs, just use small, local models. If your computer is better than mine, you can use heavier models that will yield better results, and the inverse is true: we’ll talk a bit about how to select the model that’s right for you.`
`This article is split into two parts. The one you’re reading, and a second, non-technical one about the AI bubble, the environmental costs, and the financial consequences of these tools, that you can read here</a>.`
`Let’s dive right in!`

The resources at my disposal</h1> My laptop config</h2> Meet my laptop, the ASUS PC Portable Gamer FX552VE-DM380T, bought in February 2018: I wouldn’t call it old, I’d say it is middle-aged. I am confident I can squeeze a few more years out of it, and fully intend to do so. I use Linux, so Microsoft cannot force me to throw it out</a> when Windows 10 support ends in October 2026. I’ve doubled the amount of RAM a few years ago (from 6 GB to 12), and the graphics card it has is an NVIDIA 1050Ti with only 2 GB of VRAM. That last figure is important. A defining characteristic of a graphics card is how much VRAM or video RAM</a> it has. This is very fast memory, way faster than your “normal” RAM, directly integrated inside the GPU. If your model fits inside that amount of memory, it will run pretty fast (of course, the raw computing power of your GPU is also important). And since your GPU is doing the heavy lifting, your main system resources are not too bothered by the additional workload: you have your RAM and CPU power at your disposal for web browsing, text editing etc. The LLMs I can run</h2> When browsing LLMs in places like the Ollama model list</a> or HuggingFace</a>, their name include the number of parameters they have e.g. 7B</code> stands for 7 billion and 0.6B</code> for 600 million parameters. These parameters are nothing more than weights in matrices, coefficients if you will. When an LLM runs, these parameters are loaded in memory, you feed the LLM an input (a prompt, a picture, a video feed…), some math happens i.e. multiplying the parameters and the input together and through layers, and you get an output, what the LLM produces. This is how all neural networks work, not just LLMs. If you want to learn more about the math behind transformers specifically, I recommend you watch this video</a> (actually, the whole series is great!). I said earlier that my graphics card only has 2 GB of VRAM. This means it can load 2 gigabytes of data, one byte being 8 bits. “Raw” LLMs store their parameters as 16-bit (or even 32-bit) values, but we can reduce this to a lot less with a technique called quantization and without losing too much precision. In a nutshell, for most use cases, Q4_K_M</code> quantization is what we want to go for, the default quantization applied for Ollama models. This reduces the parameters to 4-bit values, but LLMs also need a substantial amount of overhead or cache, to hold calculation steps while they take place. In my case, my graphics card can fully load around 2B</code> models quantized with Q4_K_M</code>. That may seem like very little. If you’ve been following the subject you probably think a 7B</code> model is already a small LLM: some models are 675B</code> parameters. But 2B</code> models and even lower can already do a lot, e.g. Microsoft released this 0.5B</code> model</a> which can generate very convincing speech from text with very low latency. If a model doesn’t fully fit in my graphics card’s VRAM, I can still try to run it by mixing RAM and VRAM. This comes at a heavy performance cost, but is still useful if you want bigger (i.e. more precise) models and don’t mind waiting for the output. Ollama</h2> I use Ollama</a> to download and run LLMs, you could also use llama.cpp</a>. Both are straight-forward to install, but if you run Linux like me, it used to be pretty hard to get your (NVIDIA specifically) graphics card configured and working. Thankfully, now that NVIDIA has a vested interest in making their graphics cards work on Linux, their drivers and CUDA have gotten a lot better. Running ollama list</code> shows me the models I’ve downloaded: NAME ID SIZE ministral-3:3b xxxxxxxxxxxx 3.0 GB yi-coder:1.5b xxxxxxxxxxxx 866 MB qwen3:4b xxxxxxxxxxxx 2.5 GB gemma3:4b-it-q4_K_M xxxxxxxxxxxx 3.3 GB hf.co/lmstudio-community/Olmo-3-7B-Instruct-GGUF:Q4_K_M xxxxxxxxxxxx 4.5 GB hf.co/hugging-quants/Llama-3.2-1B-Instruct-Q8_0-GGUF:Q8_0 xxxxxxxxxxxx 1.3 GB qwen3:1.7b xxxxxxxxxxxx 1.4 GB gemma3:1b xxxxxxxxxxxx 815 MB [...] </code></pre> </div> Some of these are heavier than others, but I can’t go much higher than a 4B</code> model without my computer no longer keeping up. Use LLMs the Unix way with the llm</code> plugin</h1> You can try out your LLMs with Ollama by doing ollama run <model-name></code>, you can also chat with your LLMs via a web interface like Open WebUI</a>. However, my favorite way of using them is with the llm</code></a> Python CLI. If you have uv</code> installed, you can run: uv tool install llm llm install llm-ollama </code></pre> You can then list the models llm</code> can use: llm models list llm models default qwen3:1.7b # set a default model </code></pre> [...] Ollama: qwen3:1.7b Ollama: gemma3:1b Default: qwen3:1.7b </code></pre> </div> The very nice thing about this CLI is that it behaves like any shell command. You can pipe context to it, and it will be inputted to an LLM. You’ll get the response back, that you can then write to a file or pipe to another command. For example, you can do things like: tail -n 100 online/gameplay.rs \ | llm prompt -s \ "Explain the handle_current_game_state Rust function" >> project_notes.md </code></pre> This will pass the last 100 lines of the online/gameplay.rs</code> file, where a function handle_current_game_state</code> is defined, then ask (via the system prompt with -s</code>) the model to explain what it does. The LLM will then append the response to a file. In this example, the project_nodes.md</code> file contains (snippets): ### **Key Functionality Overview** #### **2. Handling a Previous Game State** - If a game state is provided (`game_state.is_some()`), it: - Checks if the game has already ended (e.g., a winner has been declared). - [...] ### **Flow of the Game** 1. **Player Input**: - The player inputs a move (e.g., "e2e4"). - The system checks if the move is valid. - If valid, it sends the move to the game engine (e.g., Lichess) using `api.board_make_move`. - [...] ### **Asynchronous Nature** - The function uses `await` to handle asynchronous operations (e.g., `tx.send()`). - It waits for the `tx` channel to respond (e.g., for the opponent's move) before prompting the player again. ### **Summary** The `handle_current_game_state` function is responsible for: - Managing the flow of game states and moves. - Prompting the user for input when no previous move is available. [...] </code></pre> </div> …which is pretty accurate (not a 100%, but not far) and has been generated pretty fast on my laptop (42 seconds for 64 lines of explanations). Again, this is with a very small local model, only 1.7B</code> parameters. Check out the project</a> this is from, and help me out if it interests you 😉. </blockquote> The world is your oyster</h2> With llm</code>, you can create your own scripts for your use-cases. This is a simple yet very powerful feature. For example, let’s say I use commitlint</a> to keep a clean Git history. Here’s a script that generates a commit message from whatever I git add</code>ed, and asks the LLM to correct its input if it fails the commitlint</code> checks: #!/bin/bash # zoug.fr # temporary files for our commands' output DIFF_TEMP_FILE=$(mktemp) COMMIT_MSG_TEMP_FILE=$(mktemp) COMMIT_ERRORS_TEMP_FILE=$(mktemp) # the prompts I use # you probably should try other ones CREATE_COMMIT_PROMPT=""" You're a very experienced developer, you write short commits. Write a commit message using this template: TYPE: DESCRIPTION DETAILS Replace TYPE by 'feat', 'fix' or 'docs' Replace DESCRIPTION by a short sentence describing the changes Replace DETAILS by a longer description of the changes """ CORRECT_COMMIT_PROMPT=""" Correct the input to fix the errors. Do not try to change the commitlint configuration. Only output the corrected input. """ # how many times should we ask the LLM to correct its own message NUMBER_OF_RETRIES=2 git diff --cached >$DIFF_TEMP_FILE llm -s "${CREATE_COMMIT_PROMPT}" <${DIFF_TEMP_FILE} >${COMMIT_MSG_TEMP_FILE} # if the LLM doesn't pass the commitlint check if ! commitlint <${COMMIT_MSG_TEMP_FILE} >${COMMIT_ERRORS_TEMP_FILE}; then while [[ ${NUMBER_OF_RETRIES} -gt 0 ]]; do # commitlint outputs the inputted commit message and all errors/warnings # feed that to the LLM and ask it to correct the commit llm -s "${CORRECT_COMMIT_PROMPT}" <${COMMIT_ERRORS_TEMP_FILE} >${COMMIT_MSG_TEMP_FILE} # insert a new line after 72 characters. Small LLMs tend to struggle enforcing this rule sed -ie "s/.\{72\}/&\n/g" ${COMMIT_MSG_TEMP_FILE} commitlint <${COMMIT_MSG_TEMP_FILE} >${COMMIT_ERRORS_TEMP_FILE} [[ $? -ne 0 ]] && ((NUMBER_OF_RETRIES--)) || break done fi # open the commit message in your configured git text editor # since LLMs are not perfect, edits are often necessary git commit -F "${COMMIT_MSG_TEMP_FILE}" -e # cleanup rm $DIFF_TEMP_FILE $COMMIT_MSG_TEMP_FILE $COMMIT_ERRORS_TEMP_FILE </code></pre> With some bash</code> and “prompt engineering” (which is, really, trying random prompts until one sticks), you now have your own git commit writer. To have this, you didn’t have to send your data to anyone. Using a 1.7B</code> model on my laptop, it’s a little slow, it’s sometimes not accurate, but it works and is actually pretty useful if you follow good Git etiquette, i.e. small changes. Who needs ChatGPT? Two very cool features of llm</code> I didn’t touch on are tools and plugins. I suggest you head over to the llm</code> documentation</a> to learn more. Tools allow you to give an LLM the possibility to call any custom Python function. Plugins are a more complete way of adding functionality to the llm</code> CLI (including custom tools). There’s quite a few</a> plugins you can try, especially if you’re feeling adventurous</a>. It’s also possible to write your own</a>. </blockquote> Bring it to my editor</h1> I use Neovim as my IDE. There’s a bunch of Neovim plugins you can use for AI integration now (last time I checked, probably like 2 years ago, the landscape was very different). Of course, most of them focus on support for ChatGPT, Claude and other online services, but some like CodeCompanion</a> also support Ollama. I’ve tried some of them (only if they support Ollama), and they felt a little “too much”: I’m a simple man with a simple GPU and simple needs. What I want is a way to ask LLMs questions, get their answers, share some context from the files I’m editing, and that’s pretty much it: I don’t want the LLM to replace stuff in my files, add changes to my current git</code> commit or anything invasive like that. I’ve chosen the llm</code> plugin as my AI CLI, and there’s a Neovim plugin for people like me: sllm.nvim</a>. This plugin does a great job in staying out of the way, and uses llm</code> under the hood. With it installed, I continue using LSP, non-AI auto-complete etc., a standard Neovim config</a>, but now also have the option to ask an LLM something if I choose to do so. With LLMs directly integrated in my editor, I now ask them instead of using my web browser when I have basic questions. If I don’t remember the exact syntax to import an environment variable with Python, I can easily ask for a snippet: Same for getting the last inserted row’s ID in an sqlite3</code> database: This has been great to replace some cumbersome Stack Overflow copy and paste, and again, small models like yi-coder:1.5b</code> handle this perfectly fine and very fast. What I also found myself using a lot is the ability to select text and pass it to the LLM as context for a prompt. Let’s say I stumbled across a bit of Ansible (a server configuration tool), and want an explanation of what it does. I can select the piece of code of interest, and input a prompt: sllm.nvim</code> sends everything to the llm</code> CLI, and you get the response: This is useful in many scenarios, for example to write unit tests, to implement a new function by passing context from another one, etc. sllm.nvim</code> has other ways of building a comprehensive context depending on your needs, for example by including the whole current file, the LSP diagnostics, the contents of a given URL, etc. I won’t go into more details here, but you should definitely check out the sllm.nvim</code> GitHub repo</a> for more usage examples. The maintainer is nice to work with, so if you want to help out by opening issues or improving the plugin, don’t hesitate to do so! Stop using big bloated AI</h1> I hope I shared some useful ways you can use LLMs even if your hardware isn’t new. But what I really hope is that I can convince some of you, that use online AI services today, to switch to a local setup. My position of refusing to use ChatGPT and the like, especially as someone working with computers, is often met online by “this grumpy dude/dudette wants to stay forever stuck in a terminal and never change its ways”. There are however a lot of (other) very valid reasons. I go into more details on all this in this article’s second part, that you can read here</a>. Thank you for making it this far! I would love to hear your thoughts on the matter. You can leave a comment down below or use my contact form</a>.

Stop using big, bloated AI tools

2025-12-21T00:00:00+00:00

Foreword</h1>
The year is 2022. We are living in a world where someone, not a machine, probably wrote that article or tutorial you’re reading. Sure, the Dead Internet</a> theory is already gaining traction, but mostly, content online is created by people.
Transformers changed everything. In a 2017 research paper named “Attention Is All You Need”</a>, currently cited 207 000 times 🤯, eight scientists working for Google introduced this revolutionary new technique, and late-2022, ChatGPT (GPT for Generative, Pre-trained and Transformer) was launched. Today, NVIDIA is the most valued company on Earth, firms like OpenAI and Anthropic boast enormous valuations and make even bigger promises, and AI is being forced down our throats at every occasion.
I do not like generative AI the way it is used and pushed by our tech overlords, but I do think the underlying science, machine learning, can be very useful and not wasteful of resources if done right. Machine learning has been a thing for decades at this point: LLMs (for Large Language Models, i.e. AI chatbots) and the current transformers-induced AI boom is just one very specific way it can be used.
Like many, I’ve never used ChatGPT, Claude, Copilot, any of them. The argument “you should or you’ll get left behind” is utter nonsense: don’t take it from me, take it from people that actively use these tools everyday</a>.
The only AI that is of remote interest to me is AI I can run locally, so I’ve been keeping an eye on places like the LocalLLaMA</a> subreddit and trying out small and local LLMs. These are what we call open weight models (not to be confused with truly open source models like Olmo</a>). I want these models running on my laptop, not on a server, even if I control the server: this way, their fingerprint is kept very small (running them is no different than playing a video game).
In this article’s first part, we talked about ways to make use of small local LLMs, no matter how beefy your computer is, focusing on the `llm</code></a> Python CLI and the Neovim` `sllm.nvim</code></a> plugin: you can read it here</a>.`
`We’ll now talk more about the true benefits these tools bring, the environmental costs of using online AI, and the AI bubble and its financial consequences.`

There’s no proof you’re better off using it</h1> You probably feel like you’re coding faster by letting AI handle most of it. The truth is, especially if you’re working on complicated or legacy code bases, this study says you may be loosing time</a> because of it: [The] developers believed that the AI tools had made them 20 percent faster, on average. In reality, though, the AI-aided tasks ended up being completed 19 percent slower than those completed without AI tools. </blockquote> After reviewing screen footage, the study’s authors (METR</a>) conclude that the time spent “reviewing AI outputs, prompting AI systems, and waiting for AI generations” completely offset the gains (and then some). They however feel “optimistic” about this, saying that improvements in AI tools could lead to future efficiency gains. You could also argue that their sample size is a little small (16 developers followed). For now though, no matter what the AI start-ups say, this is at least some proof of a net loss in efficiency, in real-life scenarios, in a recent and serious study. You will however find other research with vastly different findings. Like this one (by Microsoft) that says developers complete tasks 56% faster</a> with AI (specifically their own product, GitHub Copilot). It was conducted in 2023, and in the trial they use as the study’s basis, “programmers were tasked and incentivized to implement an HTTP server in JavaScript as quickly as possible”. In my view, and unlike the study above, this is not really representative of real-life coding tasks. Another study is the 2025 DORA report</a>: this one is from Google and (which is weird to see in research, to say the least) platinum, gold and silver sponsors</a>. The report calls AI “the great amplifier”: AI accelerates software development, but that acceleration can expose weaknesses downstream. Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability. Teams working in loosely coupled architectures with fast feedback loops see gains, while those constrained by tightly coupled systems and slow processes see little or no benefit. </blockquote> I’m not saying AI never helps. What I am saying is that the benefits are unclear, even three years after ChatGPT came out, and billions upon billions invested. A couple weeks ago, it was reported</a> that Microsoft’s Azure salespeople “are seriously struggling to meet some extremely ambitious sales growth targets, cutting quotas by up to 50 percent earlier this year”; so it seems customers are also questioning these benefits. It’s one more way we destroy our planet</h1> If you don’t care about CO₂ emissions and the state of the planet, you’re either a very cynical individual and do not care what your life and that of others will look like in ~30 years, or you don’t understand the problem and its consequences well enough. Sorry, that’s just the truth. If you do care about ecology, it’s impossible to ignore the very tangible effects of “the cloud”. According to the IEA</a>, in 2022 (so “before AI”): Estimated global data centre electricity consumption in 2022 was 240-340 TWh, or around 1-1.3% of global final electricity demand. This excludes energy used for cryptocurrency mining, which was estimated to be around 110 TWh in 2022, accounting for 0.4% of annual global electricity demand. </blockquote> AI is taking this to a whole other level: data centers today are of a completely different scale. Just watch this promotional video</a> of Musk’s data center for Grok to get a sense of the sheer amount of computing power and resources poured into AI. That data center contains around 100 000 GPUs, and probably looks like a pocket calculator when compared to the Stargate project. Just look at it: Stargate is projected to have a total capacity of 5 GW of electricity</a>: that’s enough to power “4.4 million American homes”. This is not even mentioning the amount of fresh water needed for cooling (in Texas, of all places). Before the current AI boom, Bitcoin was probably the most useless, most wasteful way we were using computers and GPUs specifically (and is still up there and growing, very unfortunately). We have millions of high-end, very powerful computers around the globe, “mining Bitcoin”, which is really trying random hashes until they find one that start with a given number of 0’s</a>. It’s just useless brute-force, and we as a species are dedicating 114 million tons</a> of our global CO₂ emissions to this. I really can’t put into words how ridiculously wasteful it is. Today, we have a new player in town, and recent research suggest</a> it may already consume more than Bitcoin by the end of 2025. People are generating AI messages, images and videos at an unprecedented rate: we’re really only at the beginning of this. It’s going to have devastating consequences</h1> I won’t talk here about the effects AI is having and will have in the future on society. Won’t talk about how it has thrown us into a world where we just can’t trust anything we see online anymore. Nor will I mention that it’s created based on large-scale theft</a>: “that genie is out of the bottle”, as Linus Torvalds puts it</a>. I want to focus here on the AI bubble and what will happen when it’ll burst. Before getting into the bad stuff, some good will probably come out of it. My thoughts on the matter are largely summarized by this quote from a Cory Doctorow recent speech, and I highly recommend you read the whole thing</a>: AI is a bubble and it will burst. Most of the companies will fail. Most of the data-centers will be shuttered or sold for parts. So what will be left behind? We’ll have a bunch of coders who are really good at applied statistics. We’ll have a lot of cheap GPUs, which’ll be good news for, say, effects artists and climate scientists, who’ll be able to buy that critical hardware at pennies on the dollar. And we’ll have the open source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video, describing images, summarizing documents, automating a lot of labor-intensive graphic editing, like removing backgrounds, or airbrushing passersby out of photos. These will run on our laptops and phones, and open source hackers will find ways to push them to do things their makers never dreamt of. </blockquote> As we’ve seen in this article’s first part, “open [weight] models that run on commodity hardware” can already be very useful. But what about the financial consequences? We’re already seeing some troubling signs. Le Monde, a French newspaper, just published an article</a> about the Oracle valuation, and calls it “a symbol of the AI bubble’s bursting”. You may remember that, for a short while, Larry Ellison was the richest man on Earth after the price of Oracle stock skyrocketed back in September: that was after the announcement of a 300 billion (I mean, numbers no longer make any sense at this point) deal with OpenAI. The share reached $345 at most. Today, that same stock is worth $191: that’s a 45% plunge. Oracle doesn’t have customers, it has prisoners</a>. They’re a very profitable business, but at this point, their fate is entangled with that of NVIDIA, OpenAI, all of the big AI players: those circular deals people are worried about. Two months ago, the Financial Times published this piece</a> about 10 different AI start-ups that, all together, make up a one trillion dollars valuation. That’s 1 000 billion, or 1 000 000 million dollars: Tech has endured boom and bust cycles. The dotcom crash in 2000 decimated a generation of internet companies, and VCs are still picking through the debris left after a software investment frenzy stoked by low interest rates came to a juddering halt in 2022. But the current scale of investment is of a different magnitude. VCs invested $10.5bn into internet companies in 2000, roughly $20bn adjusted for inflation. In all of 2021, they put $135bn into software-as-a-service start-ups, according to PitchBook. VCs are on course to spend well over $200bn on AI companies this year. […] The deals with chipmakers, like VC investment, are a bet that AI demand will continue its stratospheric growth, helped along by research breakthroughs and new products. </blockquote> I’m not an economist, I am not well-versed in these subjects, but I fear it won’t be pretty. The AI bubble is estimated by some to be 17 times bigger than the dot-com bust and 4 times the subprime bubble</a>. We’re probably in for another financial crisis: AI companies will go under, people will get fired, markets around the world will be affected, buckets upon buckets of money will disappear… and Michael Burry will get richer</a>. It sucks all around, and we probably can’t do much to prevent it. But what can we do? At an individual scale, really only one thing: stop using online generative AI. Use local models if you must, and brace for impact. The faster the bubble bursts, the sooner the madness stops, and, I dare hope, the sooner we can focus on keeping a habitable planet and improving lives around the globe instead of continuing to waste resources we can’t spare. Thank you for making it this far! I would love to hear your thoughts on the matter. You can leave a comment down below or use my contact form</a>.

zoug.fr

Local LLMs on potato computers feat. the llm Python CLI and sllm.nvim

Stop using big, bloated AI tools

The world's most interesting contact form

Contact</h1>
You can send me an end-to-end encrypted</strong> email by filling out this form:</p>

zoug.fr

Local LLMs on potato computers feat. the llm Python CLI and sllm.nvim

My laptop config</h2> Meet my laptop, the ASUS PC Portable Gamer FX552VE-DM380T</em>, bought in February 2018:</p>

Stop using big, bloated AI tools

The world's most interesting contact form