Local LLMs in software modernization: what role can they play today?

Sources: r/ClaudeCode and Alibaba Cloud Community

Abstract

An anonymous poster on r/ClaudeCode reported that Alibaba's Qwen3.8-27B, an open-weight model compressed to roughly 22GB so it runs on a single machine, outperformed Claude Code on Opus 5 at high effort on recent real-world bug-fixing tasks. The measured result is twelve solves against eleven out of twenty-one, one seed, which the author describes as provisional, alongside a claimed advantage in median time to fix. The model ran in Pi, an open-source coding harness, and the only change made to the model itself was a swapped chat template. The second source, published by Alibaba Cloud, sets out what running a 27B model at home costs: about 24GB of VRAM, roughly 1,300 to 1,800 dollars for a used build, 4-bit quantization as the mainstream compromise, and 16GB of VRAM as the size that does not fit. It closes by recommending a six dollar per month hosted plan. We reviewed both with Kishan Chamman, our CTO.

Review and insights

The benchmark compares two harnesses, and only incidentally two models. Pi ships bare. Whoever runs it writes the skills, sets the instructions and decides what the task even is. Claude Code arrives configured by Anthropic and knows how to code before anyone types anything. So the comparison in that post is between a setup someone tuned by hand and a setup that came out of the box, and the result can be completely real while saying very little about the two models underneath it. The closest analogy is iOS against Android. One hands over a fixed set of tools and expects them to be used as intended. The other ships bare and gets shaped by whoever owns it, which suits some teams and irritates others.

A change in output cannot be traced to the model or to the harness from the outside. Plenty of people have the impression that Opus 5 and Sonnet 5 are more verbose than 4.8, spending more tokens and more minutes to arrive at the same place, and we share it. We cannot substantiate it, so it stays an impression. Anthropic did publish that the tokenizer changed with this generation. Past that it is guesswork, because the model and the harness produce output together and both get updated on someone else's schedule. Owning the harness closes that gap, and it is a better argument for local models than the quality claim this post is built on.

The reason to put a model on our own hardware is access. Some codebases sit on machines with no network connection. A box with a model on it can be carried into that room and do the work where the data already is. The same holds in a building with bad wifi, which is easy to forget from an office with fibre. Once the model is local the rest follows: inference happens on the machine, every intermediate conclusion stays on the machine, and the worries about training, telemetry and guardrails that block content stop applying. Neither source mentions any of this, because both are busy arguing about quality and price.

Both sources compare a purchase to a price that is being subsidized. The Alibaba piece puts a 1,300 dollar machine next to a six dollar monthly plan, and six dollars a month is not what that compute costs to serve. The same goes for the two hundred dollar subscriptions the Reddit thread treats as the going rate. We know how wide the gap is because we pay API rates on purpose, so that prototype work stays outside training, and a couple of prototypes ran to a few thousand dollars in tokens. A subscription and an API key buy the same compute at wildly different prices, and the cheap one is a customer acquisition cost. Any build-or-subscribe sum resting on it will move when the subsidy does. We wrote about CAPEX and OPEX earlier this year, and a machine for local inference belongs in that conversation, depreciation and electricity included.

The hardware that makes this pleasant is enthusiast-hardware. One commenter in the thread reports good results on an RTX 4090 running a 200k context, which was the most expensive consumer card on the market when it arrived, and the 5090 holds that position now. Cards like that are not for everyone. A MacBook Pro with 64GB of memory can run a model this size and it will be slow. Then there is a constraint that does not show up in any benchmark number: a model loaded into memory is holding memory that the working day needs, and a machine that can only run the chatbot is a machine nobody can work on. Add a second person asking it questions and the hardware requirement climbs again, roughly in step with the number of people using it. A local model is a single-seat tool until somebody builds it into infrastructure.

The hard part of modernization is human, and none of this touches it. An LLM cannot infer why things were done a certain way, as Kishan put it in the recording. We still sit down with the developers and teams who made those decisions, and that work has not sped up at all. The legwork around it has: building a picture of a codebase, reading repository history to see who owned which parts and what changed when, working through a documentation set faster than any team manages by hand. We came at this from the other direction when we reviewed a platform that promised to engineer the human work away, and moving the model onto local hardware does not change the shape of the problem. Planning stays with us as well. A model leans toward whatever it was trained on, and it can be talked into playing an independent sparring partner, but there is no golden rule for these decisions. Several approaches are defensible, each with a price, and somebody has to decide which risks the organization takes and what it is willing to spend.

The link to software modernization

The assessment is where a local model pays for itself first, and what makes it work is a skill. We run something close to it internally already: a skill that walks a set of folders, reads the documents inside and pulls out the requirements, set up with one line of instruction. An assessment has the same shape, pointed at a repository, a documentation set and a stack of interview transcripts, with our own assessment framework as the context it works from. The newer harnesses need far less written instruction than this used to take, because they infer most of it from a short set of guidelines.

Context decides most of what can stay local. A cloud endpoint offers a million tokens and a well-equipped local machine tops out around a quarter of that, which bites hardest in the assessment, the most context-hungry part of the project. The material decides the rest. An organization that cannot let its source code, its documentation or its recorded interviews leave the building is in a different position from one that signed a zero-retention agreement and is comfortable with it.

So this is a decision per phase. Analysis of material that cannot cross a network boundary can run on hardware inside the boundary, slower and with less context, and it leaves behind an artifact the organization owns. Long-context synthesis and the heavy generation work stay where the capacity is. Organizations that answer the question once for the whole project end up either with hardware they use twice a year or with material outside the building that was never allowed to leave.

Concluding remarks

The post gets the important thing right. A model running on hardware a team already owns can do real work on real code now, and that was not true six months ago. The conclusion drawn from it is another matter, because twelve solves against eleven, one seed, a custom harness against a default one, settles nothing. Someone in the same thread ran the setup against a 17,000 line React codebase and found it strong on implementation and weak on review, where its top critical finding was a missing file that was not missing. The Alibaba piece is straight about the hardware and crooked about the conclusion, laying the costs out accurately and then recommending its own subscription.

Two things are worth avoiding here. Buying hardware to solve what is really a configuration problem is the first. Settling the local question on today's prices is the second, since those prices are being subsidized. The cheaper move is to work out which phases of the work touch material that cannot leave the organization. That answer does not expire when the next model ships.

Questions we kept coming back to

What is a local LLM, in plain terms? A model whose files sit on a machine in the building, answering questions without sending anything over the internet. The model in this story is around 22 gigabytes, which fits on a well-specified laptop or a small desktop, and it runs with no account, no subscription and no connection. The model itself still comes from somebody, since Qwen is Alibaba's, so what changes is where it runs and who sees the questions put to it. We have argued before for keeping models swappable, and this is that argument with the hardware included.

What does one cost to own? A capable second-hand machine runs 1,300 to 1,800 dollars at mid-2026 prices, and the honest figure includes what comes after. Electricity for a machine under load through a working day, replacement within a couple of years as models grow past it, and the hours somebody spends setting it up and keeping it current. Cloud access for the same work is a monthly line with none of that attached. On money alone the two options sit close enough together that the deciding factor is usually something else.

Is it as good, and is it fast enough? It is a compressed copy of the model, so it is measurably weaker, and it is slower than a cloud service. Compression, called quantization, shrinks a model to fit the memory available, and published benchmark scores are always measured on the uncompressed original. In practice that shows up as more mistakes on the hardest reasoning and more variation between runs of the same request, which matters in an assessment that has to hold up as the basis for a plan. Speed has two parts. Text comes out at around 26 to 30 tokens per second on a mainstream setup, tokens being roughly word-sized chunks, so it writes faster than anybody reads. The waiting happens before that, because loading a large set of documents into the model can take several minutes before the first word appears.

Can a whole team use one machine? Not comfortably. One person at a time gets the whole machine, and every additional person putting questions to it raises the hardware requirement roughly in step. That makes a local model a tool for one specialist or one phase of work. Turning it into something a department shares means buying servers and running them, which is a separate project with its own budget and its own owner.

Does somebody have to look after it? Yes, and this is the cost most easily missed. A working setup involves a particular model file, a particular compression of it, the harness it runs inside and a set of instructions that shape how it behaves, and any one of those can quietly spoil the output when it changes. One tester in the thread we reviewed found that a stray piece of text inside a file could switch the model's reasoning off silently, with nothing recorded anywhere. An organization doing this seriously needs a named owner for it, the same way somebody owns the build server.

Where to start

Before this becomes a tooling decision, an organization needs to know which parts of its estate hold material that cannot leave, and what a serious analysis of that material would take. A modernization assessment produces both, along with the picture of the current system that any tool, local or hosted, needs before it can contribute anything. That is where we would start.

Full video episode

The first step to start your modernization journey

Software modernization and architectural rebuilds lie at the heart of Eli5. We solve complexity to deliver direct business value by focusing on pragmatic, cloud-native transitions.

Before you decide whether to wrap your legacy system, buy a new SaaS product, or use AI to build custom tools, you need total visibility into your current tech landscape.

Would you like to book a free brainstorm to discuss your legacy stack? It is the essential first step to turning your technical debt into a scalable, modular future.

‍

‍

‍

‍