Colibri Shows How to Run GLM-5.2 on a Low-Spec PC, but Storage Sets the Pace

Colibri has shown how to run GLM-5.2 on a low-spec PC, despite the model containing 744 billion parameters. The open-source engine keeps only 9.9 GB of quantized dense weights in memory. It retrieves the remaining expert weights from an SSD as GLM-5.2 generates each token.

That sounds like a way around the hardware wall surrounding large local models. However, Colibri does not make the model small or instantly responsive. Its original consumer laptop produced roughly 0.05 to 0.1 tokens per second during cold decoding.

The conflict is therefore more interesting than the headline. Colibri makes a huge model fit, but fitting is not the same as delivering useful interactive performance. The project shifts the limiting resource from memory capacity toward storage bandwidth, caching behavior, and patience.

Developer Vincenzo, known online as JustVugg, introduced the project through a Show HN discussion. He described a one-person experiment built on a 12-core laptop with 25 GB of usable RAM. The submission attracted hundreds of comments from developers debating whether slow, local access to a large model has practical value.

That debate pressures both sides of local AI. Developers can no longer treat insufficient RAM as an absolute reason that a model cannot run. At the same time, local inference advocates must distinguish technical execution from a usable product experience.

Colibri’s answer is not a smaller model. It is a different memory hierarchy. The engine treats RAM, optional VRAM, and NVMe storage as tiers that can hold different parts of the same model.

The result is an unusual proof of concept with broader implications. Future consumer AI systems might not need every model parameter sitting in expensive high-speed memory. They might instead predict which parameters matter next, move them early, and accept a measurable performance tradeoff.

How Colibri Runs GLM-5.2 on a Low-Spec PC

Colibri changes where GLM-5.2’s parameters wait, rather than changing which model performs the work.

GLM-5.2 is a mixture-of-experts model, or MoE. An MoE contains many specialized feed-forward modules, but its router selects only a small subset for each token. That sparse activation makes the model’s total parameter count much larger than its active computation on any single step.

The model has about 744 billion total parameters, according to the project. Only about 40 billion become active for each token. Colibri uses that difference to separate weights that must remain available continuously from weights that can be retrieved when selected.

Attention components, embeddings, and shared experts form the dense portion. Colibri quantizes those weights to int4, a four-bit representation that reduces storage and memory use. The project says this resident portion covers about 17 billion parameters and occupies approximately 9.9 GB of RAM.

The routed experts live elsewhere. The current Colibri architecture describes 19,456 routed experts across 75 MoE layers and the model’s prediction heads. Each expert occupies roughly 19 MB in int4 form, while the complete converted model requires about 370 GB of disk space.

The original Show HN post reported 21,504 routed modules. The repository now reports 19,456 after subsequent development. That change is a useful reminder that Colibri remains an active project rather than a fixed commercial specification.

When GLM-5.2 processes a token, its router chooses the experts needed for each layer. Colibri requests those weights from an SSD, stages them in available memory, performs the calculation, and then makes room for later experts. A least-recently-used cache retains modules that are likely to return.

The project compares this behavior with just-in-time compilation. A compiler does not optimize every path before a program starts. It observes which paths become important and spends resources on those sections.

Colibri applies that logic to model weights. Frequently selected experts can remain in RAM or VRAM, while colder experts stay on disk. The system gradually records routing activity and pins the hottest modules when memory permits.

This arrangement does not alter the router’s decisions merely to save memory. According to the repository, storage placement changes speed but should not silently change weight precision or routing semantics. The project reports token-level validation against a Transformers reference implementation.

That distinction matters because aggressive expert pruning would be a different claim. A pruned runtime might ignore selected experts or substitute smaller components. Colibri instead attempts to execute the quantized model faithfully while paying for limited memory through additional data movement.

The engine also compresses the key-value cache, which stores attention information from earlier tokens. Its implementation uses GLM-5.2’s latent attention structure to reduce the stored state per token. Colibri can preserve that cache between sessions, avoiding a full prompt replay when a conversation restarts.

These techniques answer the narrow feasibility question. A consumer system with enough SSD capacity can load the required dense weights, fetch routed experts, and produce valid output. The much harder question concerns how long that output takes.

The RAM Barrier Became a Storage Problem

Colibri does not eliminate GLM-5.2’s hardware demand. It transfers much of that demand from memory capacity to repeated storage reads.

A conventional local inference setup tries to keep most or all model weights in RAM or VRAM. That approach allows processors to access parameters without waiting for an SSD during every generated token. However, a model with hundreds of billions of parameters exceeds the memory available in ordinary consumer systems.

Colibri exploits the gap between total and active parameters, but active does not mean resident. The model still needs selected experts during every layer of every token. If those experts are absent from memory, the engine must retrieve them before computation can continue.

The project estimates that routed weights changing from token to token represent about 11 GB of data. Cache hits and repeated routing can reduce physical reads, but a cold workload can still demand sustained SSD activity. Storage latency and throughput therefore sit directly inside the decoding path.

This is why the original laptop managed only around 0.05 to 0.1 tokens per second. A single token can take between 10 and 20 seconds at those rates. A short answer containing 100 generated tokens could require many minutes.

That performance does not resemble an interactive cloud chatbot. The model might be suitable for an unattended experiment, a long-running analysis, or verification that a particular prompt works. It is far less suitable for rapid coding assistance that depends on frequent exchanges.

Colibri includes several methods designed to narrow the gap. Its asynchronous input and output pool retrieves missing experts while resident experts are still computing. A look-ahead process attempts to predict routing for the next layer and prefetch the corresponding weights.

The repository says next-layer routing was 71.6 percent predictable in its measurements. That figure comes from the project rather than an independent laboratory. Still, it explains why routing structure can create a useful cache instead of producing completely random storage requests.

The engine also groups duplicate expert requests across batched positions. If several positions need the same expert, it reads that expert once. Adjacent matrices are stored together so one operation can retrieve the necessary data.

A second SSD can add another source of read bandwidth. Colibri supports a mirrored model distributed across two drives through deterministic expert placement. The approach requires another copy, or at least a partial mirror, so its capacity requirements can become substantial.

Faster hardware changes the balance. More RAM allows additional experts to remain pinned. More VRAM creates a faster tier for the hottest modules. A quicker SSD reduces the delay for experts that remain cold.

The current repository shows a system with six RTX 5090 GPUs generating about four tokens per second when all experts are resident. That example no longer represents low-spec computing, but it demonstrates the same engine operating across different storage tiers.

The project therefore exposes a continuum rather than a binary result. At one extreme, a 25 GB machine streams almost everything and responds slowly. At the other, a large GPU system holds the experts in fast memory and removes disk access from decoding.

Most users will fall somewhere between those points. Their results will depend on SSD bandwidth, cache size, model usage patterns, CPU throughput, and the operating system’s storage behavior. “Runs locally” cannot summarize all those differences.

This reframing matters beyond Colibri. Consumer AI hardware discussions often focus on total memory as if it alone determines model access. Colibri demonstrates that model architecture and data placement can loosen that constraint, while revealing the next bottleneck underneath it.

Fit Versus Speed Is the Real Contest

The central contest is not local AI versus cloud AI. It is mathematical feasibility versus useful response time.

Colibri’s original goal was deliberately modest. JustVugg wrote that he wanted GLM-5.2 to work on his computer “even slowly.” By that standard, the project reached its target.

The developer converted the model to int4, implemented its attention path, and streamed experts without exhausting available RAM. The engine produced tokens from a model that appeared far too large for the host machine. That is a meaningful engineering result.

However, most users judge an inference system through latency. They care whether a coding assistant can complete a function before their attention moves elsewhere. They care whether a local research tool can summarize documents within a work session.

At 0.05 tokens per second, feasibility offers little comfort for those interactions. Even one token per second feels slow during conversation. The original numbers place Colibri closer to offline batch processing than responsive assistance.

The project’s evolving benchmark records provide a more nuanced picture. Different contributors have tested faster SSDs, larger memory pools, Apple Silicon, and dedicated GPUs. Results vary because each configuration changes the share of experts served by storage.

That variation is not a weakness in the concept. It is the central fact readers need when assessing the claim. Colibri cannot promise one consumer-wide speed because “consumer computer” covers very different memory and storage systems.

The Hacker News reaction captured both admiration and skepticism. Some commenters viewed 0.05 to 0.1 tokens per second as unusable. Others argued that slow local inference still serves overnight jobs, experiments, private workloads, and situations where remote access is unavailable.

Both positions can be correct. A developer testing model behavior might tolerate a long wait to avoid acquiring specialized hardware. A company placing the engine inside an interactive customer tool probably cannot.

The practical comparison also includes smaller local models. A compact model that fits entirely in RAM can generate output far faster, even if its answers are less capable on difficult reasoning tasks. Many everyday prompts do not require a 744-billion-parameter model.

That creates an uncomfortable tradeoff for the large-model approach. Users receive access to more of GLM-5.2’s capacity, but they surrender responsiveness. A smaller model offers lower theoretical capability while completing routine work sooner.

Cloud inference occupies another point on the spectrum. Hosted services keep large weights on high-bandwidth accelerators and amortize that infrastructure across users. Their disadvantages include network dependence, external data processing, provider restrictions, and limited control over the execution stack.

Colibri does not defeat that economic model simply by producing a token at home. It provides a local fallback and an experimentation platform. It also supplies evidence that sparse models can be staged across cheaper hardware when latency is secondary.

This is why the project pressures local inference claims more than cloud providers. Developers promoting local AI must now specify the workload, decoding rate, prompt-processing time, storage traffic, and output quality. Parameter count and RAM usage alone are not enough.

Hardware coverage should become equally precise. A machine might meet the memory requirement yet lack the SSD capacity for 370 GB of converted weights. Another machine might have sufficient capacity but use a drive that cannot sustain heavy reads.

Endurance also deserves attention, although the workload is dominated by reads rather than writes. Thermal throttling, virtualized storage, and drive cache design can change sustained performance. A brief disk benchmark cannot fully predict a long generation session.

Colibri’s value lies partly in making these constraints visible. It turns the question from “Can this model fit?” into “Which tier serves every expert, and how often does the engine wait?” That is a better framework for judging local AI systems.

Quantization and Verification Still Need Scrutiny

A successful generation does not establish that int4 GLM-5.2 preserves every capability users expect from the original model.

Quantization reduces the number of bits used to store each weight. This compression makes local execution possible, but it can introduce errors. The effect depends on the quantization method, model architecture, task, and sensitivity of specific layers.

Colibri says its default policy preserves model precision after conversion and keeps router semantics unchanged across storage tiers. That means the engine should not lower precision further just because RAM becomes scarce. It does not mean int4 behaves identically to the original higher-precision weights.

The repository reports token-exact validation of parts of its forward pass against a reference implementation. That kind of engineering test can catch implementation mistakes, including incorrect attention behavior or weight loading. It cannot by itself measure broad capability retention across coding, reasoning, multilingual work, and long-context tasks.

GLM-5.2 also carries its own claims. The broader GLM family was developed for reasoning, coding, and agentic tasks. The published GLM research paper describes architectural work intended to reduce inference costs while preserving long-context behavior.

Those model-level results should not automatically transfer to Colibri’s int4 container. A fair evaluation would compare the same prompts across the original model, the converted weights, and alternative local models. It would also hold chat templates, sampling settings, and context lengths constant.

The initial Show HN post acknowledged this open question. JustVugg described testing how GLM-5.2 responded after int4 conversion and whether quality remained acceptable. The project has since added benchmark tooling, but community results still require careful interpretation.

One early technical issue illustrates the risk. The repository warns that an original converted model mirror used int4 prediction heads, which caused zero draft acceptance. The current setup recommends int8 prediction heads for that component.

That problem did not necessarily make ordinary decoding incorrect. It affected speculative decoding, a method that drafts several future tokens and verifies them together. Still, it shows how one conversion choice can disable an important optimization.

A second uncertainty concerns benchmark representativeness. Standard multiple-choice tests can measure whether a runtime produces plausible results, but they do not capture every use case. Long coding sessions and tool-using agents depend on formatting, context retention, and repeated decisions.

The 370 GB download also creates a verification challenge. Users need confidence that they obtained the intended files, selected the correct prediction heads, and launched the runtime with suitable settings. Configuration errors can look like model weakness.

Colibri now provides planning and diagnostic commands that inspect hardware placement before generation. That improves transparency. The runtime can show which weights will occupy VRAM, RAM, or disk, helping users identify an avoidable bottleneck.

Independent coverage has maintained an appropriately cautious tone. A hardware analysis characterized the project as a proof of concept and emphasized its slow original decoding speed. It also identified NVMe access as the first major constraint on limited machines.

That description remains useful even as the repository adds GPU support and caching improvements. The essential 25 GB claim concerns execution, not a guarantee of production-ready throughput. Readers should resist collapsing those ideas.

The project’s open development model helps. Developers can inspect the C implementation, reproduce measurements, and submit results from different systems. Yet popularity, stars, and successful screenshots cannot replace controlled quality testing.

A careful conclusion is therefore possible. Colibri has credible evidence that GLM-5.2 can execute on a memory-constrained consumer machine. The broader claim that this configuration delivers the model’s full practical value remains workload-dependent and incompletely verified.

Who Should Actually Try Colibri

Colibri makes the most sense when local control and model access matter more than immediate answers.

The first audience is inference researchers. Colibri exposes routing, expert residency, cache heat, and storage placement in a comparatively compact codebase. That makes it useful for studying how sparse models behave outside a data center.

A developer can observe which experts a workload activates and how frequently those experts return. That information can support better prefetching, placement, and scheduling policies. It may also reveal whether specialized workloads use a much smaller working set than general chat.

The second audience is open-weight enthusiasts who want to inspect GLM-5.2 directly. They might test prompts, compare quantization behavior, or verify that the weights can run without a hosted API. For them, slow output can be acceptable because access itself is the objective.

Private batch work represents another possible use. A machine could process sensitive material overnight without sending prompts to an external service. This scenario still requires proper endpoint security, storage encryption, and access controls. Local execution alone does not create a complete privacy program.

Disconnected environments also have a reason to care. A local copy can continue operating when a network service becomes unavailable. However, downloading and storing the model requires substantial preparation, and updates will not arrive automatically.

Colibri is less compelling for someone who wants a responsive everyday chatbot on an ordinary laptop. A smaller local model will usually provide a better interactive experience. It can remain in memory and avoid retrieving gigabytes of expert weights during decoding.

The same applies to coding workflows built around rapid feedback. Developers frequently ask follow-up questions, inspect partial output, and change direction. Waiting many seconds for each token undermines that loop even when the eventual answer is strong.

Teams should also consider operational complexity. The current model container occupies roughly 372 GB, while a second-drive mirror requires additional capacity. The system needs a compatible build or release, sufficient free memory, and a fast storage path.

Colibri’s setup has become more approachable since the first Show HN release. The repository provides prebuilt packages for Linux, macOS, and Windows. Its engine is pure C, although the launcher, conversion tools, and optional API gateway use Python.

An OpenAI-compatible endpoint lets existing clients send requests to the local engine. That interoperability is important because it separates the inference experiment from the user interface. Developers can keep their existing tools while changing the backend.

Still, compatibility does not create equal performance. An application designed around cloud-speed token streaming might time out or provide a poor experience. Any integration needs generous limits and clear progress reporting.

A better use pattern is asynchronous. A user submits a bounded task, allows the machine to work, and returns later. Repository analysis, document classification, or scheduled evaluation can tolerate latency better than chat.

Even those workloads need measurement. Prompt ingestion, generated length, expert cache reuse, and SSD temperature can change completion time. Teams should test representative jobs rather than extrapolating from a short demonstration.

The key decision is not whether Colibri is impressive. It is whether GLM-5.2’s additional capability provides enough benefit to justify slower output and extensive local storage. For many users, the answer will remain no.

For researchers and determined enthusiasts, the answer can be yes. Colibri offers something smaller runtimes cannot: direct access to an unusually large sparse model on hardware that would normally reject it before producing one token.

Three Signals Will Decide What Comes Next

Colibri’s next phase will be judged by reproducible speed, preserved quality, and support for additional sparse models.

The first signal is independent performance on common consumer hardware. Results should include cold and warm decoding, time to first token, prompt-processing speed, RAM use, and sustained disk throughput. A single peak rate cannot describe the experience.

Measurements from widely available laptops and desktops would establish whether learned caching makes the engine faster over repeated workloads. They would also show how much improvement comes from faster storage versus additional RAM or VRAM.

If multiple systems approach usable asynchronous speeds without storing most experts in memory, Colibri’s central argument becomes stronger. If results remain near the original cold rate, its 25 GB mode will remain primarily an engineering demonstration.

The second signal is quality evaluation for the recommended int4 conversion. Colibri needs comparisons against higher-precision GLM-5.2 across coding, reasoning, multilingual prompts, structured output, and long contexts. Those tests should publish configurations and raw outputs.

Strong quality retention would validate the decision to trade latency for access to a much larger model. Significant degradation would favor smaller models that fit fully in memory and run at higher precision.

The third signal is whether the memory hierarchy generalizes. The repository says GLM-5.2 and OLMoE already run, while additional MoE families remain on the roadmap. Broader support would turn Colibri from a model-specific experiment into a reusable inference design.

Generalization will not be automatic. MoE models differ in expert layout, attention architecture, routing behavior, and prediction heads. Each integration must preserve semantics and deliver enough cache structure to justify disk streaming.

Success would pressure other local runtimes to treat SSDs as an active model tier. It could also influence model developers to design expert layouts that cooperate with hierarchical memory. Predictable routing and compact expert modules would become deployment advantages.

Failure would still leave a useful result. Colibri has already demonstrated that insufficient RAM does not always make a sparse model impossible to execute. It has also shown why memory capacity cannot be separated from bandwidth and latency.

For readers considering how to run GLM-5.2 on a low-spec PC, the immediate question is simple: do you need responsive conversation or controlled access to the weights? Choose a smaller resident model for speed. Test Colibri when local ownership, experimentation, or offline execution outweighs the wait.

Then record the entire configuration and share reproducible results. Colibri’s future depends less on another dramatic parameter count than on ordinary machines producing comparable evidence.