# I'm learning more about KV Cache and quantizing, and can now read 5% more tweets about local llms

By Morgan Linton · Published 2026-04-25 · X No. 11 · 602 words · https://blog.morganlinton.com/on-x/im-learning-more-about-kv-cache-and-quantizing-and-can-now-read-5-more-tweets/

![](../../assets/on-x/HGtk6kCbYAAWfOZ.jpg)

Rewind a month ago, and I'd say 99% of the tweets about all the fun stuff people have been doing with local llms looked like absolute gibberish to me.

Today, I'm proud to say I can make sense of maybe 6% or so. I won't pretend to be much further than that. The local llm community on X is insanely sharp, and fascinating to learn from.

The last month for me has been a lot of trying to run local llms on my M1 Mac Studio with 32GB of ram, and comparing that to my M4 Mac Mini with 16gb of ram, and learning all the wacky tradeoffs and complexities. Overall, I'm starting to be able to decode the names of models on Huggingface, but I still have a long way to go.

Lately, what I've been the most interested in is the concept of quantizing the KV cache. And honestly, this might not even be the right way to say it so I won't be shocked if I get flamed in the comments by someone that actually knows what they're doing.

But I think I'm starting to get it, so thought I'd share what I've learned so far, as I know there have to be at least some people like me out there that are still trying to wrap their heads around all of this.

So first, KV cache — this is just the short term memory for a model. And I think sometimes pictures help, so here’s a better illustration of what exactly KV cache is.

![](../../assets/on-x/HGwtyzSbIAA1hDG.jpg)

Out of the box, your KV cache is nice and clean, all the memories stay in nice high fidelity, but you can’t fit as many in a small space, especially something as small as a puny little Mac Mini with 16gb of ram. This is where quantization comes in. You can squish the KV cache, to something like 8 bit, which is written as q8, and then you can fit more in your short term memory.

But of course, there's a catch. As you quantize, things become a bit more fuzzy. The best example I've read is comparing coloring with a colored pencil vs. a crayon. You're going to get a lot more detail and fidelity with the colored pencil. The crayon, is your quantized KV cache, you still get all the pretty colors, but not as much detail.

![](../../assets/on-x/HGwt6A6aUAAZch5.jpg)

In model output terms, the more you quantize, it seems like, and here's where I'm still a bit fuzzy, you also can get lower fidelity in the responses, but not all the time. Some responses are easy enough for a model to do totally fine with, even when quantized, other, more complex things, maybe not so much.

If you want to be able to do more, more less, with local llms, it feels like quantization of the KV cache is pretty effective, to a point, and finding that point is what the people who are making these different version of the model seem to be doing. How much can they quantize the KV cache before the model starts to get too dumb essentially.

Okay, I just wanted to get all that out here. As you can probably tell, I'm not writing this with AI, this is just me, sitting here, pounding on the keyboard, sharing what's in my head, and maybe no quite accurate yet, but hoping to get there.

That being said, I think April was the tipping point for local llms, and Qwen 3.6 is what tipped it. If you want to learn more about KV Caching, here’s a great article to read — [https://huggingface.co/blog/not-lain/kv-caching](https://huggingface.co/blog/not-lain/kv-caching)
