Which one is the fastest and how fast is it? Even the "fast" ones doesn't seem to even reach close to consumer NVIDIA GPUs released years ago when it comes to prompt processing.
indeed. make up your mind, and also "slow/fast" mean very little given plethora of models to choose from. fast for what, slow for what. fast with which harness, etc...
But more memory is available so a larger model can be loaded. It depends on the use case which is better. Any chat or voice model will have better UX with nvidia but document or code generation will be better with apple.
I hope we will soon have an open-source project for training such small LLMs where one can just pick the architecture (like Qwen / DeepSeek / etc.), parameter count, dataset, ... and then let it run on a local/rented GPUs...
I've been training different architecture 1.33b and 3.33b models using nvidia megatron for about a month. You can train a 1.33b on a 46gb mac and 3.33b is about as big as you can go with a single 96gb blackwell with regular checkpoints etc
I’m with you. But what are we going to get? I think this goal sits in a funny place between knowledge and convenience.
On the one hand, tons of “products” promising this.
On the other hand, I’m sure we can find student works—sharing the code they created from a course or book. And I expect there will be gaps, niches filled by pro-coders who see a pro need and fill it.
What I don’t want to do is see the market for model training filled with whatever Microsoft thinks will make money.
Yep. All modern Intel and AMD architectures, save maybe some Atom variants, effectively JIT compile the incoming instructions. It's part of the reason the faster-than-static Java/JVM promises never came to fruition. The goal posts moved after the mainstream chips coopted the approach, in a sense, and they continue to move as pipelines get deeper and broader and transistor count goes up.
Transmeta was a classic failure--right idea, wrong place & time.
Intel and AMD were more or less 'Jitting' since the PPro and K5, and that's ignoring NexGen's Nx586 (which was shipping in hardware about a year before the PPro).
All of them are way before Transmeta shipped a CPU. Where Transmeta tried to innovate was by using VLIW rather than a RISC-like core, along with their other special tech (Which they later licensed to lots of companies, including Intel,) to provide reasonable mobile performance with a low power draw. Oh, there's also the bit where Transmeta CMS is much more software based (likely, partially to push enough to software and avoid an x86 CPU license lawsuit) but even modern Intel chips AFAIK are still doing more translation on the CPU layer.
IBM’s microcode is really something. Not too long ago I heard about writing “millicode” as something that’s not at the microcode level, but not quite up at the ISA level.
Millicode used custom extensions to the ISA but is otherwise regular s390 code. For example there is a separate register file and the "original" registers are available with either (I don't remember exactly) special instructions or special memory addresses.
Intel by the way did the same for SGX, they called it xucode.
Is it though? Is it really? Outside the HN rant circles which want to return back to times where you needed an adapter for every single laptop model or be shit out of luck for connecting your mouse or projector?
Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B?
What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B?
I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...
If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario.
My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16.
Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all.
For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used:
llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16
So long as they have flash attention enabled, Llama.cpp enables Sliding Window Attention by default for Gemma 4 models. Even if they're using Ollama or LM Studio I would expect those to mostly be doing the right things.
You cannot wear out an SSD through AI inference alone. LLM weights are only read from the SSD, and read operations do not contribute to SSD wear. SSD wear is primarily caused by write and erase operations.
reply