Ollama

In the previous post, I ran Ornith 1.5 9B on my 32 GB MacBook Air using Ollama and recorded baseline token-generation speeds on short prompts. Ollama is great for getting up and running quickly, but under the hood, its inference engine is built on llama.cpp. In this post, we go one level down: building and running llama.cpp directly, offloading inference to Apple Silicon’s Metal GPU, and comparing performance numbers side by side with Ollama on the exact same model and quantization level. ...

I have a 32 GB MacBook Air. It is not a workstation GPU box, but its unified memory makes it a surprisingly capable machine for local models, provided I choose models that fit and keep expectations sensible. This is the first post in a small, practical series about running models locally. I am starting with Ollama because it gets a model running quickly without building a runtime from source or hand-managing dependencies. ...