Mixture of Experts (MoE) architectures offer an attractive theoretical escape hatch. While the total parameter count can be large (30B to 70B+), only a sparse subset of expert layers activates for any given token. However, standard local runtimes still require loading the entire weight footprint into memory, which puts 30B+ models out of reach for everyday consumer laptops with 8 GB of VRAM.

Enter FreeToken by FlashML, backed by their research paper (arXiv:2608.16157). FreeToken is an edge inference runtime designed specifically to run frontier MoE models on consumer hardware by dynamically managing a hierarchical cache between GPU VRAM and host RAM.

I recently downloaded FreeToken to put their claims to the test on my modest laptop GPU. Here is what the setup looked like, how FreeToken handles memory under the hood, and the real-world generation numbers I observed.

The Test Rig

My test machine is a portable laptop, not a high-end multi-GPU workstation:

• CPU: 13th Gen Intel Core i9-13900H (14 cores, 20 threads)
• GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
• Host RAM: 32 GB DDR5 (31.7 GiB usable)
• Operating System: Windows 11
• Runtime: FreeToken Desktop (v0.2.0-beta.17)

On paper, an 8 GB VRAM budget makes running a 35-billion parameter model look impossible. In traditional setups, 8 GB VRAM limits you to 7B or 8B parameter models in 4-bit quantizations (such as Q4_K_M). Attempting to load a 35B model typically triggers out-of-memory errors or slows inference to a crawl.

Installing FreeToken and Exploring the Library

The FreeToken desktop app is available to download directly from the FlashML website. It provides a self-contained installer that is quick to set up on Windows. When you launch it, the interface automatically detects your hardware specs, available VRAM, and system RAM, and presents a curated library of models optimized for edge offloading.

For this test, I selected Qwen3.6-35B-A3B NVFP4 (nvidia/Qwen3.6-35B-A3B-NVFP4). This is a 35-billion parameter MoE model using NVIDIA’s 4-bit floating point (NVFP4) format, with an initial download size of 21.9 GiB.

Before loading the model, the console dashboard showed clean baseline resource usage:

• GPU VRAM: 0.4 / 8.0 GiB
• Host RAM: 11.0 / 31.7 GiB
• GPU Temp: 42°C at idle

Weight Conversion: The FTW Format

Once the 21.9 GiB download finished, FreeToken flagged the model as needing conversion before it could be launched.

FreeToken converts raw Hugging Face weights into its proprietary format called FTW (FreeToken Weight format). During this process, the engine repacks and organizes the tensors into memory-mapped structures optimized for rapid streaming between host RAM and GPU memory.

The conversion took just over a minute on the i9-13900H and NVMe storage, producing a compact 19.5 GiB weight package.

Loading the Model and Memory Allocation

Starting the model triggers the weight allocation phase. Instead of attempting to cram the entire 19.5 GiB into the 8 GB VRAM, FreeToken partitions the workload across both memory tiers.

Once loaded, the memory profile was eye-opening:

• VRAM Usage: 6.8 GiB / 8.0 GiB (85% utilization, leaving a healthy buffer for OS display compositing)
• Host RAM Usage: 30.2 GiB / 31.7 GiB (95% utilization)
• GPU Status: Running at 45°C, drawing only 7W at idle

The Cache Architecture Under the Hood

The console view provides detailed insight into how FreeToken manages this memory footprint:

1. MoE Expert Cache: FreeToken allocated 916 active expert slots in VRAM out of a total pool of 10,240 slots, consuming 1.51 GiB of VRAM. As different experts are needed during generation, they are dynamically paged in and out from the host RAM cache.
2. KV Cache: 8K tokens of context were allocated in VRAM, consuming just 0.16 GiB.
3. Mamba State Slots: 24 state slots were reserved, using 1.44 GiB of VRAM.

This hierarchical design is the core reason the model can run without crashing. Inactive experts reside in system RAM (filling ~30 GB of system memory), while the active attention layers, KV cache, and currently triggered experts reside in the 8 GB VRAM.

Real-World Performance & Generation Speed

To evaluate real-world performance, I tested a conversational prompt with reasoning enabled:

User: “hi, how are you ?”

The model engaged its full thinking process, taking 1 minute and 9 seconds to plan and structure its response, and then generated 393 tokens at 4.3 tokens per second.

While 4.3 tokens per second is not real-time voice conversational speed, it is remarkably steady for running a 35-billion parameter model on an 8 GB laptop GPU. For tasks like code review, background agent execution, document analysis, or local drafting, 4.3 tok/s is completely practical.

The model detail sheet confirms the architecture and licensing details:

• Base Repository: nvidia/Qwen3.6-35B-A3B-NVFP4
• Context Length: 262,144 tokens (256K context window)
• License: Apache 2.0

Built-in Agentic Tooling & Local Endpoints

FreeToken also includes built-in support for developer tools and coding agents.

The Apps tab exposes both OpenAI-compatible and Anthropic-compatible local HTTP endpoints:

• OpenAI Endpoint: http://127.0.0.1:1919/v1
• Anthropic Messages Endpoint: http://127.0.0.1:1919/v1/messages

FreeToken provides quick configuration presets for popular coding assistants:

• Claude Code: Configurable via local Anthropic endpoint routing
• Codex / opencode / openclaw: One-click environment setups
• Hermes / DeepSeek Harness: Direct command-line integration

Having drop-in compatibility with both OpenAI and Anthropic API schemas means you can point tools like Claude Code directly to your localhost port without running a reverse proxy.

Where FreeToken Fits

FreeToken is a relatively young product, but its approach solves a real constraint for local inference.

Strengths

• Bypasses the VRAM limit: Running a 35B MoE model on an 8 GB laptop GPU without manual layer-by-layer offload tuning works out of the box.
• Hierarchical caching: The dynamic MoE expert cache keeps VRAM usage around 6.8 GiB, preventing driver resets and CUDA out-of-memory crashes.
• Local developer endpoints: Dual support for OpenAI and Anthropic endpoints makes integration with modern coding agents straightforward.
• Simple setup on Windows: No manual compilation, no CUDA toolkit troubleshooting, and a clean desktop GUI.

Limitations

• System RAM is the real requirement: While VRAM requirements drop significantly, your host RAM must be large enough to hold the repacked weights. On my machine, the model took 30.2 GB of system RAM. If your laptop only has 16 GB of RAM, you will not be able to run 35B models this way.
• Initial conversion step: Each new model requires a one-time repacking step into the FTW format before the first run.
• Generation throughput: At roughly 4.3 tokens per second, it is well suited for asynchronous workloads, coding agents, and complex reasoning queries, but not for instant conversational back-and-forth.

Wrap-up

If you have a laptop with 32 GB of system RAM and a modest 8 GB NVIDIA GPU, FreeToken is worth checking out. It makes 35B-class MoE models runnable on consumer hardware that would otherwise choke on them, keeping your data completely local.

You can download the desktop app directly from FlashML, check out the project on GitHub, and read their research paper on arXiv:2608.16157.

This is the first post in a small, practical series about running models locally. I am starting with Ollama because it gets a model running quickly without building a runtime from source or hand-managing dependencies.

Why Ollama

Ollama manages model downloads, exposes a straightforward CLI, and starts a local HTTP API. On Apple silicon, it supports the Apple GPU; Ollama’s current macOS requirement is Sonoma (14) or newer. The app stores models and configuration under ~/.ollama. The macOS documentation is the useful reference for install, storage, and logs.

Under the hood, Ollama packages model management and an API around inference backends including llama.cpp. That distinction is useful: Ollama is the convenient front door; llama.cpp is a lower-level route I can use later when I want to compare runtimes directly.

Step 1: Install Ollama

Download the macOS app from ollama.com/download, mount the DMG, and drag Ollama to /Applications. Start it once. If the CLI is not already available, the app will offer to add it to your path.

Confirm that both the CLI and the local server are available:

ollama --version
ollama list
ollama ps

On a new installation, ollama list and ollama ps should be empty. The first reports downloaded models; the second reports models currently loaded into memory.

Step 2: Pull a Model

For this machine, I started with Ornith 1.5 9B:

ollama pull ornith-1.5:9b

This feels deliberately familiar if you work with containers: ollama pull downloads the model and its layers, while ollama run starts an interactive session. The current Ollama build of ornith-1.5:9b is 6.6 GB with a 256K context window, which is a comfortable starting point on a 32 GB laptop. Ollama’s model page lists the available tags; the 35B download is 23 GB, so I would not make that the default on an Air.

After the pull completes, confirm it is available:

ollama list

Step 3: Run It Interactively

Start a chat session with the model:

ollama run ornith-1.5:9b

Use a question that resembles the work you actually do. I tested a simple greeting first, then moved on to infrastructure questions. Exit the interactive prompt with /exit or Ctrl-D.

For an initial sanity check, the model was responsive and produced a natural answer. That is useful confirmation that the model loads and runs locally, but it is not a benchmark. A real comparison needs the same prompt, context length, generation settings, and output length.

Step 4: Inspect Performance with --verbose

Ollama’s --verbose flag is a quick way to see timings after every response:

ollama run ornith-1.5:9b --verbose

On this MacBook Air, my short greeting test produced 114 output tokens at 16.68 tokens/sec, with a total duration of 7.22 seconds. The model’s thinking trace was visible before its answer.

I ran the same kind of test with Gemma 4 E4B:

ollama run gemma4:e4b --verbose

That run produced 228 output tokens at 27.62 tokens/sec, with a total duration of 8.49 seconds. It is faster in this small test, but it also generated a different and longer response. These figures are useful as a personal baseline, not as an apples-to-apples model ranking.

Ornith 1.5 9B vs. Gemma 4 E4B

Both models fit well on a 32 GB MacBook Air, but they are aimed at slightly different trade-offs.

The E in Gemma 4 E4B means effective parameters. Ollama describes E4B as a 4.5B-effective-parameter edge model (8B including embeddings), with a 128K context window and text, image, and audio support. It is designed to do useful local work without the memory cost of the larger Gemma 4 workstation models. The Gemma 4 library page has the current tags, sizes, and capabilities.

In practice, I would start with ornith-1.5:9b if I want the smaller download and a roomy 256K context window, or gemma4:e4b if I want the efficient Gemma 4 feature set. Neither of these two quick runs says which model is universally better; use the prompts you care about and record the result.

Step 5: Call the Local API

The terminal chat is only the first test. Ollama exposes an API locally at http://localhost:11434/api, so the model can be part of a script or an application.

curl http://localhost:11434/api/chat \
  -d '{
    "model": "ornith-1.5:9b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

The response includes the answer and useful timing fields. Keep this endpoint local by default. If I later expose it to another device, I will put authentication and a proper reverse proxy in front of it. I will not publish port 11434 directly. Ollama’s API documentation covers the local base URL and client libraries.

Step 6: Try the App UI

The CLI is great for testing and scripts, but the Ollama app also gives me a simple chat interface. Here, Ornith is selected in the model picker and used for a weather question.

The app can also give a model access to web tools. In this example, Ornith searched for current Singapore weather before answering. That extends the model with fresh online information when the tool is enabled, but it does not replace the model’s built-in training knowledge. Treat retrieved results as sources to verify, especially for technical or time-sensitive answers.

The UI is useful when I want to compare prompts casually. The local API is the path I will use when I want to integrate models into tooling.

Commands Worth Remembering

# Download without entering an interactive chat
ollama pull ornith-1.5:9b

# Start an interactive chat
ollama run ornith-1.5:9b

# Show downloaded models and their disk usage
ollama list

# Show models currently loaded by the runner
ollama ps

# Remove a model I no longer need
ollama rm ornith-1.5:9b

Models are not small. Treat ollama pull the same way you would a sizeable docker pull: check disk space before collecting a pile of models “just in case.”

What This MacBook Air Is Good At

An 8B to 9B class model is a good fit for private note summarisation, explaining logs, drafting YAML, lightweight coding help, and experimenting with local integrations. This is not where I expect a 70B-class model to be effortless or where I would host production inference.

There are many ways to run models locally, but Ollama is an excellent Apple-silicon starting point. It gets the plumbing out of the way so I can spend time evaluating the models themselves.

Next, I will run llama.cpp directly on the same machine, then see whether a vLLM setup is worth comparing. Stay tuned.

What I Do

Professionally, I have about 9 years of overall experience under my belt. I actually started out writing code as a .NET developer before pivoting hard into DevOps and infrastructure engineering. Since 2019, I have been deep in the Kubernetes ecosystem, architecting and managing infrastructure across physical data centers and public clouds.

Over the years, that has meant running everything from standard high-traffic web platforms to mission-critical telco workloads, airgapped enterprise environments, and AI-native GPU clusters. Nowadays, my work revolves heavily around container orchestration, SUSE and Rancher setups, vLLM inference deployments, and homelab GPU testing.

Who I Am

At my core, I am a tech guy through and through. Whether it is tackling an infrastructure puzzle at work, geeking out over the latest gadgets, or dreaming of walking the floor at CES, technology is simply what makes the most sense to me.

Outside of terminal windows and YAML files, you will usually find me playing FPS PC games, binging a good web series, or unwinding with music. I also have a massive bucket list goal: to travel the entire world before I hit my deathbed.

Why This Blog Exists

This site is where I will document the problems I run into and the solutions that worked. Expect practical write-ups on Kubernetes quirks, AI infrastructure experiments, airgapped deployments, and whatever interesting challenge I am currently knee-deep in.

No fixed publishing schedule. No corporate polish guarantee. Just authentic notes from the trenches, mostly so future-me (and maybe you) does not have to re-solve the same headache twice.

More soon.