Overview

oMLX is a native macOS inference server built on Apple's MLX framework, designed for the way coding agents actually work. Its paged SSD KV cache persists every cache block to disk, so when an agent circles back to a previously seen prefix it's restored in milliseconds instead of recomputed — dropping agent time-to-first-token from 30-90s to under 5s from the second turn onward. It serves LLM, VLM, embedding, and reranker models simultaneously, does continuous batching for up to 4x throughput at concurrency, and exposes drop-in OpenAI and Anthropic-compatible APIs (including a native /v1/messages endpoint) for Claude Code, Cursor, and other clients. A signed, notarized menu bar app and web dashboard handle model management, chat, and real-time metrics.

Minimum macOS: 15.0 (Sequoia)

Architecture: Apple Silicon

Key Features

Paged SSD KV caching persists cache blocks to disk (no eviction, survives restarts)
Sub-5s TTFT for agents from the second turn onward
Continuous batching for up to 4.14x generation speedup at 8x concurrency
Native macOS menu bar app (signed, notarized, auto-update — not Electron)
Multi-model serving: LLM, VLM, embedding, and reranker models at once
OpenAI + Anthropic drop-in APIs with native /v1/messages endpoint
Works with Claude Code, Cursor, and OpenAI-compatible clients
Tool calling (JSON, Qwen, Gemma, GLM, MiniMax) and MCP integration
Web dashboard for model management, chat, and real-time metrics
Built on Apple MLX for Apple Silicon