Overview

ToshLLM is a free, open-source macOS app that runs modern open-weight language models locally on Intel Macs with discrete AMD GPUs - the machines most local-LLM tools skip in favour of Apple Silicon. It ships a Metal-optimized inference engine with custom attention and matmul paths, detects your hardware, and recommends models with per-GPU VRAM and RAM estimates so you know what will actually fit. Around that sits a native SwiftUI app: persistent chat with Markdown and code copy, agent tools and MCP integration, on-device dictation via Apple's Speech framework, vision for image analysis, and beta local image generation. It exposes an OpenAI-compatible API on 127.0.0.1:8080 with a router mode that auto-loads the right model per request, supports multi-GPU and eGPU setups, and includes built-in benchmarking with historical comparison charts. Everything runs on-device with no accounts, no telemetry, and no per-token bill.

Pricing: Free (GPL-3.0 open source)

Minimum macOS: 14.0 (Sonoma)

Architecture: Intel

Key Features

Metal-optimized inference engine with custom attention and matmul paths
Built for Intel Macs with discrete AMD GPUs, including multi-GPU and eGPU setups
Hardware detection with per-model VRAM and RAM estimates
Native SwiftUI chat with persistent conversations, Markdown, and code copy
Agent tools and Model Context Protocol (MCP) integration
On-device voice dictation using Apple's Speech framework
Vision support for image analysis with automatic projector matching
Local image generation in beta - text-to-image, image-to-image, and upscaling
OpenAI-compatible API at http://127.0.0.1:8080
Router mode that auto-loads the right model per request
Mixture-of-Experts optimization with automatic --n-cpu-moe calculation
Automatic lossless speculative decoding
Built-in benchmarking with historical comparison charts
Profiles, menu bar mode, and auto-start
Bilingual interface and docs in English and Spanish
Fully offline - no accounts, no telemetry, no per-token cost