Research
One question drives our research: how does an on-device model keep learning after it ships? Neural Imprint is our answer; the inference-efficiency work below is what lets it run on phones and PCs, across Apple and Intel silicon. Every piece unblocks a real product.
Neural Imprint: Models That Grow With Their Users — On-Device AI Evolution Without Weight Updates
Our paper on on-device continual learning. Instead of updating weights (fine-tuning) or re-injecting text every turn (RAG), Neural Imprint distills a compact representation of the user from local behavioral data and persists it as a recoverable inference state that survives across sessions. Base weights stay frozen, nothing leaves the device, and every state is verifiable and instantly reversible.
- ▶Validated on 8 application domains, 2 model sizes (Qwen3.5-4B / 9B) and 2 iPhones, using synthetic behavioral data (3,283 records)
- ▶16/16 A-library health gates pass (8 domains × 2 model sizes)
- ▶16/16 personalized profile responses complete with zero tool calls; 16/16 autonomous tool invocations correct
- ▶Entire pipeline runs within the 6 GB memory limit of consumer iPhones, zero cloud dependency
- ▶Manuscript, 2026 — full text not yet public
On-Device Speech and 9B Dialogue on Intel: A Streaming Engine Carried Over From Apple
The streaming speech engine we built for Apple silicon, carried to Intel Core Ultra (CPU + Arc iGPU) on our own OpenVINO-based inference runtime. Speech recognition, a 9B language model and speech synthesis stay resident on one machine; delivered twice to a world-leading PC manufacturer.
- ▶Stateful streaming vocoder: 0.4 s blocks, 115.9–116.4 dB SNR against whole-sentence output
- ▶First sound in 0.37 s, speech synthesis RTF 0.73; the open-source reference takes 15–20 s at 2.1–2.9
- ▶Neural Imprint state saved and restored on OpenVINO: 9B first token 0.97 → 0.45 s on the same content
- ▶Split predictor and CPU + iGPU pipeline; 357/357 sentences complete over 30 minutes of continuous synthesis
Layer-Streaming Offloading: Running 9B+ LLMs on 8GB Edge Devices
Per-layer weight streaming from NVMe storage enables models exceeding device memory to run inference on iPad and iPhone. 88% peak memory reduction with verified bandwidth scaling across Apple Silicon devices.
- ▶60–88% memory reduction: 27B model runs with only 1.7 GB peak memory
- ▶9B-6bit OOM on 8GB iPad proves streaming is necessary for 9B+ models
- ▶iPad/iPhone TPS ratio = 1.92x perfectly matches 2x bandwidth ratio
Does Speculative Decoding Help Mixture-of-Experts?
306-run empirical study showing that speculative decoding provides 1.18–1.30× speedup on Qwen3.5-35B-A3B MoE despite <4% draft acceptance, through a batch verification amortization mechanism that reduces memory bandwidth cost.
- ▶1.30× MoE speedup with 0.8B draft at γ=16, <0.2% acceptance
- ▶Speedup scales with total params (memory bandwidth), not active params
- ▶Batch verification amortization: new SD mechanism beyond acceptance rate
Efficient On-Device LLM Inference on Apple Silicon: From Quantization to Speculative Decoding
Systematic benchmarking of 7 GGUF quantization levels and speculative decoding for Qwen3.5 on three Apple Silicon machines (M2 Ultra, M1 Max, M2 Pro), establishing Q6_K as Pareto-optimal and a ≥2.5× draft/target speed ratio as the SD viability rule.
- ▶Q6_K Pareto-optimal: 1.68× faster, 59% smaller, 0.54% PPL loss
- ▶+25.7% throughput via speculative decoding (0.8B→9B, k=4)
- ▶GGML_RPC cross-device SD: 79% overhead — not production-viable
Cross-Domain Personal Data Integration on Consumer Hardware
Integrating finance, diet, mood, and reading data entirely on consumer Apple Silicon, producing emergent cross-domain insights with zero data leakage.
- ▶1.48x cross-domain insight emergence (IIR)
- ▶125.5x federation compression, zero data leakage
- ▶49.9 TPS real-time inference (35B on M2 Ultra)
ANE Batch Prefill for On-Device Parallel LLM Inference
Enabling concurrent ANE prefill and GPU decode on Apple Silicon via fused batch matrix-vector kernels, achieving 11.3x speedup over sequential dispatch.
- ▶11.3x batch dispatch speedup (268 tok/s)
- ▶79% power reduction with concurrent pipeline
- ▶27ms TTFT on multi-turn conversations
Disaggregated LLM Inference on Apple Silicon
Benchmarking CoreML ANE prefill + MLX GPU decode for Qwen3.5 on Apple Silicon, with four inference strategies compared.
- ▶ANE prefill matches GPU at ~410 tokens
- ▶282x GPU power reduction during prefill
- ▶4 inference pipelines benchmarked
On-Device Text-to-Speech
Native Swift implementation of Qwen3 TTS 0.6B for real-time, on-device speech synthesis.
- ▶67% model compression (2.35 GB → 808 MB)
- ▶Real-time synthesis (RTF 0.68x)
- ▶12 languages supported
On-Device Vision Language Model
Multi-stage compression pipeline for deploying Gemma 3 4B VLM on consumer hardware.
- ▶25% model compression (2.8 GB → 2.1 GB)
- ▶110 tok/s text generation
- ▶3.4x image processing speedup
MLX Memory Optimization Research
Exploring memory optimization techniques for the MLX framework on Apple Silicon.
- ▶Up to 20x faster mmap loading
- ▶Zero-copy model loading
- ▶Comprehensive benchmarks