Research
Our research covers Neural Imprint, model loading, quantization, speech, and multimodal inference.
Neural Imprint: Models That Grow With Their Users — On-Device AI Evolution Without Weight Updates
Neural Imprint consolidates local records into inference state that can be saved and restored, carrying learning into later sessions while base weights remain frozen.
- ▶Validated on 8 application domains, 2 model sizes (Qwen3.5-4B / 9B) and 2 iPhones, using synthetic behavioral data (3,283 records)
- ▶16/16 A-library health gates pass (8 domains × 2 model sizes)
- ▶16/16 personalized profile responses complete with zero tool calls; 16/16 autonomous tool invocations correct
- ▶Entire pipeline runs within the 6 GB memory limit of consumer iPhones, zero cloud dependency
- ▶Manuscript, 2026 — full text not yet public
On-Device Speech and 9B Dialogue on Intel: A Streaming Engine Carried Over From Apple
The streaming speech engine we built for Apple silicon, carried to Intel Core Ultra (CPU + Arc iGPU) on our own OpenVINO-based inference runtime. Speech recognition, a 9B language model and speech synthesis stay resident on one machine; delivered twice to a PC manufacturer.
- ▶Stateful streaming vocoder: 0.4 s blocks, 115.9–116.4 dB SNR against whole-sentence output
- ▶First sound in 0.37 s, speech synthesis RTF 0.73; the open-source reference takes 15–20 s at 2.1–2.9
- ▶First successful validation of the NI algorithm and architecture on a non-Apple platform
- ▶Split predictor and CPU + iGPU pipeline; 357/357 sentences complete over 30 minutes of continuous synthesis
Layer-Streaming Offloading: Running 9B+ LLMs on 8GB Edge Devices
Per-layer weight streaming from NVMe storage enables models exceeding device memory to run inference on iPad and iPhone. 88% peak memory reduction with verified bandwidth scaling across Apple Silicon devices.
- ▶60–88% memory reduction: 27B model runs with only 1.7 GB peak memory
- ▶The tested 9B 6-bit configuration ran out of memory on an 8GB iPad; streaming can reduce memory requirements for such configurations
- ▶The measured throughput ratio was 1.92×, close to the devices’ 2× nominal bandwidth ratio
Does Speculative Decoding Help Mixture-of-Experts?
A study of 306 experiments compares speculative-decoding configurations for Qwen3.5-35B-A3B MoE. Tested configurations showed 1.18–1.30× speedups with draft acceptance below 4%; amortized batch verification is a hypothesis for this result.
- ▶1.30× MoE speedup with 0.8B draft at γ=16, <0.2% acceptance
- ▶In these experiments, speedups were associated with total parameter count and memory-bandwidth costs
- ▶One possible explanation: batched verification amortizes the bandwidth cost of reading weights
Efficient On-Device LLM Inference on Apple Silicon: From Quantization to Speculative Decoding
Benchmarks compare seven GGUF quantization levels and speculative decoding for Qwen3.5 on M2 Ultra, M1 Max, and M2 Pro. Q6_K lies on the quality, size, and speed trade-off frontier within the tested configurations; a draft-to-target speed ratio around 2.5× is a reference point from this experiment.
- ▶Tested Q6_K configuration: 1.68× speed, 59% smaller size, 0.54% PPL loss
- ▶+25.7% throughput via speculative decoding (0.8B→9B, k=4)
- ▶GGML_RPC cross-device SD: 79% overhead — not production-viable
Cross-Domain Personal Data Integration on Consumer Hardware
Our study compares output scores for single-domain and four-domain inputs across finance, diet, mood, and reading, and tests summary transfer between devices.
- ▶Output-score comparison for one life domain and a combination of four domains
- ▶The test reports 125.5× summary compression; summaries may still contain private information
- ▶49.9 TPS real-time inference (35B on M2 Ultra)
ANE Batch Prefill for On-Device Parallel LLM Inference
Fused batched matrix-vector kernels run ANE prefill alongside GPU decode on Apple Silicon. Batch-dispatch throughput was 11.3× the baseline in the tested configuration; this is a stage-specific measurement.
- ▶11.3x batch dispatch speedup (268 tok/s)
- ▶79% lower power in the concurrent-pipeline test
- ▶27ms time to first token in the multi-turn test
Disaggregated LLM Inference on Apple Silicon
Benchmarking CoreML ANE prefill + MLX GPU decode for Qwen3.5 on Apple Silicon, with four inference strategies compared.
- ▶ANE prefill matches GPU at ~410 tokens
- ▶282× lower GPU power during the prefill test
- ▶4 inference pipelines benchmarked
On-Device Text-to-Speech
Native Swift implementation of Qwen3 TTS 0.6B for real-time, on-device speech synthesis.
- ▶67% model compression (2.35 GB → 808 MB)
- ▶Real-time synthesis (RTF 0.68x)
- ▶12 languages supported
On-Device Vision Language Model
Multi-stage compression pipeline for deploying Gemma 3 4B VLM on consumer hardware.
- ▶25% model compression (2.8 GB → 2.1 GB)
- ▶110 tok/s text generation
- ▶3.4x image processing speedup
MLX Memory Optimization Research
Exploring memory optimization techniques for the MLX framework on Apple Silicon.
- ▶Up to 20× faster file mapping in mmap tests
- ▶Zero-copy model loading
- ▶Comprehensive benchmarks