Local LLM

AI Core can use a native local LLM provider when the runtime artifact includes that capability. SDK users do not need the model-llama-cpp source crate; they need a compatible AI Core artifact and an explicit local .gguf model path.

Requirements

Use an AI Core runtime artifact that includes native local LLM support for your target platform. Native acceleration is a provider build detail rather than a separate model-selection mode. Hosts can pass an optional backend preference: auto, cpu, or metal. If omitted, the provider uses auto. The provider reports available backends through debug logging; host/UI code does not set build-time availability constants. If a preferred backend cannot initialize at runtime, the provider falls back to automatic backend selection without retrying the failed preferred backend.

The provider expects a .gguf model file. AI Core does not scan model directories or choose a model automatically for the host. Store the selected model path in host-owned settings and pass it in config or process environment.

Runtime Config

The host-owned config snapshot and model-selection fields are documented in Runtime Contracts. That page shows the supported models entries, model_selection rules, and the LocalModelConfig shape that host code passes into AI Core.

Do not rely on implicit model discovery. The host should resolve the .gguf path itself and send the selected file path to AI Core through runtime config or the process boundary.

Use CLI model debug logging to verify the backend selected at runtime. The provider rejects prompts whose tokenized input already fills the configured context window; the practical prompt limit is context_tokens - 1, and generation stops before exceeding context_tokens.

Backend preference can be global or per model. local_llm.backend applies to all local LLM models. models[].local.backend overrides it for one .gguf model, which lets a host run a small routing model on GPU while starting a larger default model on CPU.

Concurrency is configured per built_in_local model:

{
  "local": {
    "path": "/models/llm/qwen3-1.7b.gguf",
    "context_tokens": 4096,
    "backend": "auto",
    "max_concurrent_requests": 1,
    "max_pending_requests": 0
  }
}

max_concurrent_requests defaults to 1 and must be greater than zero. max_pending_requests defaults to 0, which means a request is rejected immediately when all active native slots for that model are busy. Rust hosts can use LOCAL_MODEL_UNBOUNDED_CONCURRENT_REQUESTS, and JSON hosts can pass 4294967295, for an effectively unbounded active limit. That is not a memory-saving setting; use it only when the host provides external backpressure.

The native provider keeps one loaded llama_model resident per process/model/backend key after first use. Multiple FFI handles inside the same process share that resident model. Separate OS processes do not share it, so a pool of four worker processes can still load four copies of the same GGUF.

Only the model weights are shared. Each active request creates its own llama_context, KV cache, sampler, and prompt/generation buffers. Increasing max_concurrent_requests increases peak context/KV memory even though the GGUF weights are not reloaded per request in the same process.

Keep durable queues in the host. max_pending_requests is only a small in-process admission queue and should not replace a service queue, database job table, retry policy, or UI backpressure.

Packaging Guidance

For desktop apps, ship model files in an app-managed resource/cache location and store the selected path in host settings. For services, mount models into a stable path such as /opt/ai-core/models/llm.

Do not rely on current working directory, source-tree paths, or implicit model discovery. Treat the .gguf path as explicit host configuration.