Why Self-Host?

Every API call to a cloud LLM is a transaction. You're sending data to someone else's server, paying per token, and trusting that your prompts aren't being logged, trained on, or subpoenaed. For personal projects, that's probably fine. For business-critical applications handling sensitive data? It's a real concern.

LLocalAI offers an alternative: run open-source models on your own hardware, behind your own firewall, with zero external dependencies. Same OpenAI-compatible API format. Your infrastructure. Your rules.

This isn't a toy setup. LLocalAI supports text generation, embeddings, image generation, speech-to-text, and text-to-speech — all locally. Let's build it.

Hardware Reality Check

Before we start, let's be honest about requirements:

• Minimum viable: 16GB RAM, modern CPU, no GPU. You'll run quantized 7B models at around 5 tokens/sec. Usable for development.
• Comfortable: 32GB RAM, NVIDIA GPU with 8GB+ VRAM. Quantized 13B models at reasonable speed.
• Actually fast: 64GB RAM, NVIDIA GPU with 24GB+ VRAM (RTX 3090/4090 or A5000). 30B+ models at conversational speed.

No NVIDIA GPU? That's okay. LocalAI runs on CPU just fine — it's slower, but it works. Apple Silicon gets decent performance through Metal acceleration.

Installation

The fastest path is Docker:

bashdocker run -p 8080:8080 --name localai \
  -v $PWD/models:/build/models \
  localai/localai:latest-cpu

For NVIDIA GPU support, swap in the latest-gpu-nvidia-cuda-12 image and add --gpus all. That's it. LocalAI is now running on port 8080 with an OpenAI-compatible API.

Downloading and Configuring Models

LocalAI has a model gallery that makes this painless:

bash# List available models
curl http://localhost:8080/models/available
Install a model
curl http://localhost:8080/models/apply \
  -H "Content-Type: application/json" \
  -d '{"id": "huggingface://TheBloke/LLaMA-3-8B-GGUF/llama-3-8b.Q4_K_M.gguf"}'

The Q4_K_M quantization is the sweet spot for most setups — good quality, reasonable size. If you have the VRAM, Q5_K_M or Q6_K give noticeably better results. If you're memory-constrained, Q3_K_M still works but you'll notice quality drops on complex reasoning.

For custom model configuration, create a YAML file in your models directory:

yamlname: my-llama
backend: llama-cpp
parameters:
  model: llama-3-8b.Q4_K_M.gguf
  temperature: 0.7
  top_p: 0.9
context_size: 4096
threads: 8
gpu_layers: 35

Using the API

Because LocalAI is OpenAI-compatible, your existing code probably works with a one-line change:

pythonfrom openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)
response = client.chat.completions.create(
    model="my-llama",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing simply."}
    ]
)
print(response.choices[0].message.content)

This means any tool built against the OpenAI API — including many tools in our catalog — can potentially point at your local instance instead.

Embeddings: The Quiet Killer Feature

Text generation gets the headlines, but embeddings are where self-hosting really shines. Every time you send text to an embedding API, you're sending your data to someone else's server. For RAG applications processing internal documents, that's a non-starter for many organizations.

Pair local embeddings with ttxtai for a fully local RAG pipeline — your documents never leave your network, and you're not paying per-token for embeddings.

Adding It to Your Stack

LocalAI plays well with other tools:

LocalAI + nn8n: nn8n can call LocalAI's API the same way it calls OpenAI. Set up automation workflows using local models for data processing, summarization, or classification — all without external API costs.

LocalAI + FFlowise: FFlowise supports custom API endpoints. Point it at your LocalAI instance and build chatbot flows that run entirely on your hardware.

LocalAI + ttxtai: ttxtai handles the retrieval pipeline while LocalAI handles generation. Fully local RAG.

Performance Tuning

Thread count: Set threads to your physical core count (not logical cores). Hyperthreading doesn't help here and can actually hurt.

GPU offloading: Incrementally increase gpu_layers until you hit VRAM limits. Each layer offloaded dramatically improves speed.

Context size: Larger windows use more memory. If you don't need 4096 tokens, reduce it. 2048 tokens uses roughly half the memory.

Quantization: Profile your workload. For code generation, higher quantization matters more than for conversation.

When to Self-Host (And When Not To)

Self-host when: data privacy is non-negotiable, you have predictable high-volume workloads, you need air-gapped environments, or you want full control over model configuration.

Use cloud APIs when: you need frontier-class capabilities, your usage is bursty, you don't want to manage infrastructure, or time-to-production matters most.

The honest take: self-hosted models are good and getting better fast, but they're not at frontier capability yet. Know what you're trading.

Check out LLocalAI in our catalog for setup links and community resources.