Loadcore
Documentation

Loadcore docs

Everything you need to run open models across your GPU fleet with the Console, API and Agent.

Overview

Loadcore is a distributed system for running open LLMs on your own GPU servers. You install a small agent on each server, and manage the whole fleet from a web console. Every running model is exposed through an OpenAI-compatible API, so existing tools work without changes.

Your prompts and completions never touch Loadcore's servers. Inference traffic goes from your tools straight to the agent on your server. The control plane only handles authentication and orchestration.

Architecture

Loadcore has three components:

  • Loadcore Console: the web app your team uses to manage servers, models, instances, people and keys.
  • Loadcore API: the multi-tenant control plane that stores fleet state and access rules and validates API keys.
  • Loadcore Agent: a single Go binary on each GPU server that installs llama.cpp, downloads models, runs instances and serves /v1.

Agents push heartbeats, download progress and instance status to the API. Tools send inference requests directly to agents.

Loadcore Console

The console at console.loadcore.io is where admins and members work day to day:

  • Dashboard with fleet health at a glance
  • Servers to register machines and check agent connectivity
  • Models to search HuggingFace and download GGUF files to a server
  • Instances to start, stop and connect to running models
  • Chat to talk to any model you have access to
  • Organization to manage members, departments, invites and access requests
  • Settings to create and revoke your personal API keys

Loadcore API

The API is the managed control plane at api.loadcore.io. It owns organizations, roles, departments, the server registry, model and instance state, and licenses. When an agent receives a request, it asks the API whether the key is valid and which models its owner can use, and caches the answer for 5 minutes.

The API never proxies inference and never stores prompts or completions.

Loadcore Agent

The agent is a single binary with no runtime dependencies. On first start it installs the latest llama.cpp build. It then:

  • sends a heartbeat every 30 seconds with OS, architecture, GPU, RAM and public IP
  • downloads models into ~/.loadcore/models and reports progress
  • starts and stops llama.cpp processes on request
  • health-checks running instances every 15 seconds
  • serves GET /v1/models, POST /v1/chat/completions and POST /v1/completions
./loadcore-agent \
  --api-key="lc_..."                       # server key from registration
  --backend-url="https://api.loadcore.io"  # control plane
  --port=9090                              # listen port (default 9090)
  --data-dir=~/.loadcore                   # models and binaries
  --llama-cpp-version=latest               # pin a llama.cpp build

Every flag can also be set with an environment variable: LOADCORE_API_KEY, LOADCORE_BACKEND_URL, LOADCORE_PORT, LOADCORE_DATA_DIR and LOADCORE_LLAMA_CPP_VERSION.

Setup

  1. Sign up in the console and create your organization.
  2. Go to Servers and register a GPU server. Copy the one-time server key.
  3. Copy the agent binary to the server and start it with that key.
  4. Wait for the server to show as online. llama.cpp installs automatically.
  5. Download a model to the server and start an instance.
  6. Create a personal API key in Settings and connect your tools.

Servers

Each GPU server runs one agent. Register it from the Servers page, then start the agent with the server key. The server appears with the system info from its first heartbeat. Use check connection to confirm the agent is reachable from the network your tools use.

Cloud GPU servers

Any Linux GPU server you rent by the hour works the same way. When registering, open the Cloud server tab and copy the startup script. Paste it into the startup script field when you create the server at your provider, often called user data or cloud-init. The server installs the agent as a system service on first boot and appears in the console without any manual steps.

  • Choose an image with NVIDIA drivers installed so llama.cpp uses the GPU.
  • The agent stores models and binaries in /opt/loadcore.
  • For distributed instances, create all servers in the same region and private network.
  • The script contains the server key. Rotate the key if people outside your team can read the server settings at your provider.

Models

  • Search HuggingFace for GGUF models from the Models page.
  • Choose the target server and the quantization that fits its VRAM.
  • Follow download progress live. Files land in the agent's data directory.
  • When deleting, remove the entry only or also delete the file from the server.

See the model guides for VRAM needs and recommended sampling settings per model.

Instances

An instance is a running llama.cpp server for one model on one server. Pick the server, model, port, context size and GPU layers, then start it. The agent spawns the process and reports its status. Run the same model on several servers to add throughput. Each running instance has a connect button with ready-made config for common tools.

Distributed instances

When a model does not fit in the GPU memory of one server, a distributed instance can split it across several servers using llama.cpp RPC. Start the instance on the main server and tick Distribute across servers to add rpc servers. The console shows the combined GPU memory against the model size.

  • Only the main server needs the model file.
  • A distributed instance has one main server and up to 7 rpc servers, 8 servers in total.
  • Servers must be on the same fast local network. Each rpc server only accepts connections from the main server.
  • It helps you fit bigger models, not run them faster. To spread load, run separate instances and choose between them.

Organizations

Everything belongs to an organization: members, servers, models, instances and departments.

  • Admins manage members, departments, invites, servers and models, and can use every model.
  • Members see the fleet and use the models their departments are granted.
  • Invite people by email, or approve access requests from people who asked to join.

Departments

Departments decide who can use which model. Create departments, add members, and grant access per model. A member in several departments gets the union of their grants.

Example: Engineering is granted qwen3-coder-next, Research is granted deepseek-v4.1-flash. Someone in both can use both.

API keys

  • Create personal keys in Settings. They start with sk-lc- and are shown once.
  • A key carries its owner's model permissions.
  • The agent validates keys with the API and caches the result for 5 minutes.
  • Revoke a key at any time. Revocation takes effect once the agent cache expires.

Connecting tools

Point any OpenAI-compatible tool at http://your-server:9090/v1 with your personal key.

curl

curl http://your-server:9090/v1/chat/completions \
  -H "Authorization: Bearer sk-lc-your-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model.gguf", "messages": [{"role": "user", "content": "Hello"}]}'

Python

from openai import OpenAI

client = OpenAI(base_url="http://your-server:9090/v1", api_key="sk-lc-your-key")
client.chat.completions.create(
    model="your-model.gguf",
    messages=[{"role": "user", "content": "Hello"}],
)

opencode

{
  "provider": {
    "loadcore": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Loadcore",
      "options": {
        "baseURL": "http://your-server:9090/v1",
        "apiKey": "{env:LOADCORE_API_KEY}"
      },
      "models": { "your-model.gguf": { "name": "Your model" } }
    }
  },
  "model": "loadcore/your-model.gguf"
}

aider

aider \
  --model openai/your-model.gguf \
  --openai-api-key sk-lc-your-key \
  --openai-api-base http://your-server:9090/v1