feat: add vLLM/local LLM support

- Add vllm provider configuration in config schema
- Auto-detect vLLM endpoints and use hosted_vllm/ prefix for LiteLLM
- Pass api_base directly to acompletion for custom endpoints
- Add vLLM status display in CLI status command
- Add vLLM setup documentation in README
This commit is contained in:
ZhihaoZhang97
2026-02-02 11:23:04 +11:00
parent 959c4dadf8
commit 2b19dcf9fd
4 changed files with 61 additions and 2 deletions

View File

@@ -116,6 +116,43 @@ nanobot agent -m "What is 2+2?"
That's it! You have a working AI assistant in 2 minutes.
## 🖥️ Local Models (vLLM)
Run nanobot with your own local models using vLLM or any OpenAI-compatible server.
**1. Start your vLLM server**
```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
```
**2. Configure** (`~/.nanobot/config.json`)
```json
{
"providers": {
"vllm": {
"apiKey": "dummy",
"apiBase": "http://localhost:8000/v1"
}
},
"agents": {
"defaults": {
"model": "meta-llama/Llama-3.1-8B-Instruct"
}
}
}
```
**3. Chat**
```bash
nanobot agent -m "Hello from my local LLM!"
```
> [!TIP]
> The `apiKey` can be any non-empty string for local servers that don't require authentication.
## 💬 Chat Apps
Talk to your nanobot through Telegram or WhatsApp — anytime, anywhere.