Hermes Agent, Qwen3.6 27B, llama.cpp, Docker, Telegram, systemd, and a surprisingly educational firewall problem
Running a large language model locally is no longer particularly unusual. Running one as a reliable, always-available agent on a production workstation is a different challenge entirely.
Our goal was not simply to launch a model and open a chat window. We wanted a system that could:
- run a capable 27-billion-parameter model locally,
- use an NVIDIA RTX 3080 without making the desktop unusable,
- expose an OpenAI-compatible API,
- connect to Hermes Agent,
- receive requests through Telegram,
- execute tools inside an isolated Docker container,
- access the internet from that container,
- survive logouts and reboots,
- recover from service failures,
- and update itself without silently breaking the working system.
The final result is a local agent built around:
Arch Linux
Intel Core i9, 13th generation
64 GB system memory
NVIDIA GeForce RTX 3080, 10 GB VRAM
llama.cpp
Qwen3.6 27B Q4 GGUF
Hermes Agent
Docker
Telegram
systemd user services
firewalld and nftables
This article documents the complete journey, including the false starts, performance measurements, networking issue, security decisions, and maintenance framework.
1. The target architecture
The final architecture looks like this:
Telegram
│
▼
hermes-gateway.service
│
▼
Hermes Agent
┌────┴────┐
│ │
│ └──────────────┐
▼ ▼
Local OpenAI-compatible API Docker sandbox
http://127.0.0.1:8080/v1 hermes-net
│ │
▼ ▼
llama-qwen.service Tool execution
│ /workspace
▼ │
Qwen3.6 27B Q4 GGUF ▼
RTX 3080 + CPU offload Internet via br0
There are two important security boundaries:
- The model server only listens on
127.0.0.1. - Tool execution happens inside Docker, not directly on the workstation.
Hermes can read and write only an explicitly mounted workspace:
/home/johannes/HermesWorkspace
It does not receive access to the complete home directory, SSH keys, GPG keys, cloud credentials, or the Docker socket.
2. Hardware reality: a 27B model on a 10 GB GPU
The model we selected was:
Qwen3.6-27B-UD-Q4_K_XL.gguf
Its GGUF file is approximately 18 GB. That immediately means it cannot fit completely into the 10 GB VRAM of an RTX 3080.
The workstation nevertheless had enough total memory:
System RAM: 64 GB
Available RAM during setup: approximately 51 GB
GPU VRAM: 10 GB
Desktop VRAM usage: approximately 2.4 GB
Free disk space: approximately 270 GB
The solution was partial GPU offloading.
llama.cpp places as many layers as possible on the RTX 3080 and keeps the remaining weights in system memory. The CPU therefore remains an important part of inference.
This setup is not fast in the same way as a fully GPU-resident 27B model, but it is usable.
Our measured generation speed was roughly:
2.1 to 2.3 tokens per second
Prompt processing was faster, especially after increasing batch-processing threads.
The important lesson was that a model fitting into total RAM does not imply that it will run efficiently. VRAM determines how much of the model can remain on the GPU, while memory bandwidth and CPU architecture determine the performance of the offloaded portion.
3. Installing llama.cpp on Arch Linux
The required packages were installed through Pacman:
sudo pacman -Syu
sudo pacman -S --needed \
llama-cpp \
cuda \
curl \
jq
The NVIDIA runtime was already functional:
nvidia-smi
llama.cpp device detection was verified with:
llama-server --list-devices
The important part was that the RTX 3080 appeared as a CUDA device.
The model was stored under:
mkdir -p "$HOME/.local/share/llama-models"
For example:
MODEL_DIR="$HOME/.local/share/llama-models"
MODEL_FILE="$MODEL_DIR/Qwen3.6-27B-UD-Q4_K_XL.gguf"
curl \
--location \
--fail \
--continue-at - \
--output "$MODEL_FILE" \
"https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-UD-Q4_K_XL.gguf"
4. The first local model server
The first successful manual launch looked conceptually like this:
llama-server \
--model "$HOME/.local/share/llama-models/Qwen3.6-27B-UD-Q4_K_XL.gguf" \
--alias "qwen36-27b-local" \
--host 127.0.0.1 \
--port 8080 \
--ctx-size 65536 \
--parallel 1 \
--jinja \
--flash-attn auto \
--threads 6 \
--threads-batch 6 \
--n-gpu-layers auto \
--fit on \
--fit-target 2048 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--sleep-idle-seconds 300 \
--no-mmproj
Several options were especially important.
Localhost-only binding
--host 127.0.0.1
The API is not exposed to the local network.
A single inference slot
--parallel 1
The available context is not divided between several concurrent slots.
Tool-calling template support
--jinja
Hermes relies on correct chat-template and tool-call formatting.
Automatic GPU fitting
--n-gpu-layers auto
--fit on
--fit-target 2048
llama.cpp determines how many layers fit on the GPU while leaving approximately 2 GB of safety margin.
That reserve mattered because the RTX 3080 also drove the desktop session.
Reduced KV-cache memory
--cache-type-k q8_0
--cache-type-v q8_0
The quantized KV cache reduced memory pressure compared with a full-precision cache.
Releasing resources while idle
--sleep-idle-seconds 300
After five minutes without inference, llama.cpp releases model resources. The next actual request wakes it again.
This made the setup much more suitable for a workstation that was also being used for normal development work.
5. Verifying the OpenAI-compatible API
The health endpoint was tested first:
curl --fail --silent \
http://127.0.0.1:8080/health |
jq
Expected output:
{
"status": "ok"
}
The first completion test revealed an interesting issue:
curl --fail --silent \
http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen36-27b-local",
"messages": [
{
"role": "user",
"content": "Reply only with: Model works."
}
],
"max_tokens": 32
}' |
jq
The response contained an empty content field but a populated reasoning_content field.
The model had consumed the complete 32-token output allowance while reasoning. The generation ended with:
"finish_reason": "length"
For deterministic functional tests, thinking was disabled through the chat template:
curl --fail --silent \
http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen36-27b-local",
"messages": [
{
"role": "user",
"content": "Reply only with: Model works."
}
],
"chat_template_kwargs": {
"enable_thinking": false
},
"max_tokens": 64,
"temperature": 0
}' |
jq
This produced the expected direct response.
6. Turning llama.cpp into a user service
The model server needed to start automatically and remain available to Hermes.
We created:
~/.config/systemd/user/llama-qwen.service
A representative final unit looked like this:
[Unit]
Description=Qwen3.6 27B llama.cpp server
After=graphical-session.target network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/llama-server \
--model %h/.local/share/llama-models/Qwen3.6-27B-UD-Q4_K_XL.gguf \
--alias qwen36-27b-local \
--host 127.0.0.1 \
--port 8080 \
--ctx-size 65536 \
--parallel 1 \
--jinja \
--flash-attn auto \
--threads 6 \
--threads-batch 16 \
--n-gpu-layers auto \
--fit on \
--fit-target 2048 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--sleep-idle-seconds 300 \
--no-mmproj
Restart=on-failure
RestartSec=10
Nice=10
CPUQuota=1600%
MemoryHigh=40G
MemoryMax=48G
OOMPolicy=stop
[Install]
WantedBy=default.target
The service was enabled with:
systemctl --user daemon-reload
systemctl --user enable --now llama-qwen.service
Status and logs:
systemctl --user status llama-qwen.service
journalctl --user \
-u llama-qwen.service \
-f
Because systemd user lingering was enabled later for the Telegram gateway, the user services could remain active independently of an interactive login.
7. CPU scaling: more threads were not automatically faster
Initially, the server used:
--threads 6
--threads-batch 6
CPUQuota=600%
Only a small number of CPU cores were visibly active, so we increased the limits to:
--threads 12
--threads-batch 16
CPUQuota=1600%
Then we ran three identical benchmarks:
for i in 1 2 3; do
curl --silent \
http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen36-27b-local",
"messages": [{
"role": "user",
"content": "Explain in approximately 200 words how CPU caches work."
}],
"chat_template_kwargs": {
"enable_thinking": false
},
"max_tokens": 300,
"temperature": 0
}' |
jq '{
prompt_tps: .timings.prompt_per_second,
generation_tps: .timings.predicted_per_second,
generated_tokens: .timings.predicted_n
}'
done
The results were approximately:
{
"prompt_tps": 15.56,
"generation_tps": 2.18
}
{
"prompt_tps": 6.51,
"generation_tps": 2.16
}
{
"prompt_tps": 6.61,
"generation_tps": 2.15
}
The original six-thread generation had been approximately:
2.25 tokens per second
Increasing generation threads did not help. It slightly reduced performance.
That led to a split configuration:
--threads 6
--threads-batch 16
The reasoning was:
- Token generation remained memory-bandwidth-bound.
- Additional generation threads added synchronization overhead.
- Larger Hermes prompts could still benefit from more batch-processing threads.
This was one of the most useful lessons of the project:
CPU utilization is not itself a performance metric. Tokens per second are.
8. Installing Hermes Agent
Hermes was installed through its installer:
curl -fsSL \
https://hermes-agent.nousresearch.com/install.sh \
-o /tmp/hermes-install.sh
less /tmp/hermes-install.sh
bash /tmp/hermes-install.sh
The first setup created a Nous Portal account and selected a hosted free model:
stepfun/step-3.7-flash:free
This caused our first major configuration mistake.
Telegram requests worked, but no inference requests appeared in the local llama.cpp logs.
The reason was simple: Hermes was still configured with:
provider: nous
The requests were going to a hosted model.
We changed the model provider to a custom OpenAI-compatible endpoint:
Base URL: http://127.0.0.1:8080/v1
Model: qwen36-27b-local
Context length: 65536
A simplified Hermes model configuration looked like:
model:
provider: custom
default: qwen36-27b-local
base_url: http://127.0.0.1:8080/v1
context_length: 65536
After restarting the gateway and resetting the Telegram session, /status showed:
Model: qwen36-27b-local
Provider: custom
Endpoint: http://127.0.0.1:8080/v1
At that point, Telegram messages finally appeared in the local llama.cpp logs.
9. Telegram as the remote interface
Hermes created and configured a Telegram bot during setup.
The gateway was installed as a user service:
~/.config/systemd/user/hermes-gateway.service
It was enabled and started with:
systemctl --user enable --now hermes-gateway.service
The gateway could be inspected with:
hermes gateway status
journalctl --user \
-u hermes-gateway.service \
-f
A Telegram allowlist restricted access to the owner’s Telegram user ID.
This was essential. A tool-capable agent exposed through a messaging platform must not be usable by arbitrary Telegram accounts.
The final proof that local inference was working was a Telegram request such as:
Reply exactly with: Local model works.
The response came back through Telegram, and a corresponding inference request appeared in the llama.cpp service log.
10. Why the local terminal backend was unacceptable
The Hermes installer initially retained:
terminal backend: local
That meant shell commands could run directly on the production workstation with the same permissions as the current user.
On a development machine, that potentially exposes:
~/.ssh
~/.gnupg
~/.config
cloud credentials
Git credentials
source repositories
password databases
browser profiles
We changed the backend to Docker.
The workspace was created with:
mkdir -p "$HOME/HermesWorkspace"
chmod 700 "$HOME/HermesWorkspace"
The relevant Hermes terminal configuration became:
terminal:
backend: docker
docker_image: "nikolaik/python-nodejs:python3.11-nodejs20"
docker_mount_cwd_to_workspace: false
docker_run_as_host_user: true
docker_volumes:
- "/home/johannes/HermesWorkspace:/workspace"
docker_network: true
container_cpu: 16
container_memory: 16384
container_persistent: true
docker_persist_across_processes: true
timeout: 300
The persistent containers were later verified with:
for CID in $(docker ps -q --filter label=hermes-agent=1); do
docker inspect "$CID" \
--format 'Name={{.Name}} CPUs={{.HostConfig.NanoCpus}} Memory={{.HostConfig.Memory}} Network={{.HostConfig.NetworkMode}}'
done
The output confirmed:
CPUs=16000000000
Memory=17179869184
That means:
16 CPU cores
16 GiB memory
11. Giving the Docker sandbox internet access
The workstation used a Linux bridge named br0 as its active external interface:
enp6s0 → br0 → 192.168.178.1 → internet
The host routing table showed:
default via 192.168.178.1 dev br0
The DNS server was the local FritzBox:
192.168.178.1
A dedicated Docker network was created:
docker network create hermes-net
Hermes was configured with:
terminal:
docker_network: true
docker_extra_args:
- "--network=hermes-net"
- "--dns=192.168.178.1"
- "--dns-search=fritz.box"
A direct Docker test initially failed:
docker run --rm \
--network hermes-net \
--dns 192.168.178.1 \
--dns-search fritz.box \
nikolaik/python-nodejs:python3.11-nodejs20 \
sh -lc '
getent ahostsv4 example.com
curl --connect-timeout 5 --max-time 15 \
-fsSI https://example.com |
head -n 1
'
The result was:
curl: (28) Resolving timed out
At first, this looked like a DNS problem.
It was not.
12. The real networking problem: an nftables forward chain
Testing with host networking succeeded:
docker run --rm \
--network host \
--dns 192.168.178.1 \
nikolaik/python-nodejs:python3.11-nodejs20 \
getent ahostsv4 example.com
Testing direct TCP access inside the Docker bridge failed:
FritzBox DNS/TCP: timed out
FritzBox DNS/UDP: timed out
Direct Internet/TCP: timed out
The failure was therefore not limited to DNS. All forwarded traffic from the Docker bridge was being dropped.
Docker itself looked correct:
net.ipv4.ip_forward = 1
Masquerading rules existed:
172.19.0.0/16 → MASQUERADE
The critical discovery was this nftables configuration:
table inet filter {
chain input {
type filter hook input priority filter
policy drop
ct state invalid drop
ct state { established, related } accept
iif lo accept
meta l4proto { icmp, ipv6-icmp } accept
tcp dport 22 accept
}
chain forward {
type filter hook forward priority filter
policy drop
}
}
The forward chain was empty and had a default drop policy.
It executed before Firewalld’s forwarding chains and therefore dropped all forwarded packets before Docker and Firewalld could accept them.
Because Firewalld was already responsible for zone-based forwarding, the empty custom forward chain was removed from /etc/nftables.conf.
The remaining firewall kept the input policy:
table inet filter {
chain input {
type filter hook input priority filter
policy drop
ct state invalid drop
ct state { established, related } accept
iif lo accept
meta l4proto { icmp, ipv6-icmp } accept
tcp dport 22 accept
}
}
The syntax was checked with:
sudo nft -c -f /etc/nftables.conf
The configuration was applied:
sudo nft -f /etc/nftables.conf
The Docker test immediately succeeded:
=== DNS ===
172.66.147.243
104.20.23.154
=== HTTPS ===
HTTP/2 200
This was the most important troubleshooting lesson of the project:
An
acceptin one nftables base chain does not necessarily protect a packet from a later chain, while adropis final.
Multiple firewall managers can create technically valid but operationally conflicting rule sets.
13. Final Docker verification
The actual Hermes containers were inspected:
CID="$(
docker ps \
--filter label=hermes-agent=1 \
--quiet |
head -n 1
)"
docker inspect "$CID" |
jq '.[0] | {
name: .Name,
network_mode: .HostConfig.NetworkMode,
dns: .HostConfig.Dns,
dns_search: .HostConfig.DnsSearch,
cpus: (.HostConfig.NanoCpus / 1000000000),
memory_gib: (.HostConfig.Memory / 1073741824),
mounts: [.Mounts[] | {
source: .Source,
destination: .Destination,
writable: .RW
}]
}'
The expected values were:
{
"network_mode": "hermes-net",
"dns": [
"192.168.178.1"
],
"dns_search": [
"fritz.box"
],
"cpus": 16,
"memory_gib": 16
}
The mounts included:
/home/johannes/HermesWorkspace → /workspace
and Hermes-managed internal directories.
Importantly, the following were not mounted:
/home/johannes
/home/johannes/.ssh
/home/johannes/.gnupg
/var/run/docker.sock
A Telegram tool request finally returned:
Hermes network works
At that point, local inference, Telegram, Docker tool execution, DNS, HTTPS, and workspace access were all working together.
14. Building a maintenance framework
A working local agent is useful. A working local agent that silently stops after an update is not.
We therefore added a maintenance framework built around systemd user services and timers.
Its responsibilities were divided into four categories.
Lightweight health check
Executed every ten minutes:
- verify
llama-qwen.service, - verify
hermes-gateway.service, - verify the llama.cpp health endpoint,
- restart failed services,
- confirm that
hermes-netexists, - notify through Telegram when recovery fails or succeeds.
A simplified health-check script:
#!/usr/bin/env bash
set -Eeuo pipefail
LLAMA_SERVICE="llama-qwen.service"
GATEWAY_SERVICE="hermes-gateway.service"
LLAMA_URL="http://127.0.0.1:8080"
DOCKER_NETWORK="hermes-net"
wait_for_llama() {
local attempts="${1:-90}"
for ((i=1; i<=attempts; i++)); do
if curl --fail --silent --max-time 5 \
"$LLAMA_URL/health" |
jq -e '.status == "ok"' >/dev/null 2>&1; then
return 0
fi
sleep 2
done
return 1
}
if ! systemctl --user is-active --quiet "$LLAMA_SERVICE"; then
systemctl --user restart "$LLAMA_SERVICE"
fi
if ! wait_for_llama 90; then
systemctl --user restart "$LLAMA_SERVICE"
wait_for_llama 120
fi
if ! systemctl --user is-active --quiet "$GATEWAY_SERVICE"; then
systemctl --user restart "$GATEWAY_SERVICE"
fi
docker network inspect "$DOCKER_NETWORK" >/dev/null
The timer:
[Unit]
Description=Run Hermes stack health check regularly
[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
AccuracySec=30s
Unit=hermes-stack-health.service
[Install]
WantedBy=timers.target
Deep canary test
The weekly canary performs a real end-to-end check:
- local chat completion,
- Hermes configuration validation,
- gateway status,
- Docker DNS,
- Docker HTTPS access.
A simplified completion test:
curl --fail --silent --show-error \
--max-time 600 \
http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen36-27b-local",
"messages": [{
"role": "user",
"content": "Reply only with OK."
}],
"chat_template_kwargs": {
"enable_thinking": false
},
"max_tokens": 16,
"temperature": 0
}'
The Docker canary:
docker run --rm --pull=never \
--network hermes-net \
--dns 192.168.178.1 \
--dns-search fritz.box \
nikolaik/python-nodejs:python3.11-nodejs20 \
sh -lc '
getent hosts example.com >/dev/null &&
curl --connect-timeout 5 \
--max-time 15 \
-fsS https://example.com >/dev/null
'
Update detection
The update checker reports:
- pending Arch Linux packages,
- a newer Hermes Git commit,
- a newer Docker image digest,
- local modifications that would make an automated update unsafe.
Arch packages are checked, but not installed automatically.
That boundary was intentional.
An unattended Arch Linux upgrade could simultaneously replace:
Linux kernel
NVIDIA driver
CUDA libraries
Docker
llama.cpp
systemd
firewalld
Automating all of that on a production workstation would prioritize freshness over availability.
Hermes update with backup and rollback
Hermes itself was suitable for controlled automatic updating.
The update script:
- checks for local repository modifications,
- creates a compressed backup,
- stops the Telegram gateway,
- runs
hermes update, - validates or migrates the configuration,
- restarts the gateway,
- checks the services,
- automatically restores the backup if any step fails.
A special case appeared immediately.
Hermes created an untracked file:
.install_method
The updater initially treated it as an unsafe local modification and aborted.
The filter was corrected to ignore exactly that known marker:
dirty="$(
git -C "$HERMES_REPO" \
status --porcelain --untracked-files=all |
grep -vxF '?? .install_method' || true
)"
All other modifications still block automatic updates.
After this correction, the update completed successfully:
Code updated
Web UI built
Configuration is up to date
Skills are up to date
Update complete
Hermes successfully updated
15. Scheduling maintenance around real working hours
The workstation is normally used Monday through Friday, approximately from 09:00 to 17:00.
A Sunday update timer was therefore a poor fit. With persistent timers, a missed Sunday update could be executed on Monday morning, exactly when the workstation was needed.
The schedule was adjusted.
Hermes update
[Unit]
Description=Weekly Hermes Agent update during working hours
[Timer]
OnCalendar=Wed *-*-* 16:15:00
Persistent=false
RandomizedDelaySec=5min
AccuracySec=1min
Unit=hermes-stack-hermes-update.service
[Install]
WantedBy=timers.target
Deep check
[Unit]
Description=Weekly Hermes stack canary after update
[Timer]
OnCalendar=Thu *-*-* 09:15:00
Persistent=false
RandomizedDelaySec=5min
AccuracySec=1min
Unit=hermes-stack-deep-check.service
[Install]
WantedBy=timers.target
This provides a practical maintenance rhythm:
Wednesday afternoon:
update Hermes
Thursday morning:
run an independent canary test
With:
Persistent=false
a missed maintenance window is skipped instead of being unexpectedly executed during the next boot.
16. What remains local, and what does not
The final setup runs the language model locally.
That does not automatically mean every Hermes capability is local.
Depending on configuration, these tools may still use hosted services:
web search
browser automation
image generation
video generation
speech-to-text
text-to-speech
This separation is useful.
The model can remain local while selected tools use external providers. It also means that “local agent” should be defined precisely:
Local inference: yes
Local tool execution: yes
External web services: optional
For a completely offline configuration, external toolsets must be disabled and the Docker network can be set to none.
For this project, internet access was intentionally enabled because the agent was expected to perform tasks such as:
dotnet restore
npm install
git clone
web requests
dependency checks
17. Operational commands
The completed system can be managed with a small set of commands.
Model server
systemctl --user status llama-qwen.service
systemctl --user restart llama-qwen.service
systemctl --user stop llama-qwen.service
journalctl --user -u llama-qwen.service -f
Hermes gateway
systemctl --user status hermes-gateway.service
systemctl --user restart hermes-gateway.service
journalctl --user -u hermes-gateway.service -f
Health checks
systemctl --user start hermes-stack-health.service
systemctl --user start hermes-stack-deep-check.service
Hermes update
systemctl --user start hermes-stack-hermes-update.service
journalctl --user \
-u hermes-stack-hermes-update.service \
-f
Timers
systemctl --user list-timers 'hermes-stack-*'
Docker sandboxes
docker ps \
--filter label=hermes-agent=1
Local API
curl --fail --silent \
http://127.0.0.1:8080/health |
jq
18. Lessons learned
A local model is only one component
The difficult parts were not loading the GGUF file. The real work involved:
service management
resource limits
tool isolation
authentication
networking
firewall interaction
update safety
rollback
observability
More CPU usage does not guarantee more performance
Moving from six to twelve generation threads increased CPU activity but slightly reduced tokens per second.
Benchmark the actual workload.
Persistent containers preserve old settings
Changing Hermes configuration does not necessarily update already-created containers.
After changing CPU, memory, DNS, or network settings, persistent Hermes containers may need to be recreated.
A DNS failure may not be a DNS problem
The container showed a DNS timeout, but direct IP connectivity also failed.
The root cause was an nftables forward policy, not the DNS server.
Multiple firewall managers require careful ordering
Docker, iptables-nft, Firewalld, libvirt, and a custom nftables ruleset can all be individually correct while being collectively incompatible.
Automatic updates need explicit boundaries
We automated:
health recovery
canary checks
Hermes updates
backup
rollback
notifications
We did not automate:
full Arch upgrades
NVIDIA driver upgrades
llama.cpp package upgrades
GGUF model replacement
That division protects availability.
Final Summary
We started with a simple question: could Hermes Agent and a 27B Qwen GGUF model run locally on an Arch Linux workstation with 64 GB RAM and an RTX 3080?
The answer was yes, but not as a trivial one-command installation.
The model had to run with partial CPU offloading because 10 GB of VRAM was insufficient for the complete 18 GB quantized model. llama.cpp provided the required OpenAI-compatible API, automatic GPU fitting, quantized KV caches, and idle sleeping.
Hermes initially used a hosted free model, so local requests never reached llama.cpp. Reconfiguring Hermes with a custom endpoint moved the inference path completely onto the workstation.
Telegram turned the agent into a remotely accessible service. Docker converted dangerous host-level tool execution into a restricted sandbox with a dedicated workspace, CPU and memory limits, and explicitly controlled network access.
The most difficult technical problem appeared as a DNS timeout but was ultimately caused by an empty nftables forward chain with a default drop policy. Removing the conflicting forwarding chain allowed Firewalld and Docker to manage routing correctly.
Performance tuning demonstrated that maximizing CPU threads was not the right strategy. Six generation threads performed slightly better than twelve, while additional batch threads helped with large Hermes prompts.
Finally, systemd health checks, canary tests, update detection, backups, rollback, and Telegram notifications turned the experimental setup into an operational service.
The finished agent now provides:
Local Qwen3.6 27B inference
OpenAI-compatible API
Telegram access
Docker-isolated tool execution
Internet access through a controlled bridge
Automatic service recovery
Controlled Hermes updates
Backup and rollback
Working-hours-aware maintenance
The central lesson is that building a useful local agent is less about launching a model and more about engineering a small, secure, observable, and recoverable platform around it.
Views: 0
