On-device inference routing
A drop-in OpenAI-compatible proxy that runs on the vehicle. It reads temperature, memory, load and link state on every request, then sends it to an on-device model, to the cloud, or to a safe default — and signs that decision into a record. Change one base URL; nothing else moves.
The gap
On a bench, inference is fast, cool and connected. On a vehicle it is none of the three, and that gap is where the failures live.
A satellite backhaul, an acoustic modem, a tether, a congested mesh. Calling a cloud model is an assumption that stops holding the moment the platform goes where it was built to go.
Pull the cooling on a loaded board and it starts slowing itself down inside four minutes. A stock local model server quietly used up all 8 GB of memory in half an hour. Both measured; neither showed up on the desk.
After an incident, the question is what the system decided and under what conditions. Without a record written at the time, the answer is reconstruction from memory.
Capabilities
Adoption is a base URL, not a rewrite. Every ✓ below is a result from a run on real hardware.
Every request is weighed against the live state of the device and how each model option is performing, then sent to the best one available — on-device first, or cloud first, whichever the mission calls for.
Put the failure conditions in deliberately, against whatever inference you already run. We point it at our own hardware — which is how we caught our own record noting conditions a moment too late. Found, fixed and re-verified on the board.
Declare what the platform does under named conditions — which model answers, when to hold for a human, when to fall back to a safe default — as configuration, reviewed like any other requirement.
Every decision joins a signed, tamper-evident chain on the device that made it, and any one of them can be pulled back out and run again against the rules you have now — or the ones you are considering.
Use cases
Compute that has to keep answering when the uplink will not — and account for what it did afterwards.
ProblemA tether or acoustic modem carries kilobits while a hull survey produces continuous imagery. For most of the dive, the cloud is unreachable.
SolutionRequests are answered on the vehicle by default; the cloud is used only in the windows where the rules and the measured link allow it. The vehicle surfaces with a signed record of what answered what.
On the benchAn hour of real link cuts: 840 requests, none failed, every urgent request held through the outage, and the whole record verified afterwards.
ProblemSatellite backhaul is metered and shared. A sun-loaded avionics bay throttles the compute exactly when tasking gets heavy.
SolutionRoutine work stays on the aircraft. Escalation is gated on priority, measured link latency and the temperature the board is actually running at — so the expensive path is spent on requests that earn it.
On the benchCooling pulled under load: the temperature rule raised the alarm 42–56 seconds before the hardware began slowing itself down, repeatably across four runs.
ProblemA fleet on shared Wi-Fi hits latency spikes and dead zones as it moves. An assistant that stalls mid-aisle is a stopped robot and a ticket.
SolutionLocal inference answers through the spike; the cloud is used when the link is genuinely healthy. Latency and spend are logged per robot, so fleet cost is a number rather than an estimate.
On the benchAn 8-hour shift: 6,226 requests, zero failures, memory flat to within 1 MB, and no thermal throttling.
Integration
Routedge accepts the same requests your client library already sends. Point the client at it and leave the rest alone — body, streaming and response shape are unchanged.
Your application · Pythonone line changes
# before — the client talks straight to a cloud endpoint
client = OpenAI(base_url="https://api.provider.example/v1")
# after — same client, now routed on the device
client = OpenAI(base_url="http://localhost:8000/v1")
# every call below is untouched
msgs = [{"role": "user", "content": "Describe this frame."}]
resp = client.chat.completions.create(messages=msgs)Anything that speaks HTTP · curlsame API
# the endpoint your stack already targets, on the device
curl http://localhost:8000/v1/chat/completions \
-H "content-type: application/json" \
-d '{"messages": [{"role": "user", "content": "..."}],
"stream": true}'
# no link, no cloud key, no change: the same call is
# answered by the on-device model instead.Architecture
One on-device process on the advisory path between your application and its models. Everything else attaches to that single point.
Fig. 1 — Runtime topologyOne process, on the platform
Fig. 2 — LifecycleBench to field and back
Platforms
A small Linux service on the platform's own compute. The cloud is an upgrade, not a dependency.
Routedge governs the AI advisory path — which model answers, what the platform does as conditions degrade, and what record that leaves. It is not a certified safety component and is not built to sit in a control loop that drives hardware directly. Where a request returns slowly or not at all, the application around it stays responsible for behaving correctly. Its job is to make degradation predictable and to leave evidence of it — not to promise it will not happen.
Every measured figure here comes from bench runs on one 8 GB Raspberry Pi 5; nothing is claimed for other hardware until it has been run there. Those figures are for a link that drops cleanly. A link that goes quiet instead of refusing is the harder case: it now fails fast rather than hanging, and an automated test holds the request through it — but that one has not been re-run on the Pi. Better you hear it from us than find it on the second call.
Contact
In active development, focused on defense and dual-use autonomy. If you are putting inference on hardware that has to keep working as conditions get worse, tell us what you are building.