On-device inference routing
Routedge is a drop-in OpenAI-compatible proxy that runs on the vehicle. It reads temperature, memory, load and link state on every request, then sends it to an on-device model, to the cloud, or to a safe default — and signs that decision into a record. Change one base URL; nothing else in your application moves.
The gap
On a bench, inference is fast, cool, and connected. On a vehicle it is none of the three — and that difference is where the failures live.
A satellite backhaul, an acoustic modem, a tether, a congested mesh. Calling a cloud model is an assumption that stops holding the moment the platform goes where it was built to go.
Single-board compute in a sealed bay runs hot, runs short on memory, and slows down. The local model that met latency on your desk misses it under sustained load in the field.
After an incident the questions are what the system decided and what conditions it decided under. Without a record written at the time, the answer is reconstruction from memory.
Capabilities
Routedge speaks the OpenAI chat-completions wire format your application already uses to reach a model, so adoption is a base URL rather than a rewrite. Everything below rides on that one integration point.
Every request is weighed against live device state and live endpoint health, then sent down an ordered fallback ladder. Routine work stays on the device; the cloud is used when the question is worth the link — or the reverse, when the mission says so.
Inject the conditions that break edge inference against any OpenAI-compatible endpoint, on demand. Each run reports how inference failed, not just that it did — a dropped link is classified differently from a hung connection or a slow answer.
Declare what the platform does under named conditions — which model answers, when to hold a request for a human, when to return a safe default — as configuration reviewed like any other requirement.
Each routing decision is appended to a hash-chained log and signed with a key held on the device that made it. Take one of those decisions, reconstruct the conditions around it, and run it again on a bench — against the rules you have now, or the rules you are considering.
Use cases
Compute that has to keep answering when the uplink will not — and that has to account for what it did afterwards.
ProblemA tether or acoustic modem carries kilobits, not megabits, while a hull survey produces continuous imagery. For most of the dive a cloud vision model is simply not reachable.
SolutionVision requests are answered on the vehicle by default, with the cloud rung used only in the surface windows where the rules and the measured link both allow it. The vehicle comes back with a signed record of which model answered what.
ProblemSatellite backhaul is metered and shared: sending every frame to a cloud model is expensive when it works and useless when it doesn't. A sun-loaded avionics bay throttles the onboard compute exactly when tasking gets heavy.
SolutionRoutine classification stays on the aircraft. Escalation to the cloud is gated on request priority, measured link latency, and the temperature the board is actually running at — so the expensive path is spent on the requests that earn it.
ProblemA fleet on shared Wi-Fi hits latency spikes and dead zones as it moves. An operator-facing assistant that stalls mid-aisle becomes a stopped robot and a support ticket.
SolutionLocal inference answers through the spike, and the cloud model is used when the link is genuinely healthy. Per-request latency and cloud spend are logged for every robot, so fleet cost is a number rather than an estimate.
Integration
Routedge listens on the device it runs on and accepts the same chat-completions requests your client library already sends. Point the client at it and leave the rest of the application alone — the request body, the streaming behavior, and the response shape are unchanged.
Your application · Pythonone line changes
# before — the client talks straight to a cloud endpoint
client = OpenAI(base_url="https://api.provider.example/v1")
# after — same client, now routed on the device
client = OpenAI(base_url="http://localhost:8000/v1")
# every call below is untouched
msgs = [{"role": "user", "content": "Describe this frame."}]
resp = client.chat.completions.create(messages=msgs)Anything that speaks HTTP · curlsame wire format
# the endpoint your stack already targets, on the device
curl http://localhost:8000/v1/chat/completions \
-H "content-type: application/json" \
-d '{"messages": [{"role": "user", "content": "..."}],
"stream": true}'
# no link, no cloud key, no change: the same call is
# answered by the on-device model instead.Architecture
One on-device process sits on the advisory path between your application and its models. Testing, rules, records and replay all attach to that single point.
Fig. 1 — Runtime topologyOne process, on the platform
Fig. 2 — LifecycleBench to field and back
Platforms
A small Linux service on the platform's own compute. It needs no cloud connection to operate and treats the cloud as an upgrade rather than a dependency.
Routedge governs the AI advisory path — which model answers a request, what the platform does when conditions degrade, and what record that leaves behind. It is not a certified safety component, and it is not built to sit in a control loop that actuates hardware directly. Where a request through it returns slowly or not at all, the application and the systems around it remain responsible for behaving correctly. Its job is to make degradation predictable and to leave evidence of it — not to promise it will not happen.
Contact
In active development, focused on defense and dual-use autonomy. If you are putting inference on hardware that has to keep working as conditions get worse, tell us what you are building.