jev-drone: MuJoCo drone with a judgment model in the loop

Fig. 1The full run, unedited. Chase camera on the left; on the right, the 64×48 onboard depth image, free space by sector, and the live judgment probabilities. 47 s, up to 3.6 m/s, zero contacts.

Fig. 2The recorded trajectory, drawn from the flight tape rather than sketched. Orange marks where the active judgment was climb: both beams, and nowhere else. The gate shutters are drawn where they were at the moment of crossing.On a small screen, scroll the figure sideways.

Abstract

We put a small judgment model (TypeSafe’s Jev) inside the control loop of a simulated Skydio X2 quadrotor that chases a ground rover through an obstacle course with a camera as its only sensor. The model is not the perception layer and it does not run at control rate. It answers three small, typed questions about a symbolic scene, and a 50 Hz reflex layer keeps the veto.

The same stack with the model disabled is safe but stuck: it never crashes and never gets past the second station, because flying over an obstacle is not something a steer-to-the-wider-side heuristic can express. With Jev engaged the drone clears all 77.5 m with no collisions. The claim is deliberately narrow, and the caveats are stated below.

Four loops, and only the slowest one is a model

Nothing about the flight is scripted. The drone is the Skydio X2 from MuJoCo Menagerie, a real airframe with four rotors. Each layer runs at the rate its job demands, and the judgment model sits at the bottom of that ladder, two hundred times slower than the controller.

500Hz

Geometric flight controllerAttitude, thrust-priority motor mixing, slew-limited commands. flight.Pilot

50Hz

Guidance and safety reflexAlways owns safety. Overrides any judgment when something is close. run.Guidance

15Hz

Camera to symbolic sceneDepth and segmentation buffers become five range sectors, an obstruction height and a target bearing. flight.Eye

2.5Hz

Tactical judgmentJev reads the scene as JSON and answers three typed questions. Advisory only. tactics.py

Fig. 3The rate ladder. Tick spacing is compressed, but the ordering is the point: blue layers are ordinary code, the orange layer is Jev.

Code decides when to ask. On an open corridor with the target in view there is nothing to decide, so no call is made. Scenes are fingerprinted, so an unchanged situation reuses the last judgment. Code also keeps the veto. Jev can propose climb, but guidance refuses it unless the obstruction’s measured top edge is actually within the aircraft’s climb ceiling.

sense · 15 HzOnboard camera

64×48 depth and segmentation. No ground truth, no map, no GPS.

perceive · numpySymbolic scene

Sector ranges, top-edge height, target bearing and range.

judge · ~2.5 HzJev

Choice, Score and Noul in one call. 0.11 s median.

veto · 50 HzGuidance + reflex

Accepts, refuses or overrides the proposal.

fly · 500 HzController

Quaternion attitude error, four rotor commands.

Fig. 4One pass through the loop. Jev never touches pixels and never touches motors; it sits between a symbolic scene and a guidance layer that is free to say no.

Three questions, three answer types

Classical computer vision turns the depth and segmentation buffers into a compact scene: five forward range sectors, the height of whatever blocks the path, whether its top edge is even visible, and where the target is. Jev reads that and answers three questions in one call.

Question	Type	What comes back
`maneuver`	Choice	hold_course, gap_left, gap_right, climb, brake or reacquire, each with a probability
`risk`	Score	A graded value from “clear and open” through “tight” to “about to hit something”
`target_truly_lost`	Noul	A probability: genuinely lost, or just briefly occluded?

in · what the camera code observed, t = 12.34 s

{
  "sector_range_m": { "far_left": 2.01, "left": 2.18,
    "center": 2.39, "right": 2.65, "far_right": 2.91 },
  "sectors_blocked_of_5": 5,
  "path_ahead_m": 2.18,
  "obstruction_top_above_drone_m": 0.14,
  "obstruction_taller_than_camera_can_see": false,
  "target": { "visible": true, "bearing_deg": -1.7,
    "range_m": 4.28, "unseen_for_s": 0 }
}

out · typed answers with probabilities

# Choice: maneuver
climb        0.83
gap_right    0.11
hold_course  0.03
gap_left     0.02
brake        0.01
reacquire    0.00

# Score: risk, clear -> about to hit
1.27

# Noul: target_truly_lost
0.04

Fig. 5A real exchange from the recorded run, 2 m short of the first beam. All five sectors are blocked, but the top edge sits 0.14 m above the drone, so the answer is inferable from the state. Jev picks climb at p = 0.83.

The judgments hold up on hand-built scenes

Six of seven correct, with strong probabilities.

Scene	Choice	p	Other
Low barrier, all 5 sectors blocked, top edge 0.45 m	`climb`	0.93	risk 1.42
Tall pillar ahead, right wide open	`gap_right`	0.72	risk 1.53
Tall pillar ahead, left wide open	`gap_left`	0.40	risk 1.58
Target gone 7 s, scene wide open	`reacquire`	0.96	lost 0.85
Boxed in, close on all sides, tall	`brake`	0.84	risk 1.89
All clear, target dead ahead	`hold_course`	0.82	risk 0.46
Brief occlusion 0.6 s, path clear	`(muddy)`	0.24	lost 0.12

The muddy case is instructive. The Choice was unconfident, but the Noul answered it cleanly (target_truly_lost = 0.12). So search behaviour is gated on the Noul, not on the Choice. Use the primitive that actually fits the question.

Five stations, each breaking a different assumption

Chase-camera view of the drone at the slalom station — Fig. 6Frames from the recorded run, in order. The red box is the rover the drone is chasing.

Chase-camera view of the drone at the low beam station — Fig. 6Frames from the recorded run, in order. The red box is the rover the drone is chasing.

The baseline is safe but stuck

The ablation is the same stack with the Jev call disabled, falling back to a greedy “steer toward the wider side” heuristic. It stops dead at station 2 every single time. With Jev the drone clears the entire course in about 47 s.

Jev engaged (1 run, 65 s)Baseline, no Jev (3 seeds)

Furthest point reached

Jev77.5 m

baseline17.7 m

Target kept in view

Jev82%

baseline~19%

Time pinned in safety reflex

Jev9%

baseline65-71%

Fig. 7Collisions were zero in every run on both sides. 80 model calls over 65 s, 0.11 s median latency, 96k tokens.

Where	x (m)	t (s)	z (m)	What happened
Beam	19	15.7	3.0	Climbed over
Turnstiles	26	20.3	1.6	Timed the sweeping arms
Gate	38	31.3	1.6	Threaded the sliding 3.2 m gap
Second beam	44	36.8	2.8	Climbed over
Cluster	50	41.3	1.6	Lost the rover in the pillars
Exit	58	46.6	1.6	Re-acquired it and closed back to 3.7 m

The cluster is worth noting. The drone genuinely loses the rover there, flies to where it estimates the rover has got to, and picks it back up. That recovery is the difference between a demo and a system.

Drone climbing over an orange beam, with the Jev panel showing the climb maneuver selected — Fig. 8Over the low beam. The panel shows climb selected.

Drone threading the gap between two cyan gate shutters while following the red rover — Fig. 9Through the sliding gate, target held in frame.

What is, and is not, the model

Layer	Who does it	Rate
Awareness: what is out there, how far, where is the target	numpy on depth and segmentation. Zero Jev.	15 Hz
Flight control: attitude, thrust, mixing	Geometric controller. Zero Jev.	500 Hz
Safety reflex: do not hit that	Code, and it overrides Jev	50 Hz
Tactical choice: over it, around it, which side, is it lost	100% Jev	~3 Hz

The state has to contain the answer

Jev initially refused to ever pick climb. That was correct. The state was five horizontal range sectors. It contained no vertical information at all, so “fly over it” was not an inferable option.

Adding the obstruction’s top-edge height, whether that top edge is visible to the camera, and the aircraft’s own climb ceiling moved climb from never chosen to p = 0.93. That was a state-design bug, not a model failure. It is the single most useful thing this project taught.

Simulator and controller lessons worth stealing

Most of the work was the simulator and the controller, not the model. One of these deserves a figure.

Fig. 10The geometric attitude error eR = ½ vee(RdᵀR − RᵀRd) has magnitude proportional to sin θ, so an upside-down aircraft sits at a stationary point with no restoring torque. A quaternion error is monotonic to 180°. Verified recovering from 179°, and the same fix extended the course run from 77 m to 88 m.

znear and zfar are fractions of the scene extent, not metresIn a 40 m arena, znear=0.05 blinded the drone inside 1.93 m, so the safety reflex never once fired.
Image-right is -y for a camera looking down +xThe target bearing sign was inverted and the yaw loop became positive feedback. The drone turned away from what it was chasing.
The camera sees its own airframeThe drone detected its own body as an obstacle at 3.94 m until own-body geoms were masked out.
Clipping negative motor commands destroys thrust and torque togetherThrust-priority mixing plus a slew-limited velocity command took peak tilt from 84° (unrecoverable tumble) to a 40° transient.
A quadrotor cannot thrust downward, and the controller has to knowOn a hard descent the desired thrust axis points at the floor and the controller faithfully commands inverted flight. Clamping desired vertical acceleration positive was worth more than every gain-tuning attempt combined.
Lateral control needs a position loop, not a velocity commandA held lateral velocity has no notion of where to stop. Dead-reckon laterally and correct only on plausible wall readings: 0.07 m median error against ground truth.
A lost target needs a world-frame estimate, not a bearingConvert bearing, range and own pose into a world position, carry it forward with the observed velocity, and fly there. Good to 0.14 m and 0.12 m/s, differentiated over about 1 s.
A reflex must brake for what is in your path, not what is beside youWalls 1.2 m to either side triggered a permanent brake, so the gate was not hard, it was forbidden. Splitting out the middle sectors fixed it.
Separate the tracking field of view from the threat coneA wide lens helps you see the target but ruins threat assessment: things 60° off the nose were never in the way.
Latch one setpoint per camera frameRe-applying a heading correction at 50 Hz off a 15 Hz camera applies the same error three times and spins the aircraft.
Sim time is not wall-clock timeThe sim ran at 10x real time, so 152 of 182 judgment requests hit a full queue and the model influenced nothing. Pace the sim to real time or you are not testing anything.

The tunnel experiment: Jev as the only navigator

A separate, harder setup: a 620 m enclosed tunnel, 15 obstacles, chasing a car at 9 m/s, with no reflex and no hand-written avoidance at all. Two graded Score questions (how hard to steer and which way; should it change height) map straight onto the control command, so the answer is the steering signal rather than a label to act on.

Measured live in a 500 Hz loop	Value
Decision latency, median	0.118 s
Decision latency, p90	0.164 s
Sequential ceiling	7.4 Hz
Pipelined, 4 to 6 workers, zero errors in ~2000 calls	21 decisions/s

Perception0.030 s · 3%

Jev decision0.118 s · 14%

Decision age0.024 s · 3%

Airframe translating 2 m sideways0.700 s · 80%

Fig. 11The reaction budget at 9 m/s. The model is 13% of the loop; the vehicle is 80%. Making the decisions faster cannot help, and commanding the airframe harder is what makes it tumble.

Drone chasing a red car toward a white obstacle in an enclosed tunnel, with steer and height score sliders on the Jev panel — Fig. 12Inside the tunnel. Here the panel shows graded steer and height scores instead of a discrete maneuver.

Status: partial. The aircraft is stable, flies full episodes and recovers from beyond 90°. In the best runs it dodged with zero collisions and 94% target visibility. But it does not reliably clear the whole tunnel: it tends to over-commit a dodge and end up against a wall. This is written up as a partial result, not a working system.

Run it yourself

The ablation runs without an API key. The Jev runs need one from TypeSafe.

git clone https://github.com/RomanSlack/jev-drone.git && cd jev-drone
./setup.sh                      # venv + Skydio X2 model
cp .env.example .env            # add your key
set -a && . ./.env && set +a
export MUJOCO_GL=glfw           # or egl on a headless box

.venv/bin/python run.py --seconds 65 --seeds 1                       # Jev engaged
.venv/bin/python run.py --no-jev --fast --seconds 65 --seeds 0 1 2   # ablation, free
.venv/bin/python run.py --seconds 65 --seeds 1 --video course.mp4    # telemetry video

Fig. 13A video run also writes a tape file, so visuals can be re-cut with replay.py without flying again or spending credits.

Every knob a human should review lives at the top of tactics.py: the three questions, the option rubrics, and every threshold that changes how the aircraft reacts to a judgment.

Questions people ask

Is the language model flying the drone?

No. Flight control runs at 500 Hz in a geometric controller, perception is numpy on depth and segmentation buffers, and a 50 Hz safety reflex overrides everything. Jev only makes the tactical choice (over it, around it, which side, is the target lost) about 2.5 times a second, and code can refuse its answer.

Does Jev see the camera image?

No. Jev is not a vision model. It receives a small JSON description of the scene, produced by classical computer vision, and returns typed answers with probabilities.

Why does the baseline fail at the second station?

The baseline steers toward the wider side. The low beam spans the entire corridor, so no side is wider. Going over an obstacle is not something that heuristic can express, so it stops at 17.7 m on every seed without ever crashing.

How much does a flight cost in model calls?

The recorded 65 s run made 80 calls and used 96k tokens at 0.11 s median latency. Code decides when a judgment is worth asking for, and unchanged scenes reuse the last answer.

Can I reproduce this?

Yes. The repository is MIT licensed. setup.sh builds the environment and fetches the Skydio X2 model from MuJoCo Menagerie. The no-Jev ablation runs for free; the Jev runs need a TypeSafe API key.