What the Elevation Map Cannot See

What the elevation map cannot see

Legged robots usually read the ground through an elevation map built from depth or LiDAR. It is very good at boxes. It is much worse at a fallen bottle, and it cannot represent a phone or a water or oil stain at all, although stepping on any of them can end a walk. We group the hazards in our benchmark by how they appear in an elevation map:

One scene, three layers

Drag the divider. The left side is the scene’s texture; the right side is either the same surface coloured by height, or the elevation map a robot would get. In the real scene the surface is a handheld scan and the map comes from FAST-LIO2 with elevation mapping on the robot (5 cm cells). In the simulated scene the surface is the simulator’s own, and the map is synthesized with a LiDAR noise model fitted on the real map: a 6.2 cm lateral blur fitted on the real box edge, and 5.6 mm of floor noise. Fitted on nothing else, the model predicts the real bottle at 3.3 cm (measured 3.7 cm) and the phone at 0.6 cm (measured 0.5 cm).

Scene

Real sceneSimulation

Right side

LiDAR elevation mapScan, coloured by height

RGB — what the camera seesLiDAR elevation map

box — visible fallen bottle — ambiguous phone, spill — invisibleheight 025 cm

Height across each hazard

Height profiles through one object of each kind in the real scene, scan against map. The box survives almost intact, at 15.1 cm in the scan and 14.8 cm in the map. The fallen bottle loses about half its height, dropping from 6.8 cm to 3.7 cm. The phone (1.0 cm in the scan) and the spill both sit inside the map’s own floor noise. A policy that sees only geometry gets no signal from either of them. Hover a panel to read off both heights.

The same comparison as a camera move, in the real scene and in simulation:

Method

The locomotion policy keeps the network of an attention-based map-encoding (AME) policy that already traverses challenging terrain, and adds an independent semantic branch next to its elevation branch. The elevation branch encodes the shape of the terrain. The semantic branch adds a steppability prior that marks safe support and hazardous regions. Training has three stages:

1. Semantic teacher. Starting from the pretrained AME policy, PPO on challenging terrain with visible, ambiguous and invisible hazards added. The teacher sees ground-truth elevation and semantic maps and is trained with two foot-placement rewards, described below.
2. Robust teacher. PPO continues with curriculum randomisation of dynamics, payload, proprioceptive noise, pushes and external forces.
3. Deployable student. DAgger distils the teacher into a student that sees only a waist-mounted D435. Depth feeds the elevation branch. RGB, together with features from a frozen DINOv2-S backbone, feeds the semantic branch. On hardware, stereo depth from Fast-FoundationStereo replaces raw depth.

Rewards for stepping over hazards

Two terms shape how a foot crosses a hazard: a clearance reward while the swing foot moves over it, and a penalty for lowering the foot before it has passed. Both are active only for a swing foot whose sole footprint overlaps a hazard, and are exactly zero elsewhere, so the nominal gait is untouched.

With $d_i = z_i - (h_i + \epsilon)$ the clearance of foot $i$ above the top of the hazard beneath it, $H_i$ the overlap indicator and $g_i$ the swing gate (active only while the robot is commanded to move):

The horizontal-speed gate keeps a stationary raised foot from collecting clearance reward. The penalty grows as a descending foot nears the hazard; contact itself ends the episode.

What the student looks at

The salient regions are the walkable stake tops in the depth branch and the spills in the RGB branch. The policy attends to the terrain and the hazards around its feet rather than to its own legs.

Anti-deviation: closing the loop with navigation

Avoiding a hazard takes sidesteps, shorter steps or turns, and those leave the robot somewhere the VLN policy did not plan for. Executed open-loop, each waypoint $w_t$ becomes a constant velocity command, the deviation carries into the next VLN decision, and the error accumulates. Anti-deviation anchors the target pose at decision time, $p_t^{\text{tar}} = p_t \oplus w_t$, estimates the current pose $\hat p_t$ from the same monocular RGB stream (LingBot-Map in simulation), and maps the residual to a compensatory command:

$$d_t = p_t^{\text{tar}} - \hat p_t, \qquad c_t^{\text{comp}} = \kappa_{\text{ad}}(d_t)$$

The compensatory command takes the slot of the velocity command. The locomotion policy keeps its inputs and action interface unchanged.

Real-robot locomotion

The same student policy runs on a Unitree G1 with a waist-mounted D435. The course mixes boxes, fallen bottles, phones and water spills. The baselines are Elevation-only, a student distilled from the same teacher with the same schedule, depth camera and perturbations but without the semantic branch, and Sonic, a blind whole-body controller driven by a human expert with a joystick.

Three floor surfaces

Five trials per method and surface. A trial is a clean success only if it finishes without a fall, without leaving the corridor and without touching any obstacle or hazard. Contacts are counted but do not stop the trial. All numbers are percentages of trials.

Elevation-only touches an invisible hazard in every trial on every surface. Even with an expert giving the velocity commands, Sonic leaves the corridor in 80–100% of trials:

Disturbances

Locomotion in simulation

The simulated benchmark covers procedurally generated corridors in four terrain groups (uneven ground, stairs, stakes and stonebridge), each at three difficulty levels, with the six hazard types placed along the way. On top of that come perturbations to dynamics, payload, pushes, sensing, lighting, surface colour, camera pose and camera timing, all scaled together; at scale 1.0 the robot is, for example, pushed at up to 0.8 m/s every 3–7 s and carries up to ±2 kg of extra mass. All clips below are at the hardest level, L3.

Same layout, two policies

Hazard layouts are drawn from a seed that does not depend on the policy, so both policies walk exactly the same ground. On the left, Ours with its RGB and depth inputs; on the right, Elevation-only with its depth input. Over one continuous run the floor is re-textured live, switching between textures from the training pool.

Terrains and disturbances

With every perturbation at 1.5× its nominal scale (shoves up to 1.2 m/s, ±3 kg payload, friction, motor strength, lighting and sensor noise), with slow motion around the first shove:

Success across difficulty and perturbation

Success means finishing the corridor without leaving it, without falling and without touching any obstacle or hazard. Progress is the fraction of the distance covered before an episode ends. In clean conditions, Ours raises success over Elevation-only by 23.6, 28.5 and 25.8 points on L1, L2 and L3. The gap narrows under the strongest perturbation: at L3 and 1.5×, Ours still covers 56.7% of the distance but completes only 22.7% of episodes. Hover a column for its value; whiskers are ± one standard deviation over five seeds.

Where the semantic branch helps

On uneven ground, Ours succeeds in 50.6% of clean episodes against 9.4% for Elevation-only. The improvement comes from hazards the elevation map cannot show. ”% closed” below each group is the fraction of the Elevation-only → Oracle gap in hazard contact that Ours closes, $(\mathrm{HCR}_{\text{elev}}-\mathrm{HCR}_{\text{ours}})/(\mathrm{HCR}_{\text{elev}}-\mathrm{HCR}_{\text{oracle}})$. It is 74–100% for elevation-invisible hazards, 51–61% for ambiguous ones and only 12–24% for visible ones.

Navigation with anti-deviation

Navigation is evaluated in simulation, on the training and unseen validation splits of NaVILA’s VLN-CE-Isaac. Obstacles are added by script: 40% of them on the reference path and the rest within 1 m of it, so the robot has to avoid hazards both on the intended route and on any detour around them. The clips compare the same route and instruction, with and without anti-deviation; the green dashed line is the route the VLN waypoints intend.

Anti-deviation doubles safe success, raises success from 19.5% to 29.8% and cuts the collision rate by about two thirds. Average progress is slightly lower with it. Without execution feedback, a robot can still move roughly toward the goal while never reaching it.

Removing the semantic branch instead (keeping anti-deviation) lowers safe success in all four settings, by 3.3 to 12.3 points, and raises hazard intrusion by 8.6 to 11.2 points and the fall rate by 8.6 to 9.8 points:

Limitations

• The gap to the teacher remains. On clean uneven ground, Ours succeeds in 50.6% of episodes against the ground-truth-map Oracle’s 93.6%. Its contact rate on visible hazards is 24.7% against 2.2%, so handling ambiguous and invisible hazards alone does not make a route reliable. Because the two differ in both observations and training role, these results do not separate perception error from distillation or control limits.
• Falls under strong perturbation. At 1.5× perturbation the fall rate of Ours rises to 11.9%, against 5.8% for the Oracle and 3.1% for Elevation-only, possibly because the camera is sensitive to lighting changes.
• Memory and hardware. The waist camera sees only what is in front of the feet. Persistent semantic mapping could keep hazards that have left the field of view. The navigation framework has so far been validated only in simulation.

BibTeX

@article{yao2026elevation,
  title   = {What the Elevation Map Cannot See: Semantic-Aware Locomotion and Execution-Aware Navigation for Humanoid Robot},
  author  = {Yao, Shunyu and Liu, Songyang and Chen, Dinghao and Lei, Yuanyuan and Li, Shuai},
  journal = {arXiv preprint arXiv:2610.07396},
  year    = {2026}
}