GrokRover: Learning to Drive on Real Mars Terrain

Markus Hoehn, Gabriel Noya
Video placeholder: inference.mp4 Trained GrokRover drives to its goal, onboard camera views alongside

Trained GrokRover driving to its goal

GrokRover is trained with reinforcement learning in NVIDIA Isaac Sim, on real Jezero Crater terrain finished by Grok Imagine edits that a Grok agent prompts itself.

Pipeline

HiRISE sees 25 cm: the sharpest imagery that exists of Mars.

Elevation is estimated by comparing two photos of the same ground taken from different angles, an estimate too coarse to capture rover-scale rocks.

Grok Imagine turns what the camera sees into elevation, checked against the real measurements.

1. Seed canvas from real data: HiRISE photo | stereo elevation

2. Grok Imagine edit: rover-scale rocks and ripples in both halves

3. Recursive re-rendering: 84 edits per world, 8x texture density

4. Built USD scene, same spot: raw orbital data vs. full pipeline

Massively Parallel Training

Hundreds of GrokRovers learn to drive simultaneously on GPU.

Video placeholder: parallel-1.mp4 Parallel GrokRover environments training across a boulder field

Parallel training environments

Video placeholder: parallel-2.mp4 Overhead sweep of the full training terrain, agents everywhere

Overhead view of the training run

The Terrain Library

17 real Jezero worlds plus one imagined from scratch, each a self-contained USD scene with full real-data provenance. GrokRover trains across all of them.