GrokRover is trained with reinforcement learning in NVIDIA Isaac Sim, on real Jezero Crater terrain finished by Grok Imagine edits that a Grok agent prompts itself.
Every world starts from Jezero Crater as photographed by NASA's Mars Reconnaissance Orbiter: the maps the Perseverance rover navigated against to land.
Its HiRISE camera sees 25 cm: the sharpest imagery that exists of Mars.
Elevation is estimated by comparing two photos of the same ground taken from different angles, an estimate too coarse to capture rover-scale rocks.
Grok Imagine turns what the camera sees into elevation, checked against the real measurements.
The pipeline in one loop: real data, Grok Imagine edit, recursive re-render, drivable world
1. Seed canvas from real data: HiRISE photo | stereo elevation
2. Grok Imagine edit: rover-scale rocks and ripples in both halves
3. Recursive re-rendering: 84 Grok Imagine image edits per world, 8x texture density
4. Built USD scene, same spot: raw orbital data vs. full pipeline
Why the pipeline matters: left, driving on terrain from raw orbital data alone; right, the same place with Grok in the pipeline
Chase camera on pipeline terrain. Red sphere: the goal. Green arrow: commanded heading.
Untrained: GrokRovers wander, goals unreached
Trained: the policy drives straight to its goal
Hundreds of GrokRovers learn to drive simultaneously on GPU.
Each GrokRover sweeps the terrain with a downward-facing lidar, reading the shape of the ground across a 4 m × 4 m patch at 25 cm resolution, and knows how far and which way its goal is. From that it outputs two commands, drive and turn, and its steering does the rest. It is rewarded for closing on the goal and penalized for rolling over, wandering, or reversing.
Proximal Policy Optimization
two networks train together from the same rollouts: the policy πθ drives, the value model Vφ predicts how well the drive will go
one training hour: 512 rovers in parallel · 40 s episodes · goals ~5 m out · ~30k transitions/s · ~37M transitions
Start of training: the swarm has no idea how to drive
Trained: the same swarm drives with purpose
Trained GrokRovers driving different worlds from the terrain library.
31 real Jezero worlds plus one imagined from scratch, each a self-contained USD scene with full real-data provenance. GrokRover trains across all of them.
