World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation

The simulation-to-reality gap remains a fundamental challenge in robotics, particularly under partial observability, where time-varying hidden variables — such as contact, collision, and external disturbances — significantly affect dynamics but cannot be predicted from observation history. Learned dynamics models offer a promising path to minimize this gap by capturing domain-specific characteristics directly from data. We observe that simulators and learned models have complementary strengths: simulators explicitly model transient physics but suffer from inaccurate persistent parameters, while learned models capture domain-specific characteristics but cannot infer discontinuously evolving hidden variables.

We propose World Translation, which combines these strengths by reformulating sim-to-real gap minimization as unpaired domain translation over learned dynamics features. Our approach consists of two stages: (1) Backward Dynamics Extraction encodes the effects of hidden variables from observed state transitions into a compact representation, using transition outcomes rather than forward prediction from history; (2) Unpaired Domain Translation maps these features across domains, preserving hidden variable semantics while transforming domain characteristics. This enables the simulator to serve as a source of hidden variable information while correcting for domain mismatch. Experiments on locomotion and manipulation tasks show that World Translation achieves more accurate dynamics modeling than baselines, with the largest gains when hidden variables evolve discontinuously.