RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

Overview

RT-SAFE evaluates embodied-agent safety in simulated urban environments where the world continues to evolve during inference and action execution. Navigation tasks combine moving actors, environmental hazards, and traffic rules to measure how both action choices and decision latency affect safety.

Across eight vision-language models, high task completion often masks safety failures. In matched static and real-time evaluations, completion rates are 91.3% and 94.1%, while real-time execution increases collisions by 12.3 times. In the hardest real-time setting, only 0.7% of episodes finish without a safety event. The paper also explores offline reinforcement learning to reduce collisions while maintaining strong task completion.

See the paper, code, and project website.