NVIDIA's free AI creates practice worlds for robots and self-driving cars
NVIDIA's four new Cosmos models generate realistic video and sensor data, so robots and driving systems can train and be tested in simulation first. Their license allows commercial use.
The short version
- NVIDIA released four new free AI models in its Cosmos family that create realistic video for training robots and self-driving systems.
- They can show the same street in different weather or light, seen from up to 11 cameras at once.
- Two of the models are built for robots, including one that shows what a two-armed robot will see after each move.
What NVIDIA released
Cosmos is NVIDIA's family of world models. These are AI models that generate video of how a scene looks and how it changes. NVIDIA has added four new versions to Cosmos 3. Two are for driving, and two are for robots.
The two driving models:
- Transfer-Auto creates driving footage from up to 11 cameras, including wide fisheye lenses. It can change the weather and the lighting, and even the time of day. It can also produce matching lidar data, the laser scans many self-driving cars rely on.
- Sim-Auto is a faster version that streams new footage while a simulated car drives. Engineers can watch how a driving system reacts as a scene unfolds.
The two robot models:
- Sim-Transfer takes the output of a physics simulator and turns it into realistic video that a robot's software can learn from.
- Sim-Bimanual predicts what happens when a two-armed robot moves. You start with a picture and move the robot or the camera. The model then shows the next view.
All four are released under a license that allows commercial use. Developers can also retrain them on their own camera setups and roads.
What is actually new
Robots and self-driving systems need practice in many different scenes and conditions. These models are built to supply it.
In the driving models, the camera views match the laser scans. The streaming versions also respond to each action. A test can run as a loop, where the robot or car acts and then sees the result before acting again.
NVIDIA says the problems remain hard. The next step is to measure how well training in these simulated worlds holds up in real use.
They also aren't simple to run. You need Linux and NVIDIA chips, plus engineering skill. NVIDIA's own test used one of its top-end GB200 systems.
What it means for companies in Indonesia
Few Indonesian firms will train self-driving cars. Many more are adding robots and automated equipment to warehouses and factories. The hard part is teaching machines to cope with a messy, changing real site.
Tools like these let teams test robot behavior in simulation before going live. After retraining it for its own robots, a logistics firm could test a picking robot against glare or a new layout.
The catch is skill and hardware. This is work for engineering teams with access to NVIDIA computing, whether in-house or rented from a cloud provider. For most companies, the practical question is whether their automation partners use tools like this.
What to do next
- Ask your automation partners whether they test robots in simulation before they go live.
- If you run robots today, list the conditions that trip them up, such as glare.
- Check whether your cloud or computing supplier offers the NVIDIA chips these models need.
- Watch for results from early users before you plan a large project around simulation.