STMicroelectronics and Leopard Imaging accelerate robotics vision with NVIDIA Jetson-ready multi-sensor module

• Multimodal module combining 2D imaging, 3D depth sensing, and human-like motion perception 
• NVIDIA Holoscan Sensor Bridge ensuring multi-gigabit plug and play connectivity with Jetson platforms
• Fully supported by NVIDIA Isaac open robot development platform

STMicroelectronics and Leopard Imaging^® have introduced an all-in-one multimodal vision module for humanoid and other advanced robotics systems. Combining ST imaging, 3D scene-mapping, and motion sensing with the NVIDIA Holoscan Sensor Bridge technology, the module integrates natively with NVIDIA Jetson and NVIDIA Isaac open robot development platform, simplifying and accelerating vision system design within the size, weight, and power constraints of humanoid robots.

“Humanoid robotics is moving beyond research projects and demonstrations to deliver powerful new machines for a wide range of roles in manufacturing and automotive factories, logistics and warehousing, as well as retail and customer service,” said Marco Angelici, Vice-President of Marketing and Application for Analog Power MEMS and Sensors, at STMicroelectronics. “Our collaboration with Leopard Imaging brings market-leading ST sensors and actuators, seamlessly integrated into the NVIDIA robotics ecosystem, to accelerate the deployment of physical AI applications with human-like awareness.”

“Accessing to ST sensors and actuators directly within the ecosystem has allowed us to standardize and streamline data acquisition and logging for humanoid robot vision across the HSB interface,” said Bill Pu, CEO of Leopard Imaging. “Robot builders can use our multi-sensing vision module with Isaac tools to accelerate learning and quickly bridge the ‘sim-to-real’ gap.”

Powered by the NVIDIA Holoscan Sensor Bridge, the new module integrates seamlessly with NVIDIA Jetson over ethernet for real-time sensor data ingestion and NVIDIA Isaac open robot development platform, which offers open AI models, simulation frameworks and libraries for developers. The new module includes a build system and application programming interfaces (APIs), artificial intelligence (AI) algorithms curated for mobile robots, sample applications, domain randomization, and a simulation environment containing sensor models.

ST continues to integrate its sensors, drivers, actuators, controllers, and development tools into the NVIDIA robotics ecosystem as a key NVIDIA robotics and edge AI partner, including high-fidelity models and proof-of-concept modules.

Technical information

The Leopard Imaging Systems vision module incorporates:

For vision-based sensing, the ST VB1940 automotive-grade RGB-IR 5.1-megapixel image sensor with combined rolling shutter and global shutter modes. ST has also released a mass market and industrial version V**943, part of the ST BrightSense product family, existing in monochrome or RGB-IR, in die or packaged sensor.

For motion sensing, the LSM6DSV16X 6-axis inertial measurement unit (IMU) embeds ST machine-learning core (MLC) for AI in the edge, sensor-fusion low-power (SFLP), and Qvar electrostatic sensing for user-interface detection.

For 3D depth sensing, the VL53L9CX dToF all-in-one LiDAR module, part of the ST FlightSense product family, provides 3D depth sensing with accurate ranging up to 9 meters. With its resolution of 54 x 42 zones (near 2,300 zones) combined with a wide 55°x42° FoV providing 1° angular resolution, short and long-distance measurements and small objects detection are achievable at up to 100 fps.