Multi-Modal Terrain Classification and Sim-to-Real Transfer on a Quadruped Robot

Abstract

This research develops a multi-modal terrain classification system for a quadruped robot, utilizing data from an IMU, a time-of-flight depth camera, and an RGB camera. Data collection involved the robot walking over five terrains; grass/dirt, concrete/pavement, tile, wood, and carpet, and feature vectors were created from 4 second windows from that data. Classifiers k-Nearest Neighbors and Random Forest were trained on simulation data and a real-world dataset, resulting in the fused-sensor Random Forest model providing a 99.08% test accuracy on real world data.