Data Pyramid for Embodied Manipulation

Multimodal foundation models learned to see and to speak by consuming the whole internet.

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple…