Reborn Data for RFM
Last updated
Last updated
We are at the intersection of motion capture (mocap), virtual reality (VR), and embodied AI. By leveraging these technologies, we design immersive gaming environments that serve dual purposes: creating engaging user experiences and collecting high-quality datasets essential for advancing embodied AI research and applications.
Our approach integrates the latest advancements in mocap and VR devices, including platforms like Apple Vision Pro, enabling users to contribute to a decentralized data ecosystem while being rewarded through Web3 technology.
Reborn's data collection framework leverages advanced video capture, mocap, and VR technologies to generate high-quality datasets for embodied AI. The framework focuses on two primary types of data:
Video Hand Data Collection via Phones and Cameras Using common cameras (e.g., smartphones, GoPro), Reborn captures first-person perspective videos of fine manipulation tasks such as making a sandwich or washing dishes. This data is very useful for teaching robots fine manipulation skills, enabling them to perform complex tasks in the real world based on human-like actions.
Human Keypoints Data via Mocap Using motion capture technology, Reborn collects precise human joint data as users perform tasks in virtual environments. This data provides a detailed representation of 3D human poses and movements, essential for modeling human behavior and training AI systems.
Hand Landmarks and Manipulation Data via VR Devices VR platforms, including devices like the Vision Pro, capture hand lanrdmarks and manipulation interaction data, detailing hand gestures and object interactions in virtual settings. This data is crucial for tasks like robotic manipulation, human-computer interaction, and designing ergonomic virtual interfaces.
These datasets are gathered through immersive gaming experiences, enabling users to contribute valuable information seamlessly while engaging in interactive environments.
Reborn’s data is precious because it provides a complete, high-resolution view of human behavior through its end-to-end collection approach and achieves unmatched scale and diversity through its Web3-enabled ecosystem. This combination ensures that embodied AI systems trained on Reborn’s data are robust, adaptable, and capable of human-like performance in real-world scenarios. Reborn is setting a new standard for how data collection can drive progress in robotics and artificial intelligence.
End-to-End Data Collection: Reborn captures a complete record of human motion across daily life, from fine hand manipulations to full-body movements, in diverse real-world contexts. By integrating video, motion capture, and VR data, it provides a rich, multimodal dataset essential for training embodied AI systems to perform human-like tasks with precision and context awareness.
Scale and Diversity Enabled by Web3: Leveraging Web3 technology, Reborn ensures global participation and incentivizes contributors with tokenized rewards. This decentralized approach guarantees vast scale and diversity, conforming to the data scaling law (RFM) and enabling robust, generalizable AI models capable of adapting to varied environments and tasks.
The collected data can be processed and utilized to generate training datasets for Robotic Foundation Models (RFMs) across various scenarios:
Humanoid Task Planning
Data is collected from users wearing wearable devices as they perform specific tasks in a home environment, such as "going to the kitchen to fetch a glass of water."
These datasets enable robots to learn complete action trajectories, helping them understand how to execute complex, multi-step tasks autonomously in real-world scenarios [1].
Collaborative Robotics (Cobot)
Data generated in simulated environments captures human-robot collaboration, such as when a robot hands over a glass of water and the user reaches to accept it [2].
Mocap data enhances the modeling of synchronized actions between humans and robotic arms, facilitating smooth coordination for shared tasks.
Manipulation
VR devices like the Vision Pro capture detailed hand and arm movement data, including fine-grained finger dynamics.
This data is critical for training robotic arms to perform precise manipulation tasks, such as grasping, holding, and maneuvering objects effectively [3, 4].
These processed datasets provide versatile inputs for RFM training, equipping robots with the ability to perform complex tasks in diverse human-centered environments.
References
"Humanoid Robot Motion Planning Approaches: a Survey," Journal of Intelligent & Robotic Systems, 2023.
"Progress and prospects of the human–robot collaboration." Autonomous robots 42 (2018): 957-975.
"Trends and challenges in robot manipulation." Science 364.6446 (2019)
"Survey of Learning Approaches for Robotic In-Hand Manipulation." arXiv preprint arXiv:2401.07915 (2024).
