The six-DoF hand-based teleoperation system enables intuitive control of advanced omnidirectional aerial robots for precise and flexible operation.
Researchers at the University of Tokyo’s Dragon Lab have developed an intuitive hand-based teleoperation system that allows human operators to control omnidirectional aerial robots with high precision.
This system, which operates in six degrees of freedom (6-DoF), utilizes motion-tracking markers and a data glove to correlate the operator’s shoulder and hand movements—finger gestures included—with the drone’s position and orientation.
The system was developed to overcome the shortcomings of traditional joystick-based multirotor interfaces, providing a control method that is more natural and adaptable for drones, especially in complex or unstructured settings.
In November 2024, Chinese researchers unveiled a wearable touchpad that lets users control drones by swiping their fingers and using Bluetooth to transmit commands instantly.
As robots are increasingly expected to perform complex tasks across larger and more dynamic environments, aerial manipulation is gaining traction due to its ability to operate freely in 3D space, especially in hard-to-reach or hazardous areas.
While full autonomy remains a long-term goal, human-in-the-loop teleoperation is essential for safe and reliable manipulation in unstructured aerial environments. Traditional multirotor controllers like joysticks or keyboards fail to fully utilize the six degrees of freedom (6-DoF) that omnidirectional aerial robots offer.
The new teleoperation framework features four distinct control modes—Spherical, Cartesian, Operation, and Locking. Each mode is tailored to different task requirements. Spherical and Cartesian Modes are optimized for long-range movements: Spherical Mode treats the operator’s arm as a polar axis, allowing the drone to follow the hand’s direction, while Cartesian Mode interprets hand motion within a localized coordinate system for straight-line navigation.
Operation Mode allows for accurate manipulation by directly correlating hand movement and positioning with the drone’s actions. With Locking Mode activated, the drone can maintain its position and orientation, allowing the operator to reposition themselves or change their viewing angle without affecting the robot’s state.
The researchers employed a data glove to create a gesture recognition system for seamless mode switching. Operators can switch between tasks without external input devices by measuring finger flexion, as specific hand gestures activate different control modes.
The active control mode is displayed visually within the operator’s line of sight using text and color cues, which enhances situational awareness and decreases cognitive load.
The Dragon Lab team confirmed the effectiveness of their system with real-world tests that involved avoiding obstacles, navigating corridors, and performing a valve-turning task.
During these tests, the drone was navigated into a target area with Spherical Mode and exited with Cartesian Mode. The manipulation task required Operation Mode to align and rotate a valve mounted vertically, which offered precise control. As visual occlusion arose, the operator changed to Locking Mode to modify their position while keeping the drone’s alignment intact.
The test results indicated a smooth trajectory tracking between the hand and the drone, with a latency of about 0.3 to 0.5 seconds, which is adequate for low-speed tasks. Operators indicated that Spherical Mode seemed the most intuitive, though assessing radial distance was still difficult.
Researchers found that while the Cartesian Mode improved directional clarity, it was not as intuitive. Although Operation Mode provided the highest level of control precision, it lacked tactile feedback, resulting in difficulties in sensing contact with tangible objects. Locking Mode was crucial for keeping spatial awareness and visual control in restricted settings.
In the future, the research team plans to develop the system into a fully self-contained one. This would necessitate outfitting both the drone and the wearable devices with onboard sensors for autonomous pose estimation, thereby removing dependence on external motion capture systems. Additionally, they intend to improve the system with force feedback for bilateral teleoperation, so that the operator can experience real-time interactions.
Moreover, according to the team, the drone’s end effector can be modified for wider uses, including cleaning, inspection, or retrieving objects, which might necessitate different interaction strategies.
The details of the team’s research were published in the pre-print server arXiv.
MasterCard
