Raphael Thys
Humanoid robots, trained to play soccer with deep reinforcement learning, play a one-versus-one soccer game. Google DeepMind
Our daily news digest will keep you up to date with engineering, science and technology news, Monday to Saturday.
By subscribing, you agree to our Terms of Use and Policies You may unsubscribe at any time.
Researchers utilize miniature humanoid robots trained through deep reinforcement (deep RL) learning to demonstrate their ability to engage in agile, dynamic, and remarkably intricate one-on-one soccer matches.
The robots demonstrated their mobility abilities by walking, turning, kicking, and swiftly returning after falls. They also seamlessly transitioned between various actions. As they played strategically, the robots learned to anticipate ball movement and block their opponents’ shots.
Featured Video
Related
- Saudi Arabia’s first male humanoid robot’s ‘groping’ incident sparks debate
- Top 7 robotics stories of 2023 — Interesting Engineering
- Artificial general intelligence: Understanding the future of AI
According to the team at Google’s DeepMind, based on these results, deep RL may provide a way to teach humanoid robots basic yet secure movements. This may then be used to teach them sophisticated behaviors in dynamic environments.
The details of the research were published in the journal Science Robotics.
Advancing embodied intelligence
Artificial intelligence (AI) and robotics researchers have long aimed to create generic embodied intelligence or agents that can act in the physical world with the same dexterity, agility, and comprehension as animals or humans.
Efforts to develop intelligent embodied agents with advanced motor skills date back years, with recent strides driven by deep reinforcement learning.
Quadrupedal robots showcase diverse abilities, including locomotion, ball handling, and manipulation. However, humanoid control lags, mainly addressing basic skills due to stability and hardware constraints, relying heavily on model-based predictive control.
Simulated and real soccer environments created with a 5m x 4m pitch; real setup includes motion capture for robot and ball tracking.
Deep RL blends supervised learning and neural networks with the reward-maximizing objectives of reinforcement learning. A deep reinforcement learning approach seems appealing since bipedal robots are hard to manage with standard techniques that need programming for every movement task.
The team utilized this approach to train inexpensive, readily available robots for multi-robot soccer, surpassing anticipated agility levels. They demonstrated sensorimotor control capabilities in simulation and real-world deployment by addressing simplified one-versus-one soccer scenarios.
Their training approach involved two stages. Initially, they train two skill policies: one for getting up and one for scoring against an untrained opponent. Then, they train agents for full one-versus-one soccer Through self-play, drawing the opponent from partially trained copies of the agent. They use rewards, randomization, and perturbations to enhance exploration and ensure safe real-world transfer.
Mastering dynamic movement
The resultant agent demonstrates strong and dynamic movement abilities, including quick fall recovery, walking, turning, and kicking, and it can switch between these actions with ease.
It also gained the ability to deflect opponent shots and predict ball motions. Manually creating the agent’s tactical conduct would be impractical because it reacts to particular game conditions in this way.
After undergoing simulation training, the agent seamlessly transitioned to real robots. The good-quality transfer was made possible by combining focused dynamical randomization, perturbations during training, and control at a high enough frequency.
Gallery of robot behavior.
“In the experimental matches, the trained robots walked 181 percent faster, turned 302 percent faster, kicked the ball 34 percent faster, and took 63 percent less time to get up from a fall than robot agents working off a scripted baseline of skills,” said the team in a statement.
Additionally, the researchers observed that throughout their matches, the deep reinforcement learning-trained robots displayed emergent behaviors like spinning and pivoting on a foot’s corner that would be difficult to script.
Based on these results, the team highlights that deep RL may provide a way to teach humanoid robots basic yet secure movements, which may then be used to teach them sophisticated behaviors in dynamic environments.
Future research could explore training teams of multiple agents, as shown in preliminary two-versus-two soccer experiments where agents displayed division of labor but less agility.
Additionally, researchers aim to train agents solely from onboard sensors, posing challenges in inferring information from egocentric camera observations without external state data.
Are bending lights the future of Wi-Fi?
SwRI demonstrates robotics technology for in-space manufacturing
New e-glove could allow scuba divers shout ‘shark’ attack underwater
New e-glove could allow scuba divers shout ‘shark’ attack underwater
World’s largest cavern thermal energy storage to warm a city year-round
Can depleted oil and gas reservoirs become hydrogen vaults?
Mini brains wired: Scientists grow brain-like thinking tissues in lab
Transplant ready: Formula saves 2-year-old from chronic kidney disease
Tiny tech consumes 10,000x less power to measure qubits in single shot
Breakthrough bullet train-inspired pump powers failing hearts wirelessly
Mysterious radioactive capsule removed from Russian city bordering China