Beating Soccer Robots in the Real World by Reinforcement Learning and Sim-to-Real
Goal scored during Latin American Robotics Competition 2019 at Rio Grande - Brazil
This work presents an application of rl for the complete control of real soccer robots of the vsss, a traditional league in the larc. In the vsss league, two teams of three small robots play against each other. We propose a simulated environment in which continuous or discrete control policies can be trained, and a Sim-to-Real method to allow using the obtained policies to control a robot in the real world. The results show that the learned policies display a broad repertoire of behaviors which are difficult to specify by hand. This approach, called VSSS-RL, was able to beat the human-designed policy for the striker of the team ranked 3rd place in the 2018 LARC, in 1-vs-1 matches and won 4th place at LARC 2019 on 3-vs-3 official matches.
In Figure 1-a), we illustrate the model 3D developed by the RoboCIn team to compete in LARC. In 1-b), a real-world soccer scenario is shown to a better understanding of the IEEE Very Small Size Soccer competition. In Figure 1-c), the simulator used and adapted to train our models.
The IEEE Very Small Size Soccer (VSSS) League is one of the most traditional robot soccer competitions in the Latin American Robotics Competition (LARC), in the bellow image we have some information about the competition:
Proposed Environment Architecture
Our RL team placed 4th out of 28 in LARC 2019. These were the scores of all RoboCIn AI matches in LARC-2019:
RoboCIn AI 10 x 0 Pequi (ended by score difference)
RoboCIn AI 10 x 0 IME (ended by score difference)
RoboCIn AI 11 x 1 Pequi (ended by score difference)
RoboCIn AI 11 x 1 IME (ended by score difference)
RoboCIn AI 10 x 0 FBots (ended by score difference)
RoboCIn AI 11 x 1 FBots (ended by score difference)
RoboCIn AI 8 x 0 AraraBots
RoboCIn AI 5 x 7 Rinobot
RoboCIn AI 9 x 5 ITAndroids
RoboCIn AI 14 x 4 Pequi (ended by score difference)
RoboCIn AI 6 x 7 ITAndroids
We provide an OpenAI Gym environment for training agents for the VSSS League and two baseline agents capable of scoring 7 (DQN) and 10 (DDPG) goals in 5min against the simple AI implemented in the simulator.