When the lines of communication are open, individual agents, such as robots or drones, can work together to collaborate and accomplish a task. But what if they are not equipped with the correct equipment or if the signals are blocked, making communication impossible? Researchers at the University of Illinois at Urbana-Champaign started with this formidable challenge. They developed a method of training multiple agents to work together using multi-agent reinforcement learning, a type of artificial intelligence.
“It’s easier when agents can talk to each other,” said Huy Tran, an aerospace engineer in Illinois. “But we wanted to do it decentralized, which means they wouldn’t talk to each other. We also focused on situations where it was unclear what the different roles or jobs should be for agents.”
This scenario is a much more complex and challenging problem, Tran said, because it’s not clear what an agent should do against another agent.
“The interesting question is how do we learn to do a task together over time,” Tran said. Tran and his collaborators used machine learning to solve this problem by creating a helper function that tells the agent they are doing something useful or good for the team.
“With the team’s goals, it’s hard to know who contributed to the win,” he said. “We have developed a machine learning technique to identify when the individual agent is contributing to the team’s goal. From a sporting point of view, a footballer can score, but you also want to know other teammates’ plays that helped him to score, such as assists. These delayed effects are difficult to understand.”
Algorithms developed by researchers can also determine that a vehicle or robot is doing something that does not contribute to the goal. “It doesn’t matter much if the robot chooses to do something wrong; it’s something that isn’t useful for the ultimate goal.”
They tested their algorithms using simulated games like the popular Capture the Flag and StarCraft games. “StarCraft can be a bit more unpredictable; we’re glad to see our method working well in this environment.”
Watch Huy Tran’s video on Capture the Flag, showing related research using deep reinforcement learning to help robots assess their next move.
This algorithm applies to many real-life situations, such as military surveillance, robots working together in a warehouse, monitoring traffic lights, autonomous vehicles coordinating deliveries, or controlling a network of electricity, Trane said.
The study, “Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning,” written by Seung Hyun Kim, Neale Van Stralen, Girish Chowdhary, and Huy Tran, appears in the Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems held in May 2022.