Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. I think this is one of the more important things that happened in AI research lately. In the last few years, we have seen DeepMind defeat the best Go players in the world, and after OpenAI’s venture in the game of DOTA2, it’s time for DeepMind to shine again as they take on Starcraft 2, a real-time strategy game. The depth and the amount of skill required to play this game is simply astounding. The search space of Starcraft 2 is so vast that it exceeds both Chess, and even Go by a significant margin. Also, it is a game that requires a great deal of mechanical skill, split-second decision making and we have imperfect information as we only see what our units can see.
A nightmare situation for any AI. DeepMind invited a beloved pro player, TLO to play a few games against their new StarCraft 2 AI that goes by the name AlphaStar. Note that TLO a profesional player who is easily in top 1% of players, or even better. Mid-grandmaster for those who play StarCraft 2. This video is about what happened during this event, and later, I will make another video that describes the algorithm that was used to create this AI. The paper is still under review, so it will take a little time until I can get my hands on it. At the end of this video, you will also see the inner workings of this AI. Let’s dive in. This is an AI that looked at a few games played by human players, and after that initial step, it learns by playing against itself for about 200 years.
In our next episode, you will see how this is even possible, so I hope you are subscribed to the series. You see here that the AI controls the blue units, and TLO, the human player plays red. Right at the start of the first game, the AI did something interesting. In fact, what is interesting is what it didn’t do.
It started to create new buildings next to its nexus, instead of building a walloff that you can see here. Using a walloff is considered standard practice in most games, and the AI used these buildings to not wall off the entrance, but to shield away the workers from possible attacks. Now note that this is not unheard of, but this is also not a strategy that is widely played today and is considered non-standard. It also built more worker units than what is universally accepted as standard, we found out later that this was partly done in anticipation of losing a few of them early on.
Very cool. Then, almost before we even knew what happened, it won the first game a little more than 7 minutes in, which is very quick, noting that in-game time is a little faster than real-time. The thought process of TLO at this point is that that’s interesting, but okay, well, the AI plays aggressively and managed to pull this one off. No big deal. We will fire up the second game, in the meantime, few interesting details. The goal of setting up the details of this algorithm was that the number of actions performed by the AI roughly matches a human player, and hopefully it still plays as well, or better. It has to make meaningful strategic decisions. You see here that this checks out for the average actions every minute, but if you look here, you see around the tail end here that there are times when it performs more actions than humans and this may enable playstyles that are not accessible for human players. However, note that many times it also does miraculous things with very few actions.
Now, what about an other important detail, reaction time? The reaction time of the AI is set to 350ms, which is quite slow. That’s excellent news because this is usually a common angle of criticism for game AIs. The AI also sees the whole map at once, but it is not given more information than what its units can see. This perhaps is the most commonly misunderstood detail, so it is worth noting. So, in other words, it sees exactly what a human would see if the human would move the camera around very quickly, but, it doesn’t have to move the camera, which adds additional actions and cognitive load to the human, so one might say that the AI has an edge here. The AI plays these games independently, what’s more, each game was played by a different AI, which also means that they do not memorize what happened in the last game like a human would. Early in the next game, we can see the utility of the walloff in action which is able to completely prevent the AIs early attack.
Later that game, the AI used disruptors, a unit, which if controlled with such level of expertise, can decimate the army of the opponent with area damage by killing multiple units at once. It has done an outstanding job picking away at the army of TLO. Then, after getting a significant advantage, AlphaStar loses it with a few sloppy plays and by deciding to engage aggressively while standing in tight choke points. You can see that this is not such a great idea. This was quite surprising as this is considered to be StarCraft 101 knowledge right there. During the remainder of the match, the commentators mentioned that they play and watch matches all the time and the AI came up with an army composition that they have never seen during a professional match. And, the AI won this one too. After this game it became clear that these agents can play any style in the game. Which is terrifying. Here you can see an alternative visualization that shows a little more of the inner workings of the neural network. We can see what information it gets from the game, a visualization of neurons that get activated within the network, what locations and units are considered for the next actions, and whether the AI predicts itself as the winner or loser of the game.
If you look carefully, you will also see the moment when the agent becomes certain that it will win this game. I could look at this all day long, and if you feel the same way, make sure to visit the video description, I have a link to the source video for you. The final result against TLO was 5 to 0, so that’s something, and he mentioned that the AlphaStar played very much like a human does and almost always managed to outmaneuver him. However, TLO also mentioned that he is confident that upon playing more training matches against these agents, he would be able to defeat the AI. I hope he will be given a chance to do that.
This AI seems strong, but still beatable. I would also note that many of you would probably expect the later versions of AlphaStar to be way better than this one. The good news is that the story continues and we’ll see whether that’s true! So at this point, the DeepMind scientists said that “maybe we could try to be a bit more ambitious”, and asked “can you bring us someone better”? And in the meantime, pressed that training button on the AI again. In comes MaNa, a top tier pro player. One of the best Protoss players in the world. This was a nerve-wracking moment for DeepMind scientists as well, because their agents played against each other, so they only knew the AI’s winrate against a different AI, but they didn’t know how they would compete against a top pro player.
It may still have holes in its strategy. Who knows what would happen? Understandably, they had very little confidence in winning this one. What they didn’t expect is that this new AI was not slightly improved, or somewhat improved. No, no, no. This new AI was next level. This set of improved agents among many other skills, had incredibly crisp micromanagement of each individual unit. In the first game, we’ve seen it pulling back injured units but still letting them attack from afar masterfully, leading to an early win for the AI against Mana in the first game. He and the commentators were equally shocked by how well the agent played. And I will add that I remember from watching many games from a now inactive player by the name MarineKing a few years ago. And I vividly remember that he played some of his games so well, the commentators said that there’s no better way to put it, he played like a god. I am almost afraid to say that this micromanagement was even more crisp than that. This AI plays phenomenal games.
In later matches, the AI did things that seemed like blunders, like attacking on ramps and standing in choke points, or using unfavorable unit compositions and refusing to change it and, get this, it still won all of those games 5 to 0. Against a top pro player. Let that sink in. The competition was closed by a match where the AI was asked to also do the camera management. The agent was still very competent, but somewhat weaker and as a result, lost this game, hence the “0 or 1” part in the title. My impression is that it was asked to do something that it was not designed for, and expect a future version to be able to handle this use case as well. I will also commend Mana for his solid game plan for this game, and also, huge respect for DeepMind for their sportsmanship. Interestingly, in this match, Mana also started using a worker oversaturation strategy that I mentioned earlier.
This he learned from AlphaStar and used it in his winning game. Isn’t that amazing? DeepMind also offered a reddit AMA where anyone could ask them questions to make sure to clear up any confusion, for instance, the actions per minute part has been addressed, I’ve included a link to that for you in the description. To go from a turn-based perfect information game, like Go, to a real time strategy game of imperfect information in about a year sounds like science fiction to me.
And yet, here it is. Also, note that DeepMind’s goal is not to create a godlike StarCraft 2 AI. They want to solve intelligence, not StarCraft 2, and they used the game as a vehicle to demonstrate its long-term decision making capabilities against human players. One more important thing to emphasize is that the building blocks of AlphaStar are meant to be reasonably general AI algorithms, which means that parts of this AI can be reused for other things, for instance, Demis Hassabis mentioned weather prediction and climate modeling as examples. If you take only one thought from this video, let it be this one. I urge you to watch all the matches because what you are witnessing may very well be history in the making. I put a link to the whole event in the video description, plus plenty more materials, including other people’s analysis, Mana’s personal experience of the event, his breakdown of his games and what was going through his head during the event.
I highly recommend checking out his 5th game, but really, go through them all, it’s a ton of fun! I made sure to include a more skeptical analysis of the game as well to give you a balanced portfolio of insights. Also, huge respect for DeepMind and the players who practiced their chops for many many years and have played really well under immense pressure. Thank you all for this delightful event. It really made my day. And the ultimate question is, how long did it take to train these agents? 2 weeks. Wow. And what’s more, after the training step, the AI can be deployed on an inexpensive consumer desktop machine. And this is only the first version. This is just a taste, and it would be hard to overstate how big of a milestone this is.
And now, scientists at DeepMind have sufficient data to calculate the amount of resources they need to spend to train the next, even more improved agents. I am confident that they will also take into consideration the feedback from the StarCraft community when creating this next version. What a time to be alive! What do you think about all this? Any predictions? Is this harder than DOTA2? Let me know in the comments section below. And remember, we humans build up new strategies by learning from each other, and of course, the AI, as you have seen here, doesn’t care about any of that. It doesn’t need intuition and can come up with unusual strategies. The difference now is that these strategies work against some of the best human players. Now it’s time for us to finally start learning from an AI. gg. Thanks for watching and for your generous support, and I’ll see you next time!