Tip:
Highlight text to annotate it
X
>>
LANDIS: Hi, everyone. I'm Matt Landis with NGDU (ph), and we are thrilled to continue
our Serious Games Tech Talk series with Soren Johnson. As a designer, his work has included
some of the most complex and commercially successful titles out there: Civilization
IV, Spore, other web-based strategy games, and others I'm sure he can name. His Designer
Notes blog and Design of the Times column inspires and informs both novice and experienced
designers. And he continues to push the envelope for complex games that engage players and
give serious game advocates and educators like myself; hope that we can use these games
for learning, and growth, and change. And it's a pleasure to welcome Soren Johnson.
>> JOHNSON: All right. Thank you. It's great to be here. So, let's get started. So, a little
bit of my background. I've got a Masters in CS. You know, I did a lot of my work previously
in AI, which was kind of my first entrance into the videogame field. Big projects I've
worked on: Civ III, Civ IV. Civ IV was a project that I kind of led from beginning to end,
so I was very, very into the details of that project. I came on for the last year and a
half of Spore, did some work with AI, and helped design sort of the latter half of the
game. The year after that, I worked on sort of a small prototype, strategystation.com,
which is an attempt to build basically a Mod-dable site for strategy games on the Web. It's actually
built entirely with GWT, which I found to be kind of a really nice toolkit. And I'm
currently the lead designer of a web-based MMO, done inside of EA2D, which is a small
group inside of the EA Games label. But today, I'm mostly here to talk about AI. And here's
a little outline of what I'm going to be going through. First of all, I'm going to just define
two types of AI that people typically aim to make. One, I'll sort of categorize as "Good",
and the other type I'll categorize as "Fun". And I'll go into the details of, you know,
what the differences are. Then I'll talk about Civilization, because the AI for that game
kind of fits somewhere--sometimes uncomfortably between those two extremes and I'll talk about
some of the issues with that. Always a popular one, people discover that the AI cheats and
they're very upset so I'll talk about all those issues. And then, talk a little bit
about the development of the AI. And actually the AI to Civ IV was released publicly about
maybe six months after the game shipped. So if anyone actually wants to look through that
I'll forward the link to that code and hopefully we'll have time for some Q&A. So, what would
I mean when I say "Good" AI versus "Fun" AI? So "Good" AI, the idea is you're trying to
build an AI that is, you know, literally trying to beat the player. That's its pure goal.
It's meant to be a substitute for a person when, you know, you can't find someone else
to play the game with you or you don't want to wait for someone to do their turns, etcetera,
etcetera. "Fun" AI is a game where you're using AI algorithms as the actual content
in itself. And it's all--it's, you know, these are generally single players all focused on
the player's direct experience. You know, it's not about, you know, trying to replace
another human. So, you know, sort of the classic example of "Good" AI, of course, is something
like Deep Blue, you know, which is now capable of playing at the very highest levels. And,
you know, this is, you know, just--you know, you want to play chess and you want to play
against a human, you know, that problem is now solved. And people who--people who are
into this field has sort of get--take a perhaps a perverse pleasure in showing us that a lot
of the games we enjoy are now over, if you want to play against the AI; that no matter
what, they can beat us. It was proved a few years ago that checkers is solved. You know,
no matter what you do, the AI can always find a way to beat you or, you know, draw. And,
you know, as I mentioned before, you know, sort of a human substitute, you know, people
want to play all sorts of different types of, you know, war games, strategy games, board
games, and, you know, it's simply convenience. It allows them to practice on their own or
maybe they--that's just, you know, the way they prefer to play. They don't like the intensity
of playing against one of the humans or a friend. So the "Fun" AI, the algorithms themselves
are the content. Now the two examples I put up here are Black & White and The Sims. You
know, these are both examples of games which have a lot of AI in it, but that AI is all
about behavior. You know, what--and learning, in the case of the creature in Black & White.
You know, if you slap it around, it does one thing. If you pet it, it ends up, you know,
developing a different way. And The Sims is sort of this little, you know, ant farm-type
game where, you know, you place down, you know, you build your little house, place down
your own furniture. The one that makes the game interesting is seeing how the little,
you know, AI entities, you know, move around the environment and either enjoy their lives
or, you know, die because their bladder explodes. And here's an example that I'm going to do
draw on a lot in this talk. It's a game called Desktop Tower Defense which is a really fun
little flash tower defense game. I recommend everyone should play it because it's a really
neat little game. But essentially, in this game, you have--I'm sorry--you have what are
known as "creeps" come from one side of the grid and they're trying to get to the other
side. And your job is you're trying to prevent that. And it's essentially a game about path
finding. And what that means is, you're going to place down these little towers, which are
those little square things you see on the screen here. And, you know, the AI is going
to route through them and try their best to get to the other side. And you want to--you
want to make that as hard as possible for them. You want to inhibit their progress.
So, you know, in many ways, this is a game about path finding, essentially. That's what
makes the game interesting. If it didn't have, you know, a good path finding engine, you
know, it would--it wouldn't be nearly as fun. And the other main point about "Fun" AI is,
you know, focus is really on the player. And one of the best examples of this is the concept
of Aggro in MMOs like World of Warcraft which I'm showing here. And the idea here is, you
know, Aggro is a great case of "Fun" AI because it's not meant to be challenging. It's meant
to be understandable, comprehensible. And for people not familiar with this basic concept,
in MMOs, in sort of classic MMO combat, you have three different character types. You
have your tanks, your healers and your DPS. The tanks are there to absorb damage. They
are the most best defended, they have the best armor, for example. The healers are meant
to heal the units as they get hit and the DPS characters are there to actually damage
the enemies. And Aggro is a sort of an internal algorithm that determines who the AI is going
to attack, which, you know, who the monsters decide to target. And what Aggro does is,
as you attack--as you attack the monsters, it starts deciding, "Okay, that guy's been
doing a lot--is--that guy has been attacking a lot, so we're going to start targeting him."
Or, "That guy's been healing a lot so we're going to target him." But the game gives you
a number of ways to control Aggro. You know, there's various spells you can use to sort
of trick the AI to attack the wrong characters. You can, you know, there is, you know, big
Web pages and strategy guys devoted so you know how to control Aggro, how to, you know,
move your characters around so they don't suck up too much Aggro. Because, essentially,
what you want is you want your tank to get all of the Aggro of the monsters. And because
the tank isn't the one that's essentially doing all the attacking, that can be tricky.
And you'll see that this is really essentially a puzzle; it's not really AI. You know the--there
are--this is--what I'm showing right here is actually a World of Warcraft Mod; something
that people can, you know, put inside of the game where they can, you know, get quick information
about who the AI is Aggro-ing, who they want to attack and how that works. And this is
a sign that Blizzard actually wants players to be able to control Aggro. It's an active
game play element. It's not something that the AI is trying to defeat you, right, because
it would be very easy for their programmers to essentially say, "You know, Aggro is a
really kind of a dumb idea. We shouldn't let the players be able to influence it. We shouldn't
let the players be able to find out," you know, "Who is Aggro-ing who. We should instead,
you know, always focus on which character is doing the most damage." But that's not
actually fun. You know, it's more fun if, you know, Aggro becomes this little gameplay
element that you can manipulate. You know, the game is about the player after all. So
essentially, you have this spectrum which goes from, you know, "Good" AI on one side,
which is, you know--excuse me--Chess is sort of the classic example. And "Fun" AI on the
other side with Desktop Tower Defense. And, you know, there's--I'm going to talk about
a few other games and where they kind of fit in. StarCraft sort of fits on the left side
of the spectrum here, especially because it's a sort of symmetrical game. You're--you know,
when you're playing against one or two or three AI opponents, they're really playing
the same game as you are essentially. I don't mean, you know, the balance between Zerg,
Protoss and Terrans, but I mean that like they have the same goals; they're playing
on the same map. You know, they could be a human, they could be an AI; it's not really
going to matter. And also there's no real diplomacy in StarCraft, which is important.
It's something I'll talk about a lot later. A game like Heroes of Might and Magic is more
on the "Fun" side of the spectrum. You know, this is a game where you're, you know, you're
taking your hero character around the map, you're bringing an army with you, and you're
fighting a series of battles. But the game is essentially giving you a series of challenges
to overcome. It's not, you know, it's not seriously trying to, you know, defeat you
in the same way. It's more of an example of an asymmetrical game. Now, Civilization kind
of fits somewhere in the middle. And the main reason it's not on the far left like StarCraft
is because of the diplomacy issue, which I'll talk about later. But let's talk about the
few other things that define the difference between a "Good" AI and "Fun" AI. One is how
the rules change over time. On the far extreme left, a game like Chess, the rules are very
fixed. You know, they're not going to change, so that the AI becomes sort of a known problem.
Go is a good example of this because Go AI is actually much, much, much worse than Chess
AI. So this is a--this is a sort of a standard problem that, you know, the AI field is focused
on. Whereas on the other side of the scale, a game like Desktop Tower Defense actually
changes monthly. You know, they're constantly twiddling with the rules, adding new towers,
changing the way the mobs work. The point is not that there is some, you know, fixed
game that they're trying to, you know, improve the AI against, it's something that sort of
constantly changing. Symmetry is another big issue. When you're talking about "Good" AI,
generally speaking, the game is going to be very symmetrical. You know, all of the opponents
are playing the exact same type of game. On the far right side, you know, it tends to
be very asymmetrical where, you know, it's sort of like player versus environment essentially.
You know, you have to have this challenge and you're facing entities that maybe have
AI but they're playing a completely different game from what you're playing. Usually, the
only question is whether you can overcome that specific challenge. Generally speaking,
when you're talking about these type of games, "Good" AI tends to be multi-player focused
and, you know, "Fun" AI tends to be single player. You know, symmetrical games tend to
be mapped to multi-player very well because the idea is, you know, you just have two humans
seeing down, facing, usually facing off against each other, and the AI really is just a pure
replacement. Another issue is what type of tactics are available. When you're writing
"Good" AI, everything is on the table. You know, you're free to choose from any tactic
that a human might choose from. When you're a "Fun" AI, you very much actually want to
limit the tactics that your AI is going to pursue. Similar to my example earlier about
Aggro and WoW in that, you know, the AI designers purposely chose to take some things off the
table to let their AI be manipulated with Aggro. In Desktop Tower Defense, for example,
also, you know, you don't really want the mobs to develop intelligence so that they're
like, "Okay, let's actually gather our strength and let's wait," you know, "Let's form up
a big force for awhile and then we'll go through the maze." You know, that's something a human
might do if they were in control of the mobs. But we don't want the AI to do that because
that's not the point of the game, you know. The point of the game is that the player can
anticipate the way that the mobs work and anticipate the rhythm of the game. When you're
measuring the performance of your AI, there's also a difference. Usually, when you're talking
about "Good" AI, you have a very objective measure, you know. We kind of have this concept
of how Chess AI has been improving over the years, you know, every few years at a time.
And it's easy to measure that. But when you're on the right side of the scale, it's completely
subjective, right? It's just one of these things that depends upon sort of a gut feeling
about, "Are your players having fun or not?" It's almost more about a puzzle-type thing.
And when it comes to the Turing Test, you know, when you're on the far left side of
the scale, your goal really is to pass the Turing Test. And similarly with games like
Chess and Checkers, you know, they probably have done that, unless they're actually perhaps
too good at this point, that maybe a question. But they can easily pass that if we want to.
On the right side of the scale though, it's really irrelevant. No one thinks that the
mobs in Desktop Tower Defense, you know, are controlled by a human. It's obvious that they
are not. But that's not really the point. So, you know, here's a sort a quick summary
of the differences between the two. And this is one you might want to sum that up. You
know, "Good" AI is playing to win; "Fun" AI is essentially playing to lose. You know,
you're providing an interesting challenge for the human but your ultimate goal is, you
know, you want the human to have fun, you want them to win at the end. Okay, so what
about the Civ AI, you know, where does that fit in? Before we dive into that, I'll give
sort of a quick history of Civilization. Now, I assume a lot of you have played it here
so I'm probably not going to go into too much detail. But, you know, sort of a classic turn-based
game about the history of the world. Sid Meier first developed it back in 1991. It's had
four versions so far, although a fifth one is coming out in a month, I guess. And there's
usually been a version every four or five years. Civ II added more units. I mean, you
had a different view, create a scenario editor. Civ III came along in 2001. That was the first
one I was involved in. It added culture which controlled your borders and resources which,
you know, limited what you could build, which are kind of the big, big sort of gameplay
improvements there. Civ IV was in 2005; added religion and great people. Those are the big
changes. But we also went through and kind of tried to simplify all of the design craft
that had been developing, you know, over the first three versions of the game to kind of
pare back to the parts of the game that we really cared about. It's also the first game
where we really focused on multiplayer from the very beginning. That, you know, it wasn't
something that was just going to be sort of patched on as an afterthought. So, you know,
where does the Civ AI fit? Like I said, it's somewhere in the middle. So, that can lead
to some issues because it's not necessarily always clear where--what, you know, what solution
we should use for, you know, the various problems you face as an AI developer. So, for example,
when it comes to whether it's fixed rules or evolving design, Civ is kind of both. You
know, there's some core things about the Civ franchise that haven't changed in, well, I
guess in the 20 years it's been around. But obviously with each version, things change.
So it's not like we can still use the AI that Sid wrote back in 1991. It had to be rewritten
every time, and with every expansion pack, and even to somewhat times, with every patch.
It is a symmetrical game. You know, you expect that the--your AI opponents are playing the
same game as you are. You know, they're simply China while you're India or Germany; you know,
they're just--they're just a different Civ. Civ is generally known as a great single player
game. You know, there is a multiplayer community, but that's really--where probably 95% of our
players spend their time is in the single player. It is a game with limited options
and some people might actually not assume that, you know, right off the bat. But there's
actually a lot of things we don't let the AI do, because we feel like it would be annoying.
They would be, you know, types of play that we only really want that--we only really expect
a human should do and a lot of these issues come down to diplomacy. There is--there are
some subjective factors here but there is--there definitely are some objective test where you
can use to make sure our Civ AI is still going well. We have--we made some automated testing
with Civ IV where, you know, in every night I could see whether the AI was improving or
getting worse. And so, you know, those are some ways--are some ways to do it objectively.
And this is probably maybe the most different one. If people would like it to pass the Turing
Test, but it never will. It completely fails on the Turing Test. And that's because we
really want--it's kind of a funny thing about humans themselves. When we write the AI for
Civ, we actually want them to role-play. You know, we want Isabella to play a certain way.
We want Montezuma to play a certain way. We want Gandhi to be, you know, nicer than the
other leaders, perhaps, but even though he's not always. But when you play humans, you
don't actually really expect them to role-play. I mean, they're going to do whatever they
can to win, you know, including some very nasty things. And then maybe, like I said,
that may be okay for Montezuma, but it's not okay for everyone else. So the trick here
is trying to, you know, strike a balance. You know, where--what should we do with the
Civ AI when it comes in these really fuzzy areas where it's not clear whether it should
be "Good" AI or "Fun" AI. And making this problem worse is that every player is different.
I say there's really kind of three types of players for Civ. There is the Challenge players.
And these are people who really try to work their way up the difficulty scale. You know,
they want to eventually beat the game at Deity. You know, that's what defines Civ for them.
You know, there's the Sandbox players, they don't really care. You know, they probably
come to Civ from other games like SimCity or something like that and they're just kind
of enjoying the experience of going through history and, you know, and playing with the
different world wonders and with the different leaders, and, you know, just seeing how things
go. Then there's probably also the Narrative players, who really get into the concept of
role-playing, you know, where they really want play a leader a certain way and they
really want the AI to have--to have it's--their own personalities and their own styles. So
for these players, you know, we really--it was really important for us that the AI maintains
a memory of the things you've done, that if you've stab them in the back earlier in the
game, they're not going to trust you later in the game. And, you know, furthermore, it's
okay they even let the AI fall for traps. You know, that if the player has, you know,
gone to a lot of effort to trick them into thinking that something is one way when it's
really not, you know, we want the AI to make a mistake. You know, that gives the player
a lot of satisfaction. You know, at the end of the day though, we real--we want the players
to win or to at least understand why they lost. You know, that's really the number one
thing. And so on top of that, we need difficulty levels for a number of reasons. First of all,
it lets the sandbigs--it lets the Sandbox players off easy. If you play a game on Settler
on Civ IV, it's very, very hard to lose. In fact, the AI will never declare a war on you,
period, you know. I mean, you can--if you really try, you can lose, of course. But,
you know, generally speaking, if you just kind of flow through the game, you're going
to do fine. But on the other end of the scale, difficulty levels will give Challenge players
something to aim for. You know, a goal. You know, working their way up, up the scale.
And, you know, different tactics are available at different levels, which makes the game--you
know, there are certain things you could just never do at Deity, you know, if you want to--if
you want to compete and vice, you know, vice versa on Settler. So it kind of, you know,
just sort of changes the nature of the game. So, that leads to the question of sort of
AI cheating, because the AI in Civ is so complex, that there's no way we can just change the
difficult levels by making the AI dumber. Instead, it usually comes down to giving the
bonuses on the lower levels and giving the penalties on the higher levels. And AI cheating
is a very, very, very, subjective issue. And in fact, humans are notoriously poor at judging
cheats. You know, I've had years of experience of, you know, looking through the forums,
seeing how people respond to the games. And so it was very interesting to read so many
times they perceive the AI cheating when it's not cheating, and when it is cheating they
don't notice it, you know. It's very, very strange. And I don't how many people have
maybe played a game called Puzzle Quest, but it has kind of a very interesting issue with
cheating. This is a sort of a variation of Bejeweled, you know, which is a Match 3 Game.
You know, you're trying to arrange, you know, three-colored items in row and then they disappear,
and things fall from the top of the screen. And if you get lucky when they fall, it can
lead to chains and you get, you know, a series of matches which gives you an extra bonus.
So, the issue there is when it's the AI's turn, if they get that kind of lucky drop
from the top, people perceive that the AI got lucky. You know, when the human gets the
lucky drop, they always perceive that, "Well, that's because I was smart." You know, "I
arranged it that way so I was," you know, "my luck would be--it was more likely that
I was going to succeed." But when the AI gets it, you know, "It's a dirty, filthy, stinking
cheater." All right, are we still on? Is that just one projector? Okay. The AI is getting
back. But, yeah, in the--but the AI developer for Puzzle Quest has sworn over and over and
over again that the AI does not cheat, it s just purely random. And, you know, I mean,
I can't know for sure, but I've had similar experiences, so I believe him. And in fact,
I, you know, if I was talking to him I will recommend to actually anti-cheating, where
they make sure that the AI does not get lucky drops, because it's more important that the
human feel comfortable that the AI isn't doing something nasty behind the scenes then to
actually have a fair playing field, even if it means handicapping the AI, that's totally
fine. So there's a lot of sort of philosophical questions with cheating but, you know, let's
get into the details here. So obviously, on the far left of the scale, no one wants your
Chess AI to cheat, you know. That's not going to work. You know, they should be stealing
pieces while you're not looking and, you know. And on the far right of the scale, I mean,
you don't really play it like, you know--if the AI started warping units around the grid
of Desktop Tower Defense, you now, you just quit. You know, it does--you will--you would
assume actually it would be a bug and not really cheating. Now for Civ, as I said, yes,
we definitely cheat, but, you know, how should we cheat? So first of all, here are the production
modifiers. Now, when you play a game of Civ, on the Noble level, it's essentially even.
Where all the techs and buildings and units, they all cost the same for the AI as they
do for you. On Settler, everything cost the AI 50% more, so it takes them--you know, it's
slower. On Deity, everything cost them 40% less. You know, this has a huge impact over
the game because, of course, you know, Civ's one of those games where it's kind of slows--slow--slowly
snowballs, you know, where, you know, if you're able to get your grain out faster, that'll
let you grow citizens quicker which let's you work more plots, which let's you build
the next building faster, and, you know, you--it sort of grows exponentially. And here is the,
you know, the same idea in graph form. See, it's a very big difference. But actually,
even is not really even. And this is something that's bothered a certain subset of our players
a lot. They--we told--you know, we said, "Okay, Noble is essentially the even level", but
it's not really. There's a number of other cheats that the AI gets. It gets bonuses when
you're--when it's attacking Barbarians. It doesn't have to pay the same amount for unit
support or unit upgrades. It doesn't have the same issues with inflation and there's
a reason for that. The AI simply needs more help in this area. You know, when a human
plays the game, they have--they don't think of it this way, but they have a lot more options
available than the AI does. They can reload, right? They can open up the World Builder
and just get themselves some free units. They can--and even something more basic like, you
know, when you're playing the game of Civ, oftentimes, people will just leave all of
there cities empty, in their interior, of units because, you know, they know that, you
know, these units, these cities are safe, they're far from their borders and, of course,
they always have the option of reloading if something crazy happens. There's no way I
can have the AI follow a similar tactic, right? You know, if I said, "Okay, you know, let's
have the AI leave half of their cities empty." You know, they'd be wiped off the map very
quickly. So, it's just not an option to do those type of things. So, the question of
like, "Is there an actual level of--a difficulty level of Civ IV, where everything is completely
even; you know, the AI's playing the exact same game," that's really essentially an academic
question. And it's one that I'm not really concerned about. I'm only really concerned
about the player experience. And in that sense, you know, cheating should not be linear. You
know, there is--there are some things that the AI is going to need help with at every
level, regardless. So I guess this is important. However, it's very important that the game
never feels unfair. In Civ I, there was essentially free wonders. And what that means is not just
production bonuses but every once in awhile, if one of the AI was falling behind, Civ rode
in that it would just give you the Pyramids for free. Give the AI the Pyramids for free
or you're--have the Great Wall and so on and so forth. And this is something that just
felt very wrong to players especially, of course, if they were aiming for the same wonder.
And it's funny, the difference between where we give the AI just kind of a little bonus
every turn. You know, they maybe can still build the wonder before you do, but somehow
it feels more fair because, you know, you see the AI is actually building the wonder
instead of just being granted them. Even worse though was the idea of ganging up on the human,
which is that in Civ I, there's literally a piece of the code that says, "If the year
is 1900 and the human is in the lead, then declare a war on the human." And people suspected
it and it's true. But they also hated it. It's really--that was probably a mistake.
So with Civ III and Civ IV, we move to human-blind diplomacy, which means all of the diplomacy
calculations the AI uses is to determine who to attack, when they're willing to make peace,
who they're willing to trade with. At no point does it ask whether the person it's talking
to is a human or not. However, there are definitely lots of information cheats in Civ III and
Civ IV. And this is kind of a learning process because, if they--if the human can see behind
the curtain they can still feel unfair, so it's something we have to work on. And my
favorite example of this is the Amphibious Assault problem, which is sort of an interesting
exploit that players found after Civ III came out. Basically, and doing an Amphibious Assault
is a very tricky proposition for an AI programmer. And so one of the things we did is, "Okay,
we have four transports loaded up with goods, let's look at the AI Civs, let's look at our
enemy cities and try to figure out which one has the least numbers of troops in it, and
we'll go, and, you know, we'll try to invade right next to it." Players figured this out
and they figured out this way to essentially, you know, sort of trick the AI where they
said, "Okay, we're going to put one unit, where they'll leave the city empty because
we know the AI is coming with an invasion force. So I see the AI is going to start rallying
all the ships towards that city. But when it gets just a turn or two away, I'm going
to move a unit into that city and take a unit out of the city that's way on the other side
of the coast." And I think you're already anticipating what happens because, yeah, because
essentially, you see the entire fleet turn around and start going around to the other
side. And of course when they get there, you just switch them around again and then it
turns around, you got the AI stuck in a loop looking like a moron. So, you know, that's
not good at all. Although I do kind of admire their, you know, their ingenuity. We random--we
start randomly updating the assault targets that we only changed our target every eight
turns, every 12 turns; you know, it kind of changes randomly. We also ignore temporary
data like units. So, you know, the human could never just kind of change things from turn
to turn and to mess around with those calculations. Because really, I actually didn't--this is
the type thing that I don't trust myself to do it right so that there's no way a human
couldn't figure out a way to exploit it. So, let's talk about diplomacy right now because
this is kind of the big issue for, you know, the Civ AI. You know, "When is it okay for
the ally--the AI to ally with each other? Does that feel fair to the human?" Because
if you're facing a big, you know, AI alliance, you know, are they allying against you just
because you're the human or is it because that actually what makes sense in the game?
"Is it okay--should the human be trusted ever?" Like let's talk about the fake peace problem,
right. The--in--everyone I'm sure has probably done this at some point, but, you know, when
we're playing in the Civ, you're beating down some poor opponent and he comes begging to
you for peace. He says, "Here, I'll give you 200 gold if you just end the war with me."
And you're like, "Okay, sure. I'll take the 200 gold and next turn, declare war," right?
You know, because humans, of course, can't be tricked. Should the AI do something like
that, right? This is an example of something where we take an option off the table. In
Civ IV, we kind of solved this at the very top level, where we just said, "Okay, if you
make peace, you literally cannot declare war for 10 turns." But it's a good example of
something where, you know, the AI cannot really play the game the same way that the human
can. You know, we want--we want the AI to be role-playing where the actions they make
seem like what a AI character, what a historical character would do, not a human that's trying
to game the system, that's not what we're aiming for. And then there's the Tech Trading
Problem, where it really shows out that, you know, cheating is relative. Like here are
these three true sentences that I'll read here, you know. We know that when you're playing
the game, the AI should be trading techs with each other; they will all agree with that.
The AI should be trading techs fairly with each other. They shouldn't be just swapping
techs between each other because, like, "Oh, hey, you're an AI, too. Okay, well, we'll
give you a better deal." And then it's also true that the human has the agency to tell--sell
techs--sell technologies cheaper than they would normally just if they, you know, feel
like it's to their advantage. You know, the human can even give away techs if they'd like.
There's a problem here. These three sentences cannot all be true at the same time. Yet,
there's sort of an issue which is, "Should the AI be able to sell techs cheaply?" You
know, if you're saying that the AI should be able to pursue all the tactics that a human
would, then the answer would be, "Yes." But unfortunately, that leads to lots and lots
of problems. You know, namely that--the main problem is it felt like--okay, in Civ III,
I went out and said, "Okay, I'm going to have the AI try to trade techs the way humans would."
So, for example, if you're--if you're playing--you know, you're the human and you're playing
the game, Germany is your neighbor; they're very weak, they only have a couple of cities,
and they come to you asking for Iron Working, right? And you've got Iron Working, that's
a very powerful tech, you don't normally want to trade it away, but, "Okay, Germany's weak,
Germany will never be a threat, doesn't really matter to me if I give away Iron Working."
However, Germany only has 20 gold to give me, right? Clearly, Iron Working is worth
a lot more than that but I could never get more than 20 gold from Germany anyway so my
choice is either 20 gold or nothing. So in that situation, I might decide, "Okay, I am
going to trade Iron Working for much less than they'd be worth normally to Germany just
because I want the money. I might be better off than I was after making that trade." Fine,
you could say that's a good strategy. The problem is, is that if the AI pursues that
same route, it may be doing everything fairly, you know, underneath the hood. But from the
player's point of view, all they're going to see is all of the AI essentially, you know,
moving through the tech tree at the same rate and also at a very fast rate. And there'll
be these AI Civs out there that are very weak but have actually quite a lot of technologies
because, you know, all the AIs are essentially arbitraging with each other and trying to
get these small gains by trading away all their techs. In fact, this was referred to
by the name of, "tech ***" inside of our fan community. So, I tried to solve this problem
with the Civ IV AI. And I think it seems I basically solved it. It wasn't as big of an
issue with that game. Which is basically that I said, "Okay, I'll let the AI undersell it's
technologies for a third. But if at all possible you try to make up the difference in gold;
it only trades on certain random turn intervals." So it's not just constantly happening in every
turn. And almost--most importantly there's kind of a high level "Refuses to Trade With"
logic that's going on. Where even if we can make a deal there's some reason why I don't
like you; you know, you have a different religion, we have close borders, we've had conflict
in the past. The AI has used that same logic between each other as they do with a human.
And that seemed to give the system enough friction that it prevented that problem. You
know, where the AI can still undersell techs but not to such an extremity. You know, here
is really a point to--important to go back to the question of, "What's the whole point
of cheating?" You know--you know, "Are you trying to write the best AI possible? Are
you trying to beat the human? Are you trying to be fair?" You know, cheating is simply
for challenge, which is one source of fun, right? You know, when the AI's cheating, you're
trying to provide an interesting game for the players who are looking for a challenge.
But you try and do it in a way where it's still going to be, you know, a fun engaging
experience, and something that's not annoying or make the player suspicious of the AI, or
think the whole system is, you know, jerry-rigged against them. Which leads to this other issue
of, "To what extent do we design for the AI?" Because it's a--because Civ is a symmetrical
game, every option that we include in the game has to be handled by the AI as well as
for the human. Which means there's going to be some options that, you know, might be a
good design idea, but if the AI just completely can't handle it, we may have to take it out.
There can be some exceptions to that rule, but only if they come in very specific, you
know, very, very, you know, small places; things like nukes. Nukes are very powerful
but generally, we basically just disable them entirely for the AI so they don't have to
worry about handling them and players don't have to worry about handling them on the business
end, I guess. But there are a number of design changes we made over the years with the series,
especially in Civ III and Civ IV, where, you know, we changed the game design in a way
that really helped the AI managed its empire. Things like moving from localized unit support
where every city supported this specific number of units to global where it's just, you know,
all comes out of every National Treasury; the idea of closed borders where you can't
just wander freely to someone else's territory; enforce peace treaty, which I mentioned earlier,
which is the idea that when you make peace with someone you literally cannot declare
war again for another 10 turns. This ICS is a term called--it means Infinite City ***.
It's kind of a degenerate strategy where players essentially found a city on every other tile
on the map. And, you know, it's something we would never want the AI to do and we don't
really want the human to do it either. But most importantly, you know, we don't want
to have to build the way the AI in a way they're supposed to compete with that. However, every
one of these times when we make a design change in a way to help out the AI, we're also making
sure we're not just designing simply for the AI. You know, each one of these examples is
a change that the human would enjoy as well. Players really like the fact that their borders
actually meant something now. And they hated the AI--in Civ III, they hated AIs just sort
of randomly wondering through their territory. So it's a change that was really embraced
by the fan community for Civ IV. Same with enforced peace treaties because now diplomacy
in multiplayer games can be actually interesting, you know. If you come to me in a multiplayer
game of Civ III, I can say, "Hey, let's make peace." I go like, "Well, you know, it makes
no difference whether we're at war or at peace. I still can't trust you." But in Civ IV, if
I say, "Okay," you know, "here's peace." That means we literally can't attack each other
for 10 turns. You know, it actually has a meaning now. So I want to take a little bit
time here to kind of talk about some more details about the Civ AI. So one of those
things that's kind of unique about Civilization is that the AI has always been written by
the game's lead designer; which I think is--it shows that the AI is not really an afterthought.
It's really much, very much thought as a core piece of the design. You know, the way the
AI behaves is just as important as how the gameplay mechanics work. It's not actually
a very, you know, a large chunk of the code. And as an example, this is--this is the overall
Civ IV code-base. The AI is maybe just, you know, two to 3% of it. And the big issue with
AI, to me, usually is the question of, "How do you test it?" You know, it can be very--be
fairly easy to write some AI and, you know, it looks like it works but, you don't really
know until you're able to like, you know, stress test it essentially. You can't really
do traditional testing where you just--which, you know, in videogame development means,
you know, you take four, six, eight testers, you give them some checklist and you have
them play with the game and make sure the AI does, does, you know, this and that. Now
the problem is there's just so many variations to a game like Civ and so many things that
we're not even to be able to anticipate ourselves as the designers, that that's not going to
work. So, we use the--we use essentially a two-prong approach where we had some automated
testing, you know, where every night we would see, you know, how the AI has changed from
the previous day's build. But then we also gathered about a hundred of our hardcore fans
to play the game for a year or a year and a half before we released. That made a huge
difference, because they found all the things before we shipped that usually get found after
we shipped. So we were able to, you know, fix those things early. The other thing that's
important about the Civ AI is it's what I coined a term, "Soft-Coded," which that means
there's no AI scripts; there's no specific build orders. In fact, there's no enums in
the code at all. Like for example, you might imagine that in the code for a game like Civ,
there's a list of building enums, you know, where it's, you know, zero equals Temple,
and one equals Barracks, and two equals Granary. It actually was--essentially the way it worked
in Civ I, but it's something we've moved away from over the years, where everything is just
essentially data-driven. You know, all of those concepts exist in XML files that can
be changed purely as data. And, you know, we can't even--you know, the AI has to anticipate
the fact that, you know, the--this concept of a Temple might disappear. You know, we
might someday decide just to take it out or add different types. This leads to a less
brittle code. You know, obviously, if you have some sort of hardcore rebuild order if
the game balance changes, you know, that really changes, you know, that--it's suddenly now,
you know, this whole strategy doesn't work. It's also a less predictable AI which is kind
of--can be kind of a mixed blessing. So, when--well, let me give you an example here of what this
means. If you had, you know, an enum for a Temple inside of the function where you decide
what building to choose, you'd have something like this, you know. If this is the building,
you'll give it a--give it the weight of 20. And that's not the right way--that's not the
right way to do it. Instead, what you want to do is breakdown the Temple--you know, look
at the Temple at a high abstract level; what does it actually represent. Well, a Temple
gives you happiness and it gives you culture. So instead, let's assign values to how much
happiness it gives and how much culture. And beyond that, you know, maybe the city isn't
even--isn't unhappy to begin with. In which case, now the happiness that the temple brings
my city is irrelevant. So now suddenly, we have a much more, you know, nuance view of
how to give value to a Temple, something that could handle the only type of building that
we create that has this type of--has these types of values. Further, generally speaking,
when the AI is making a decision, it uses what I would call, "Probabilistic Reasoning."
Which is we look at a number of factors, we assign weights to them, we adjust those values
according to situation, we add some random noise, and then we finally, you know, come
out with the decision. And here is an example of the tech value function which is how it
decides which technology to research first. Obviously, this is a much bigger function
in the real game but here is, you know, kind of a smaller one. So first off, you know,
it says, you know, "Does this tech provide irrigation?" In that case, give it a certain
value. "Does it give open borders?" Okay, well, if it does, "Have I met anyone," because
if I haven't met anyone, that's irrelevant. If I have, okay, it gets some more value.
"Do I actually have coastal cities?" Because if I don't have coastal cities, it's unlikely--excuse
me--that I'm going to be able to do any trading with them yet. Okay, if I do, all right, we'll
give it a higher value and then finally add-in some random noise at the end. So, in a nutshell
it just kind of describes how we make the decisions. The nice thing about having a soft-coded
AI is it enables essentially very data-driven Mods. Double your Pleasure was a very popular
Mod for Civ III where, essentially, they doubled the game. You know, they added 100 buildings,
added 100 technologies, added a couple hundred units; all doing things that we didn't necessarily
anticipate originally. And the AI can essentially handle it because it use--it looks at buildings
abstractly, not in terms of, "This is a temple, this is a barracks, this is a bank." So they
can, you know, create new buildings that do things very differently and the AI can still
function. Fall from Heaven is a very popular Mod from Civ IV. That's essentially the fantasy
version of Civ, and it may have even added a magic system into the game to change the
way you can promote your units, to give them spells, for example. And similarly, it had,
you know, building types and technology types we just never anticipated and the AI was still
able to handle it. I mean, obviously, not necessarily as well as with the core game
because that's how the AI is balanced for, but still it handled it pretty well. So we
released the AI code for Civ IV about six to seven months after we shipped. It essentially
allows you to compile a custom DLL which replaces not just the AI but actually all of the core
game code. You know, the code that determines how techs are researched, you know, how buildings
are built, how diplomacy works, so on and so forth. It was 100% independent of the engine,
so it doesn't know anything about the graphics or the sound. I don't think it even necessarily
needs to be--know much about the operating system even. It just simply extends--and you
can use it to extend the game functionality. Like for example, in Fall from Heaven, they
added a magic system which is only possible because we allowed them to build their own
game DLL. Now here's a link to the Civ IV code. It's easy to find. You know, just type
in, you know, Civ IV code or AI--AI STK, and I'm sure you can find it. And that's it. And
it looks like we have a few minutes for questions. So, I'd be happy to answer anything.
>> I had a question about risk-taking as opposed to randomness, which is what I really liked
when I would play these games with a human and then afterwards, you'd be like, "Why the
hell did you do that crazy move?" And I will be like, "I was just going for it, man." Does
the AI ever do that? And when it does these weird things, does it ever sort of log it
so you could sort of see afterwards like, "Oh, that was a weird thing. It was trying.
It wasn't cheating." >> JOHNSON: It wasn't cheating? You know,
there's so much kind of noise and unpredictable-ness that's kind of inherent in the system because
the AI is such a, you know, a giant, a giant beast that I would say that sometimes it happens.
Mostly that was handled at the high level, with the diplomacy, right, because that really
changed what the AI was deciding to do. And so, that was kind of the constant question
of how random do we want the diplomacy to be. Because, usually, what you're talking
about, we're taking a big risk that usually involves declaring war on someone or deciding
that like, "I'm not going to help this guy out anymore." And I think we kind of solved
that problem with AI personalities, where some of the AIs are a lot more predictable
than others. When AIs, I mean, you know, Montezuma, Bismarck, Ghandi, Washington. So if you look
in--because actually in an AI, there's actually an XML file that defines the personalities
for those various different leaders. And you'll see--I forgot what they're called now, but
you'll see some columns. They're literally like, you know, randomness, or predictableness,
or which is I think fairly close to what you're talking about, which is you know--this is
the chance that this Civ will attack a larger opponent, you know, or try to buildup an alliance,
an unlikely an alliance against this guy. Or, you know, be really abstinent in, you
know, refusing to make peace even when they should. So, yeah, we had--we had some stuff
like that. I mean, it's--I'd like to, I'd like to--I think there's more we could do
like I'd like it to do some kind of crazier things but.
>> Yeah. I'd like that breakdown at the end. Like if it was random and whether it should
make peace so that at the end you could see, "Oh, wow, it like rolled with really low numbers
so that's why the hell it wasn't dealing with me."
>> JOHNSON: Yeah, that's the nice thing about playing a game like a D&D game where it actually
shows you like, "Oh, I actually--this crazy thing happened because this thing actually
rolled," you know, as opposed to something else. Yes?
>> How much does the AI cheats as far as like intelligence gathering? So I know in--if it
was Civ III or Civ IV, they added Espionage. But generally, it seems like the AI opponents
like have more knowledge about like how many troops you have than they would have been
able to surmise if you had close borders. >> JOHNSON: Yeah.
>> So, do they do cheating around that so they'd actually know whether you're a weaker
opponent or if you're just faking? >> JOHNSON: Yeah, that's always a tricky one
to answer. They definitely do, like they have complete knowledge. But that--I mean, the
AI is not some sort of independent entity that's kind of looking into things. Like I
only let them look at things at appropriate times, if that makes sense. I'm trying to
think of the best way to describe this. For example, I'm not--I'm not keeping track of
when units move in and out of the fog of war, because at least a really tricky AI problem
of, you know, "Can you--can you move--you know, can you move in this direction?" The
human sees like, "Okay, the AI have a bunch of units here and now they just moved in the
fog of war and now they're invisible. Well, I know they're there where I can't see them,
but I know--I know they're there." I'm trying to get the AI to be able to have a similar
type of memory where they're like, "Okay, there was a unit here a couple of turns ago.
It's probably still in this location but I'm not sure and that has to fade away over time."
That's doable but everything, when you're developing a game like this, is resource-constrained.
You know, it's not an academic project, right? So we have to say, "Okay, is it--is it worth
trying to develop this very complex memory system or should we just simply try to handle
this subjectively, where we let the AI see whatever we want to let it, but not go overboard,"
and that's essentially the line we have to walk. So they know--they know everything.
I mean, we don't limit them from knowing anything but, you know, we're making specific decisions
about when they kind of like get to look, get to peek, if that makes sense.
>> Cool. >> Hey, nice talk. Some real quick kind of
like series of questions; one, what the delay for the release of the AI code, I'm just kind
of curious; and also, did you see any--were there any interesting contributions from the
community that significantly changed the AI? >> JOHNSON: Right.
>> So the Mods you talked about were, you know, more gameplay Mods? And then maybe a
little bit about debugging if you want to work that in or maybe after this line.
>> JOHNSON: Okay. Yeah, I'd be happy to have a talk afterwards, sure.
>> All right. >> JOHNSON: Yeah. So there was a delay when
we--with the AI code. I mean, that was partially just because it was kind of a hard project
to push through legally inside of 2K. It was actually kind of--because it was this whole
thing I just kept pushing for over and over and over again and talking about it until
it kind of finally just happened, you know. And it wasn't--it wasn't necessarily an easy
project that they're going to buy-in on because they've never done anything like that before.
But there was also the issue of, games like this are big and they almost inevitably get
a number of patches the first three of four months after that. So we figured, we wanted--we
figured, it would be best to release the AI code when we knew that the code-base was not
going to change, and so that's one of the reasons we did that. And there were tons of
very, very interesting things that came through. Obviously, there were sort of interesting
gameplay Mods. Things like Fall from Heaven, there's a very interesting Dune Mod. There's
an interesting Mod where they kind of try to recreate Alpha Centauri. But there were
also some improved AI Mods where they're simply focusing on making a better AI. There's one
guy named Blake, who's from New Zealand, I think, who made a really fantastic one and
he eventually got hired as a contractor to work on the expansion packs. And the work
he did, I think, was literally just rolled into the expansion packs. So he wrote a better
AI for us and, you know, we used them and paid him for it.
>> Hi, so the Amphibious Assault I think made me think of this which is that, what a human
being would do in that situation is they'll go, "Oh, he's using this strategy on me and
I'm not going to win unless I change my behavior." >> JOHNSON: Right.
>> And there was--the human being would represent and then analyze the other human's strategy.
I noticed that these programs don't actually tend to do that. There will be sort of an
extremely soft-coded approach which is like figure out what your opponent's goal is.
>> JOHNSON: Right. >> And I noticed you guys don't do this in
any kind of AI situation, is that because it's too hard or because it would be too weird,
and too difficult for the... >> JOHNSON: I'd say it's too hard and it's
beyond our resources because what you're talking about is you need some kind of higher level
AI, that's above the AI that's watching the AI.
>> Right. >> JOHNSON: And like, "Oh, you're not making
good decisions. You need to change this." And that means there's a number of problems
there; challenges. You know, one is you just simply--that assumes you understand your AI
code better than you probably do. And in fact, there's been times we've experimented with
that and what usually happens is like, "Oh, wow, I didn't realize that that's actually
the way the AI's making the decision." So it's really hard. It'd be nice but, like I
said. >> So to follow up; is it a matter of processing
resources or a matter of... >> JOHNSON: It's a matter of developer resources.
>> Okay. I don't matter, I'm sorry. >> JOHNSON: I don't--I don't think the processor
is the issue. It's simply a question of how well you can understand your own code which
is constantly getting bigger and bigger and bigger and more complex.
>> Okay, thanks. >> Okay, so I was wondering how often, when
you were developing the AI, you'd have like some idea for something that seemed like it
would be really cool and you go and implemented it and it totally sucked, or vice versa. Or
someone was--someone's like, "Oh, that's a terrible idea," and you put it in, and it
just works amazingly well. And also just looking back, is there something that you would've
wanted add to the AI that maybe their adding for Civ V, or? And also, I guess if anyone's
on VC, you can pipe in with questions as well. >> JOHNSON: Right. Yeah, I mean, we make mistakes
all the time; all, all, all the time. That's very much what game development's all about.
Although most of the mistakes I usually think of are generally game mechanics mistakes,
the things the way we were changing the design of Civ III and Civ IV. With AI, AI is usually
catching up so that the game design--you know, the game design plows ahead and the humans,
you know, when we're testing the game, the humans are the only ones who were able to
really mess around with it and then sort of the AI comes up behind. When I think of mistakes,
it's usually stuff that comes afterwards like--reveal afterwards that makes me look dumb, I guess.
You know, things like the Amphibious Assault problem. If there's something I really wish
I could have fleshed out better, it would definitely be personalities. Like I think
there's a lot of space to grow for that where, you know, there are personalities in Civ IV,
but I feel like they could be a lot deeper, a lot richer, or remember a lot more things.
You know, when you talk to them, players get a lot of feedback if you say, "Oh, I remember
when you attacked me here," you know, stuff like that. You know, I would--I would like
to experiment more with that. >> Do--to what extent do AI Civ have an advantage
or a disadvantage because of their ability to micromanage units. Do they use the same,
the unit management that's kind of the default automatic one that the humans get or do we
do something different? >> JOHNSON: Yeah. Civ has a couple of examples
of places where it's making decisions automatically for you, especially if you turned on worker
automation or work sites to place the citizens, and it is essentially using the same code
as the AI uses. The AI has some micromanagement advantages, but I'd say those really come
to play more in real-time strategy games because time is a resource. You know, the human can't
be everywhere on the map, moving everyone around at the same time. In a turn-based game,
I mean, that's what happens to the highest players you can beat the game at Deity and
Emperor and so on. They're the ones who can spend 20 minutes doing a single turn, going
through everyone in the city, making sure everything goes in the right box exactly right.
Although that was actually one of our high-level design goals that Civ IV was trying to remove
that stuff. Like for example, in previous Civs, if you had--if you're building a granary,
and you had more hammers or shields than you needed they would kind of go to waste. So
it would be--instead of players to pan and to go back to their cities and rebalance that,
in Civ IV, we had those extra shields spillover to the next project, which means you no longer
had to do that type of micromanagement. And that kind of helps the AI and the human, and
as a high level, that's what we're trying to get to where--you know, there isn't really
a micromanagement advantage for either the AI or the human.
>> Okay. >> Okay, my only question is what makes in
I think most of the Civ games, the AI for naval combat, air combat, seemed just totally
different and much harder for the computer to figure out.
>> JOHNSON: Yeah, naval combat is especially hard. And this is the problem in a lot of
computer games. I would say the problem is just that it's so orthogonal to the rest of
the game. You know, sort of your classic land combat kind of fits in with a lot of the other
algorithms about determining, "Okay, these tiles has certain values. There's a resource
here; we want to protect it," you know, "This city is important," you know, and just spending
a lot of time thinking about path finding over the land. The water and everything is
just so much different. And especially when you're talking about Amphibious Assaults because
it's not just moving some ships; you have to build transports; you have to build the
appropriate number of land units; you have to move those land units to the right coastal
cities; you've got to move the transports to them; you have to make sure you have enough
escort units; you got to make sure those escort units are in the same--in the same cities
of the transports; you've got to make sure all those transports are ready at the same
time and make sure that they're in the path to their targets so one of them is not going
to get there two turns early. Compare that to like a land war where you're just like,
"Okay, get all my offensive units ready, move them towards the front, and go." You know,
it's really an order of magnitude difference between those problems.
>> Okay. >> JOHNSON: And you can see that in a lot
of other games, too. There's a lot of our RTS's have a problem where the AI's just really
terrible at Amphibious Assaults. And one great solution was in the RTS: Rise of Nations,
where when you take your land units to the water, they literally just turn into boats,
you know. They walk into the water, they magically become boats, you just cross the sea, and
they turn to land units again. I think--I think--I've heard Civ V is trying something
similar to this. >> Okay.
>> JOHNSON: And I think, generally, that's a good solution if you can--if it works for
your game. >> All right.
>> And I just want to pause one second to see if there's any questions from the VC.
With that, thank you very much. >> JOHNSON: Thank you.