Massively Multiplayer Open Source Game Development

>> Welcome to the Tech Talk. Today, Keith is going to talk about Multiplayer Open Source Online Role-Playing Game or [INDISTINCT] called PlaneShift. Keith has been with the project since 2001 and he's currently--he's the CTO of ChoicePay as well. So, [INDISTINCT] Keith. >> FULTON: Thanks, Andrew. Hi everybody. I'm glad to be here today. I'll first talk about--and I'll just try to introduce PlaneShift for those of you. How many in here have heard of PlaneShift before or have seen it? Okay, a few of you. Well, that's good. You know, PlaneShift has been around for a long time and has been a very ambitious project. We're trying to build an MMO in the--in the same type of [INDISTINCT] as World of Warcraft, or EverQuest, or Age of Conan, or a million of those games. Back when we first started, EverQuest and Ultima Online were kind of our competition. But considering that we're, like, four guys in garages around the world. It's hard to say we are competitors of theirs. The game is free to play and the source code of the game is completely free. So it's all GPL, it's all in SourceForge. Anybody here can go out there and pull it down and build it. It might take you a couple of days to get it to build the first time. But after that, that is pretty easy to keep it going once it's going. All the content though, we have kind of a dual license system and the content, the artwork, the quest, the puzzles, the maps, those things are all under--come by more a proprietary license and that's how we keep the game from forking and people creating all their own shard servers and having 1,000 different, you know, versions of the game. We like to say it's a way we prevent *** shift. So if you think about naked mods and things like that, we'll talk about some of that in a little bit. The project started in 2000. This is when the 3d project started. There actually was a 2d version of the project that started in 1992 with Luca in Italy and a few of his friends who are into role-playing games because they wanted to take what they were doing on the desktop and move it to just--you know, make a computer game out of it. But the modern generation game started in 2000. It got going and [INDISTINCT] in the summer of 2001. I joined the project in I think November of that year. The--I was talking to Andrew about this a little while ago and I think the 62 coders, the named people that have committed code, that's actually since our subversion conversion. We first started in CVS. And so--that number probably is totally wrong. We converted the subversion in I think summer of 2006. So that was a couple of years ago. So we've had 62 committers since then just in the last two years. So it's probably more like 150 or so overall. And then--and that's just with the code. You know, we've had dozens and dozens of contributors across the artwork, and the writers, and the various other people that have supported the game. It's relatively large, you know, for an Open Source project. We got--we use Crystal Space which itself is a very large project. So that's over a million lines of code. And then our code is about 400,000 lines divided up into about 1,200 classes. So this creates a lot of learning curve issues for us as people want to volunteer, and then they get into it, and they see what's there, and then they can't--they don't know where to start or how to do anything. So supporting those people and figuring out how to enable them to be successful and give us patches is a big part of my challenge in term--in terms of making the team more productive. We have over the past three years on the current release. You know, we've had over half a million of people who successfully registered accounts and tried the game. Most of those people login. They--you know, they create a character. They run around for a while. They want to see what it is. And then they quit and they go on their way. We only have about 100 to 200 players that play in any given day or at any given moment. And there are probably about five or 6,000 that will login in any given month. And we can talk about why some of that is related, too. You know, probably no news to anybody in here but MMOs are very complex pieces of software. Typical Open Source structure is that you have a coding team and that's it. And what you're building is you're building a better version VI or whatever your project is and that code is all there is. You might have a QA team if you're a very formal, larger Open Source project. We have a much more structured concept and we have a much larger complicated team. So the teams that we have today include the 2d team, which is doing concept art; they're doing pencil sketches and charcoal drawings and things of that nature to be grist for the mill for the 3d modelers. The 3d guys are actually doing the modeling and then they're doing animation. And generally, their specialties were the--the more beginner modelers are building items or they're building terrain. Then you have more advanced modelers are building monsters and creatures. And then the most advanced modelers are the ones building the player characters because those are the ones people are really picky about. The sound team does sound effects; every swoosh of the sword, every dramatic trumpet fanfare when you get a level or something. All of that stuff has to be built on keyboards and, you know, rendered. The rules team is figuring out how to make it into an RPG; what is the progression? How are we balancing things? What are the 200 different spells that we're going to have? Somebody has to decide all that. That's what this team does. They basically work in Excel. That's--their number one tool is Excel. Then we have a web development--oh, settings. I forgot settings. Settings are the writers. They're the poets and the dreamers, I call them. They're the kind of the prima donnas on our team where they are wanting to write these long descriptive background stories of--you know, in order for this quest to be relevant to this player, now you have to understand 1,000 years worth of history that leads up to this blacksmith who wants a sandwich from the sandwich shop. So--but those guys really enjoy doing that. The wed development team is built--really is managing the content. We have a--you know, a large data model that supports all the reference data in the system. And so when they add a new item, or they add a new quest, or they do something like that, they use web tools to do all that rather than doing it within--in-game. And then, of course, we have the traditional Open Source project teams, which are the coding team and the QA team. The key learning point for me at this point was that the content is much harder than the actual engine building. So typically, I guess most of you in this audience are probably coders. You know, I come from that background. I'm used to thinking the code is the hard part. But in this, I can write a feature in a night that will--maybe I'll build something one night, I'll build a feature to allow our combat so that your--the animations that were played in your combat depended on the level--the skill level you are using the weapon you had. And when I did that, that method now, I can link the skill level to the animation name and it was all kind of really elegant. It was like 20 lines of code to get that all going. And now that turns into 1,000 hours of animation time for the animation team to create dozens of new combat moves and to get all those rolled into the game. So that's where the cost of these things is and the effort and the hard part. You know, the two most obvious constraints that we have on this team are no money and no money. And so we run a dedicated hardware--a donated hardware. We're lucky that--because we have a relatively high profile, you know, I think the last time we needed another server, Luca sent a request out. He put a notice on the website saying, you know, we need a server. Can somebody donate us a dedicated server and all the bandwidth, et cetera. And I think in 24 hours, we had 22 responses of companies that were offering to provide that for us. So the amount of hardware hasn't been a problem. But even once we have it, we don't have a lot of advanced tools for deploying this stuff and managing on our mass basis. So for us, one server is already a lot of work to maintain. And to have dozens of them, you know, we don't have the tools to do that. And if we do that, I said they wouldn't be colocated either. So as we spread this around, we need to be thinking about a high-latency network. It's not all in one rack on a gigabit switch. No money also means it's a volunteer development team. Andrew and I are kind of long timers on that team. But a lot of people come and go. It's a revolving there. We have a lot of people that have done one patch and have disappeared. And some people are there for a year and then their--they graduate from college or they change jobs or whatever. And so there are situation changes. And so as a result, the learning curve issues as I touched on earlier are our big issue. And so we want to enable people to get comfortable with a certain area and then to branch out from there or to move on and somebody else takes over their area. Okay. So now I though at this point I would show you guys a YouTube video. For those of you who haven't scene PlaneShift just to get an idea of what it's like. So if we're flipping settings on video, now is the time to do it. And I'll maximize this. This takes about five minutes. Okay. Okay. I think the worst part of that video is that it makes the game look better than it actually is. That was--that was produced by a fan. We have a--there's a lot of fan-generated video stuff out there. And so I just kind of--when I was getting ready for this the other day, I just went out there and try to find one that look good and that's what I found. So the team can't really take credit for the--for the trailer. I think they did a nice job on that. Okay. So that's gives you--it should give you some flavor for what the game is like. You can see that it's a pretty ambitious project. And I think for a bunch of people working an hour a day and--here and there in their spare time when they're not sleeping. I do most of my coding after my wife goes to bed and--you know. So my day on this doesn't even start until usually 11:00 or 12:00 at night. So I think it's been--it's taken us years of years to do but we've come a long way with it. Okay. So now I'm going to dig into how this thing is actually designed and some of the key pieces of it that I think people here might find interesting. At a high level, you know, the server is basically doing nothing more than trying to handle thousands and thousands of events that are coming in. Events can be network messages. They can be time-triggered activities or they can be, you know, intra-server messages or sort of broadcast internally in the server. No distinction is made between these once they're in the system. So, a lot of times a network message will turn into the intra-server broadcast if it's of a certain type. The threading model, you know, is basically just one worker thread doing 90% of the work. And Andrew has got a whole bunch of thoughts on multi-core servers that we'll get to at some point. But right know that's the way it works. There's a pub/sub architecture who allows for what we call "managers" to then subscribe to these events and handle them. So add in new managers in or to specialize and only learn a certain manager is real easy to do. So the classes--these managers are basically defined along functional lines. So we have a spell manager. We have a GM command manager for the kind of police that are out there policing the players. Combat, chat, login, authentication, et cetera are all different managers in the system. And so it becomes--it's easy to find the code that you're looking for if you're trying to drill in on a specific feature. It's either fix it or enhance it or to find--to find a bug. That lowers the learning curve which means that the volunteers can get started sooner and, you know, make progress and feel like they're contributing something quicker. And it also means that it's easy to bolt in new functionality because you can add a new manager and subscribe to a new message type and then you're good to go, and the rest of the server doesn't really need to know about it. This diagram is a little bit old so there might be more managers--there might be a couple more managers at this point. But what you'll see here is that, you know, weather handles all the random rainfall, lighting, and snow, and the light and dark cycles of the day/night and so forth. The spawn manager is handling an NPC spawning, monster respawning. Those things die and we replenish throughout the map. Progression manager and NPC manager, exchange manager is handling item exchange and inventory management between the players. That's a big source of exploits and cheese by the players. So it's probably one of the most sensitive areas of our system. Entity management, I'm going to talk about it in a little bit. What else? You know, there are just a whole bunch of those. They go down the line. And so whenever we add a new functional area, we'll add a new manager. You'll also notice over on the right, the client has a very similar structure with what we call handlers where it's doing the same kind of subscription model either to network messages from the server--you know, kind of events signaled by the server or by commands issued by the player. Okay. So now I'm going to dig in to the network and entity. I'm going fast on this. I have a lot of slides and I'm kind of trying to get through them all. You know, we can spend an hour just talking about entity management or just talking about how we do our UDP implementation. But I'm trying to just skim over the top on a bunch of different subjects. And if we--if we want to drill on any of those, I'm happy to do that in the Q and A session. The base networking, it's all UDP-based. It's kind of a subset of TCP that we built because the TCP--because it guarantees ordering of packets than on one held-up packet. It can hold up every other packet behind it. And that's really not a good thing in a--in a game where you're trying to have real time synchronization across the clients. We have two priority levels of messages which are guaranteed and non-guaranteed. We need to have guaranteed ordering for some messages but not for all messages. And so I think at some point soon we'll be adding that on a pretty fine-grained level. We pack it all really tightly and we spend a lot of time on packing those network messages together because bandwidth really is our number one bottleneck in terms of how far we can scale a single server. A lot of people would think that it would be the CPU where they would think we'd run out of memory or something along those lines. But the game is efficient enough to where really bandwidth is our issue. The server is obviously--it's all running on one server today. So the entire world is all on one server. At some point we'll probably zone it out and partition it off. But we haven't had to do that yet. And so we haven't done it. An entity is something we define as any object that's added after the fact to the map. So that could be characters and monsters or it could be if somebody drops a book on the floor. Not that book is an entity. So the maps are loaded and then the entities are overlaid on top of that. Each client then gets a subset of those based on the relevance and proximity that they have to that. So there are maybe, let's say, 5,000 entities in the world that the server is managing but each client only sees their subset of those entities. And so this is my attempt--a simple diagram around that. So player F--so the server sees all of these entities in its model. But player F has a proximity list where he can only see himself and five other characters. Then player H can see himself and six other characters. And they only have that character G in common between them. So the other entities in terms of player F's client machine, any entity outside of that circle does not exist in his clan. That's really important for performance reasons and kind of network bandwidth reasons. It's also really important for cheat protection and kind of map exploits and visibility exploits. Relevance is first and foremost determined just by a threshold distance to the player. That threshold distance depends on the type of identity. So a player character or a monster could be 100 meters but a special gem stone could be one meter. And so you have to physically be within one meter of it before it exists on your client. That's another thing we do to try to prevent cheating. It has to be visible to the player. And by saying that I don't mean that it's visible on the screen to the player in terms of it's not included by a door or something. I mean that it's logically visible that if a player turns a certain way they could go see it. So this really has to do with GMs that are invisible or a player that's cast a spell of invisibility or something. If he's invisible, then other clients around him don't ever have that identity on their machines at all. So it's not hackable. And then within the same instance, the game also supports instance dungeons and instancing for various reasons. And so the instant for us is a fourth coordinate. So every entity in the system has an XYZ coordinate and then they have a W coordinate, which is their instance. And only players--players can only see other players that share the same W coordinate basically. >> Question. >> FULTON: Yes. >> [INDISTINCT] cheating. >> FULTON: Yes. >> [INDISTINCT] cheating. >> FULTON: Because it means that you have to be within a meter to see it. So if the--if the point of--if the game is to, like, find the hiding place of the gemstone, then you've got to get to within a meter. You can't just hack your client and get a print of all the entities that get propagated to your server, from the server down to you and then know the XYZ coordinates you need to go to until you're near it. >> [INDISTINCT] >> FULTON: Right. So the benefits. I mentioned the security benefits a little bit and I'm going to talk about that more in a little bit at the end. And also it's very important for minimizing bandwidth. We're sending a lot of position and velocity updates about all of these entities, you know, every second or every half second. And so the fewer entities we're sending that--the fewer client machines that we're sending that to, the less bandwidth we're using. So it makes it a much closer to kind of a linear scaling network bandwidth consumption as opposed to kind of an order and squared update model. And then it's very easy construct to use for the managers when they're broadcasting because then--like, if I'm chatting and I want to shout a message, I might shout that to everybody within 100 meters. Well, that's my--that's my entity list. And if I wanted to do everybody within 10 meters or two meters for a whisper or something, I can look down at the proximity list and look at the radius of each one of those and quickly and efficiently just broadcast to the people within that radius without looking at all 5,000 entities. So the cost of it is that the server does spend a lot time maintaining and updating the prox lists. And so there's a lot work on that. And pounding out all the bugs on that was a multi-year process I would say. Right, [INDISTINCT]? So this is what it actually looks like. It's a mess. You know, every entity has its own prox lists around it and they all are these overlapping lists. And they're all being maintained in real time as one updates or another updates. Okay. So now I'm going to shift over to scripting. We use several different types of home-made scripting languages in the tool. There are these mini--miniature languages for solving a particular purpose in the game. Generally, the benefits of these things are that they're possible for non-programmers to produce. And I'm going to show you examples that other teams are using. So the engine team will build a mini language and then we'll give that. And then the web guys will build an area of their content management tool that allows people to put those scripts under the tool and then the engine team gets out of it. So if they want 200 spells or 300 spells, we don't care. We don't have to code them all. They're just data. That also means once we put it into the database, it's proprietary content and so it keeps us from having a GPL of that data. So we have test data out there. So we have example scripts and we have a running server that spells that work but they're just demo spells to show you how the scripting language works and the they're kind of be test cases. But they're not the actual in-game scripts of the--of the--of the real game--official game. But what it does, it makes the engine team a lot more productive because we spend our time really at the meta level mostly adding new commands and capabilities to the scripting languages and then that becomes a whole new tool set for the other teams to use as they go forward. Did somebody talk or there's somebody out there? Is it you? >> [INDISTINCT] that you have the people writing scripts, worked around bugs in the underlying engine instead of reporting them? >> FULTON: Yes, yes. They do or they just don't think that it's possible. And so they don't think we ever intended it for it to be there when actually it's just a bug. And so--then they'll ask for a feature that they want. They're like, "Is it possible that we can do this?" And we'll say, "Yes, that's been in there for two years." And they're like, "Well, we tried it and it just doesn't work. We didn't think you did it." So there are some of that that goes on. And so, periodically, we try to go through and update the documentation and clean up those scripting languages. So, yes, that's a--that's an interesting problem. But I really like that meta programming stuff. We're building the tools for the other people to be productive on it. And that's a much kind of higher multiplier activity for us to be involved in. So, the first one of these that I was going to talk about was math script. This is basically what we use for the RPG rules or anywhere where there's a calculation involved. So if I am a character and I swing a sword and hit another character, what is the resulting damage of that hit? Well, there's some sort of a randomization factor. It's based on my skill or their skill, their armor, my weapon, a lot of different function, you know, features about that particular situation that leads to that result. That's all done within Math Script. So I have a couple of here that I just threw out there just to show you an illustration of it. And these are just from our subversion examples. But the idea is that you can--there's lay binding to the objects. Any object in the game as long as it implements those interface called iScriptable can be linked in to math script. I mean it can get at it's attributes with the [INDISTINCT] operator there. You can also call functions on those objects. So a worker right here has a function called get skill value that's implemented by iScriptable. And so those can be added to the engine, too. So the people that are in the non-technical ones or the none C++ people are writing this stuff and using engine features without really knowing that they're doing that. And if they need a feature, we can add it in without changing the underlying scripting language too much and get there. So it's precompiled to p-code on server startup. All of these are loaded in at the beginning. It takes the server about a minute to two minutes to startup whenever it starts because it's loading in absolutely everything and caching absolutely everything. Dynamic binding, I just mentioned. And as long as they implement iScriptable, we can bind those objects at the time that--you know, right before the math script is called. Another example is what we call progression scripts. And this is where--this is kind of how spells are done or other things like that where you want to describe the effect that a certain event has on a player or on a couple--on a set of players. And so in this case, damage--you know, there's an operator here. We use XML just because it's easy to parse. So the hit points go down by this. This value here is any math script--any single line math script. So that can be--I just put a negative sign in front of a variable there but you can also--in some cases, this is a long complicated formula. And so depending on how crazy the rules guys want with their spread sheets, you know, that's what they end up putting in there. And then they can do some messaging where system messages show up on the screen and, you know, notify players of things that have happened as well from within the scripting languages. These are compiled down to virtual function calls. You know, each of these objects here--each of these tags turns into an object and then the script runs by just calling the same virtual function across each one. And so it's again very efficient way to run the scripts. And they're preloaded and parsed at the time on startup. And then the value actually can be any single math script. Okay. This is the one that--that's near and dear to my heart because it's the least scriptish. What this is a script for how a quest is done in the game. And quest for us is a generic term that means any kind of task that an NPC would give to a player to help them get a reward or progression points. They're not all grand quests like in a novel or something. But what we found was with the XML structure--we have a similar XML structure for dialogue and a kind of action response scripting with the our NPCs. There was XML base. It looks a lot like this production script. But it was too technical for our poets. And so what we did instead was after two years of struggling with them trying to use that XML structure was we went to this. And this is designed to look like a screenplay. So it's much more like writing a dialogue back and forth. And they can kind of just free-associate and think of how they want to do it. And then with a little bit of syntactic sugar to give some hints to the engine about how to pull this all apart. We can--we can make it comprehensible to them. So you'll see that P up here stands for player. So the player initiates this quest by saying greetings. And then there's a whole [INDISTINCT] architecture around. There are a million things they can say that all equate to greetings behind the scenes. Then we have this character named Marth who responds back to that and says, "Hail," you know, "would you like to earn a little money?" The player then can say either no or yes. I can get my mouse going. So they can say no or yes. And then Marth's response are down here. And so if he says no, the first response correlates to the first player statement there and the second response correlates to the second player statement. And only the second one progresses the script. If he--if he does the first one, it says, "Well, okay. Then have a good day," and the script stops right there. And so this to our writers is understandable. This is how they have to do it. So the positive thing that advances the quest forward is always at the end and then that chains on to later in the--in the quest. There are other things. I try to put in kind of a simple example here. But there are lots of prerequisite scripting that only people of certain levels can get certain quest or if they have a certain item and so forth. You can also link these scripts together and chain them and make subroutines out of them and stuff. So--but that's all done in a sneaky way to where the settings people don't realize that they're coding subroutines, but that's what they're really doing. This is all done essentially through a preprocessor that loads in these big text blobs and kind of parses them out line by line. It creates the XML that I tried for two years to get them to use, and then it creates that from this, and then it runs that through the XML loader and it's running that through the rest of the game. Okay. So that's an overview of three of the mini-scripting things that we have. We have some others on the client as well. For effects, you saw the sparkly things going on and lots of different spell visual things. That's all done through XML structures, too, that our art team uses. So anyway there are lots of those. Let's talk about NPCs for a minute because this is the other topic that is near and dear to my heart. The basic fact of the matter is that NPC AI is hard to scale. You know, it's very CPU intensive to do true AI and to really make things intelligent. And in a world where we're trying to have one server who lasts as long as possible and to potentially support thousands of players, to have thousands of NPCs all managed by one process on one machine would really limit us in terms of what we could do with it. But today in our monolithic server, you know, that 100 players take us less than five percent CPU on that machine. And so that means on 30, we could scale that to 2,000 players on that machine. But at that point, we're using something, like, 20 megabits of bandwidth, you know, to support those players. And so that's why bandwidth is our big issue rather than CPU. But AI is computationally intensive. The smarter the monsters are, the more expensive it is to make them that smart. And the AI techniques are constantly evolving. And I wanted to put together an architecture that would allow new AI experts to show up. You know, my dream idea is that Andrew is going to be an AI expert soon. And he's going to come back and he's going to want to make the world's smartest dragon and he's going to have an eight-way box that is dedicated to make one dragon incredibly smart, okay? And so I want to be able to support that rather than to say, "No. You've exceeded your one percent budget for that dragon. That's all you can do." So what we do is we pull the AI out into its own process. And so that's what these NPC clients here are. They have their own process. They connect through the same networking layer that our clients do. And so they can attach and we can have multiple of those attached. And so the NPC client is kind of our default implementation that we ship with is very efficient and very basic NPC client. And I'm going to describe to you how we do that. But if somebody were to create the world's smartest dragon, as long as they implement the protocol that we already use for NPC client, they can snap in there and that whole server farm can be dedicated to one NPC. You're going to ask a question? >> [INDISTINCT] >> FULTON: I can barely hear you. What? >> Can you give me an example of [INDISTINCT] or what--what's the scenario where a [INDISTINCT]? >> FULTON: Well, I mean--and the--and the most basic case would be things like path finding. In a big world, if you train a monster and you chase him over three maps away, you know, and then he needs to finally get back to his home area, that would be a very expensive calculation and I don't want to hold up the server on that. It could also be that when--you know, in the high-end games, in the commercial games it's not unusual to have, you know, kind of 60 players all attacking one monster. And so in those mass raid types of environments you could imagine that you would have a very smart set of monsters that we're teaming up and, you know, kind of working as a wolf pact to attack from different sides and to optimize how they were going to do it. And so I don't want--you know, I don't know what all those scenarios might be but I want to be sure that our framework supports that if somebody thinks of something cool to do we can do it. Okay. So the key clients--in what I call the super client is that the NPC client is our default super client. But it handles today--I think it handles about 300 NPCs. And so it is one process acting like a client on behalf of 300 entities. And that's why I call it a super client because most of our regular clients are handling one entity which the player and just respond in the keyboard or what have you. It connects to the server of the network. It uses the same protocol from most things. So things like login and authentication and [INDISTINCT] propagation are all used in the same protocol. But it has a special messaging layer for bulk activity. So if you're managing 300 monsters, you're probably--like right now, it's handling all the monsters in the world. So it actually needs to know about all of the entities in the world. And it's a waste of my time and a waste of the server's time to have 300 independent proximity lists for those 300 entities when at the end of the day it's all one big activity. So there's a lot of bulk updating that's going on between the NPC clients that isn't happening with the player clients. The NPC manager class on the server is yet another manager. And it handles all of the NPC-specific protocols. And so what it does, it packs up and unpacks the things that it needs to and then the bulk updates. It unrolls those individual updates and then republishes those internally into the server. So to the rest of the server, a movement update of a player, or of a monster, or an identical in the server. And so the fact that the bulk update was done at the beginning, it doesn't matter. So, yes. So the emulation layer means that there's no distinction between player--human players and automated players almost anywhere in the game. And on the client machine there is no distinction between a monster and a remote player. So you know about yourself and that's special. But if there are five players here and five monsters on the other side, you know, to you you're getting the same network updates and that you have the same information, same stat, same everything about all of those. From a scalability standpoint, you know, the next place when we hit a bandwidth limit or a scalability limit on here, we'll pull NPC client off into its own machine and then that will connect remotely. And now instead of two really heavy processes of our game server and our NPC super client both running on the same box, as soon as we pull those apart, now we just kind of doubled our capacity without really doubling our complexity. The other thing we can do obviously is to pull [INDISTINCT] database off on its own server. So I think as we start to hit scaling problems--if we ever do--that will be the most obvious thing to do before we ever go to zoning or other things like that. So then we've already kind of talked about no limitations on AI techniques as long as we use the same net protocol. And again because it's Open Source, we can change the net protocol, too, if we need to. So, how we do NPC AI? What is the default superclient doing? Okay. And this is what I like to call "artificial stupidity" rather than artificial intelligence. My big example is Pac-Man where the ghost in Pac-Man has probably no intelligence to it. And if you think about the CPU cycle budget that the guys had [INDISTINCT] Pac-Man, you know that that thing is doing probably nothing. And, you know, that's still a pretty challenging game for most people. And so for the average case in the--you know, the most common used case of these things, the mobs don't have to be very intelligent to be fun to play against. So we really emphasize CPU efficiency and flexibility of the scripting because we didn't know what we would want. So we have a--yes. So, our superclient today controls about 300 mobile NPCs and takes generally less than 10 percent of the CPU on the machine. So when--it's not unusual the when we're--we'll--I'll look at the top on the server at a--maybe a slow time and top will be the busiest command on our--the busiest demon running on our machine compared to the server of the NPC client. So, how we do it? The first thing we do is the scripting language for the NPC--there's another one of these many scripting languages and it supports this concept of NPC types, which are basically mapped to kind of C++ classes. So they define the overall methods and behaviors of that--of that particular class of monsters. Each NPC then is assigned to that type and gets all of those capabilities. So a type consists of a set of behaviors. So walking around randomly, attacking, chasing, fighting; those are different behaviors and the monster will switch between those behaviors depending on what else is happening. The way that they switch is through reactions to perceptions, which is that second bullet there. Perceptions are either sent by the game server. Hey, somebody just hit you with a sword. Or there are perceptions generated internally like, hey, that monster is out of range or that player is out of range you need to chase him, that kind of stuff. There's an inheritance structure for the NPC types as well. So you can define a bunch of behaviors and then you can have a special monster like the boss who has all of the original behaviors plus two other behaviors real easily in the structure. The loose structure does give us a lot of flexibility and power, but I will tell you it's very hard to debug because it's hard to tell a bug in the script from a bug in the--in the client--in the superclient. So then the NPCs themselves are kind of like instances of those classes now. So, they're assigned to a certain NPC type and then they maintain their--each one maintains its own individual priority queue of the behaviors that were listed for that NPC type and then executes the behavior that is the highest priority. That priority we call the need. So the NPC doesn't have a lot of needs but then when--you know, if it--it doesn't have a lot of need to fight. It's just wandering around. It's happy. When you attack it, that--the reaction is to boost up the need to fight. The need to fight takes precedence over the need to wander around and then it starts attacking--it starts fighting back. So that's how the behaviors run. The other thing that the NPCs maintain is what we call hate list and that's the list of all the players that that monster knows about and kind of is a way for it to determine who does it hate the most and who is it going to fight. So if it has five people pounding on it, which one is it going to attack? And so that's another sort of little priority queue in the priority list in there that it uses. So the reactions only affect needs and hate and then everything else is done by just running the highest need activity. So this is an example of what NPC type looks like. In this example, it's called fighter. There's a "do nothing", which is just to stand there. And so that could loop around and just play the animation that stands and breathes and kind of sways from side to side and that's about all it does. It also has one for turning that I don't think is used by the script. Then fighting finds the target, turns to face the target and then melees with the target. And it has a range there that it has to be within three meters of that target or else it's going to be out of range. And then I have a chase one there, too, that chases the target if the target's running around. So if you look below you'll see in the--the third reaction down there. If the--if it gets an events from the server or perception that's called--that's called attack, then it knows that it's been attacked. And so it's going to override whatever behavior it currently has and immediately implement the fight behavior. And so, fight then is going to turn to face the--it's going to locate the target and turn to face it and then melee it. Damage is the fourth one down there. It adds to the fight, but then the weight factor here means that the hate list is getting updated proportionally to the amount of damage that was being done by that particular--the person who's attacking it. We have more sophisticated things in mind for that where some mobs may hate the use of magic against them more than they hate, you know, melee attacks or they may hate slashing damage more than they hate blunt damage or something like that. So we'll just have to see how that goes. But then you'll notice the second to last reaction there. So it's target out of range and it switches to the chase behavior. So, you know, it stops fighting and it starts chasing. And then when it gets back in range it starts--you know, it automatically--the need to chase drops down when it catches the guy and then fight now again becomes the highest priority one and it falls back to fighting. And then death basically tells it to stop fighting after--you know, when the player dies, it's time to stop fighting. You don't want to sit there and keep hacking. Okay. How am I doing with time? Okay. Game Security in Open Source is almost an oxymoron because the whole idea is that we're giving people the keys of the kingdom. They can build their own executables and connect to our server with them. And so it's a never-ending kind of battle and a thought process that we're always in to see what are the cheats that are possible by adding a certain feature and then do we care about those treats or not. So I put up these pictures to show you that modifying textures is a favorite pass time of the fans. And so somebody made a Simpsons mod for the game. And they have zip files up there and you can download this and overlay it on your PlaneShift install and make all your characters look like people out of The Simpsons. So, anyway. Okay. So, hacked clients are trivial. And I listed out a bunch of different ways that the stuff can be hacked. So, speed hacks. You know, everybody--anybody who's played one of these games knows that it is a giant pain. You have to run from one end of a zone to the other. And so you could have a 20-minute run to get somewhere if you--if you don't have the right teleport spell or what have you. So people will just hack their client and now they can run 10 times as fast if they press another button. Well, we need to detect that and do something about that. Same with teleportation and anti-gravity, things like that. Those are all kind of fall under the movement hack thing. Visibility mods are another one. We had a--in the case of the one meter gemstone. We had a guy who modified his client and made the gemstone 1,000 times as big. So when it was propagated it was the size of a city and he could--he could see it immediately and that's why we actually implemented that one meter range so that he couldn't see us just by modifying his mesh. You know, just the 3d model on his--on his desktop though. It was--there would be no way for us to detect that. So we have to make it inherently secure from things that we can't detect. Bots and macroing are another big problem for things--this normally applies with the trades more so than with things like combat where we have smithing, we have mining, we have wood carving and things like that. These are very repetitive activities where you're sitting there and you're just trying to get ore so that you can, you know, smelt it down and make gold bars or what have you. And so there somebody can write a macro and then, you know, dig and then move a meter over, and then dig again, and move a meter over, and make a big box and just keep doing it. It turns out to be really hard to detect. So--and we put in some to detect it and then they get around their detection because they can download our code and see what we're doing to detect it. Okay. So collision detection hacks are another one; running through walls. You know, we try to set up a maze and they just run all the way through to the end by turning off collision detection. Map hacks are another one. Don't remove the collision detection. Just take out all the walls and run to the target in the maze, okay? So there are things like that that, you know, we have a problem with that most commercial games don't. Item art hacks. I kind of talked about that with the gemstones. And then texture mods was kind of that Simpsons thing that people do for fun just to--well I've seen the Simpsons one. My personal favorite is ninja shift where they made everybody look like ninjas. And so there are bunch of those out there. So I kind of group these into three basic areas. There's kind of low impact things that there's really no defense against. Texture mods are really the first one--or kind of the main one there. But there's also really not a lot of impact to that. People do it. They think it's fun. It's harmless. It's not messing up anybody else's gameplay. It doesn't make anybody look like The Simpsons on their machine. It's only on the machine that has The Simpsons mod installed on it. So they're really kind of only affecting themselves. It's not that big of a deal. The high impact ones that are easy to defend are things like visibility mods. And this is where our proximity list come into play where we really use those to where the client simply doesn't have the information to ever show it or display it in a way that's--will give the player an advantage. And then the speed and teleport ones. Well, we can have the server not just blindly accept any dead reckoning update from the--from the clients. They can actually compare it before and after positions and make sure that that's authorized. For speed and teleporting, you can kind of just do a distance and a time interval and [INDISTINCT] and it's really straightforward. There are some--because of latency on the network and kind of jitter in that latency, in the UDP. It's harder than it sounds to be perfect but you can make it a lot better to where it's not just open season on whatever you want. Okay. So, then bots and macroing I was describing. These are the high impact ones that are hard to defend against. And these are the ones that Andrew and I have spent months and months on with very little success of [INDISTINCT]. It's extremely hard to detect automatically because, again, the code that we're using to detect it is in the--is out there for everybody to see. So smart players download that and see how we're doing it and modify their macros to evade our detection. So what we're trying to do is we're trying to focus on the law of diminishing returns so that if you, you know, mine for gold or dig for gold in the same--you know, over and over and over, maybe that's valuable the first 50 times you do it. But if you do it 1,000 times, it's not 20 times more valuable and if you do it 50 times. Eventually you'll get to where you just never find it. And so by doing that--an average human player isn't going to have the endurance to do that. They need to go often and do other activities and then come back. And that's how we're going to basically try to make botting harder to do and not worry about whether we detect it or not. We'll just not make it valuable. So we're trying to reduce the impact rather than improve the detection. Yes. >> [INDISTINCT] the problem with [INDISTINCT] like watching different repetitive tasks or... >> FULTON: Yes, yes. Absolutely. Yes. So, they could dig for gold and then they could farm and then they could go back to dig for gold. Yes. So, I mean, it's a big problem. So, the other--the other thing that we do is we try to engender a player culture where they basically read each other out. And the game masters can be alerted that somebody's botting and then the game masters will kick those people out or ban them from the game. Yes. >> [INDISTINCT] macro and then--so that everybody [INDISTINCT]. >> FULTON: Could we give them--you mean could we do that? Is that what you're asking? >> Well, yes. Instead of trying to [INDISTINCT] part of the game. >> Yes, we could do that. Star Wars Galaxy did a lot of that where you could set up automated--kind of repetitive activities and then log off. And when you log back in, you have 1,000 gold pieces from your repetitive activity. Yes. I mean those are all options and I think they kind of have to just be on a case by case basis as we figure out what people are tempted to do as to how we want to address it. But if you want to have any kind of economy in the game, you can't make everything easily automatable. Yes. >> [INDISTINCT] id you let them do that, you know. [INDISTINCT] tags along and, you know, killed [INDISTINCT]. >> FULTON: Yes. >> That's going to be a bigger [INDISTINCT] than anything else. >> FULTON: Yes. PVP in our games has been a big controversy from the beginning because some people really like PVP and they want kind of free-for-all. They want to kill anybody and gank anybody that they want at any time and they think that's the most realistic way to structure the world. Other people really are kind of the care bears out there and they don't like any of that at all. And so what we have in PlaneShift is we basically have an opt-in system to free-for-all PVP. And so we have a system called duels where players can challenge each other to duels. And then if you don't have kind of auto accept turned on, then a box will pop up of "Do you want to accept the duel or not?" If you--and then you can kind of set that to either pop up a box, auto decline, or auto accept. And so the people that are out there that like free-for-all PVPing set their thing to auto accept and they just attack each other. Yes. Andrew, is it time? I think I'm done. So the--just my last slide here. The website is planeshift.it. Luca is in Italy. He lives Terni. And so he started the project and this is his website so everything ends in .it. He's reachable there. And then my email address is there as well if you want to follow-up. And if you any of you would like to contribute to the game, we'd love to have you in--on any of those teams depending on what you're interested in contributing. >> Does anyone have a question? >> Could you explain a bit more about [INDISTINCT] close source media and why you decide to go down that route? >> FULTON: Sure. >> Because it's been one of the major factors why I personally haven't contributed to PlaneShift, contributed to some of the other [INDISTINCT] MMORPGs because I can't then reuse the [INDISTINCT] like that that's there in my own type of stuff. >> FULTON: Right. Yes. You know, there are a lot of schools of--you know, a couple different schools of thought on it and people who feel strongly about it and I've had this argument online with people a million times. But, I mean, it basically comes down to that our philosophy is that we want PlaneShift to be PlaneShift and we want the artist who are contributing to the game to know that they're contributing to our game and not to somebody else's game that is going to misuse their characters or misuse their art in a way that they don't want it used. And so I think the perception of a--some people on the team just don't care but the--but a lot of the perception is that the people out there who are rabidly Open Source and want--not only the code to be Open Source but they also want the content and all the music and everything to be reusable is basically they want--they basically want that to benefit themselves and not to benefit the rest of the community. They just want to be able to play with it. And so we're trying to keep the game to have as much integrity as possible and to kind of have its own identity. The engine's out there and it's reusable. And it's actually been reused by at least one other project or maybe two that have used the--to use the engine with their own content to create entirely different concepts from even the MMO concept. So it's been really fascinating watching them do that and we totally support that. But using--reusing the artwork and reusing the models and the music and all other stuff, I mean it's--I think the current download is somewhere around 330 meg to install the game and so there's an awful lot of RS that's out there that if somebody were to take those and use them in a million different things, then all of a sudden PlaneShift wouldn't be as unique and as special anymore as it is today. And that's something that our artists like about contributing to the game as well. Anybody else? Okay. Thanks a lot guys.