1. We are here at Build Stuff in Vilnius in 2014. I am sitting here with Yan. Yan, who are you?
I am Yan Cui. It is spelled with CUI, but it is Chinese so you pronounce it [tᶴuei]. I work for a company called Gamesys. We are based in central London, right next to Picadilly Circus and so it is a really nice location. We are a games company – Gamesys itself is a market leader in the real money gaming business, but my team focuses on a different audience: we make social games for Facebook and mobile. I work as a server side developer there for almost five years now and I am responsible for building the back end game service infrastructure for a number of our social games up to now. Depending on the game, there have been different challenges and we have had to architecture our infrastructure differently depending on the requirement of a particular game and along the way we have had to evaluate and adopt and use a number of different technologies both from a language perspective as well as from an infrastructure and database perspective as well.
2. Let’s start off with languages, maybe. That is always fun. What languages do you use?
On a server side for our social games, we have an application and components being developed in different languages, including C#, F#, Erlang and a bunch of guys are working on Python as well. Everything we do is cloud hosted. Most of the game server, the core game’s logic are hosted in the Amazon web services, developed in .NET. Then we have a bunch of cross application back office components for payments, processing and retention campaigns – those are being done in Python and hosted in Google App Engine. We started out with pretty much all C#, but I became interested in F# starting in 2010 and I have been spending a lot of time with F# in my spare time and then we came to a point where we were building a social slot game and we needed to build our social slot engine, which is ultimately a rule engine and I thought that seemed like a good fit for the functional paradigm and I said “Let’s try it in F#”. So, from there I built some prototype, demoed it to my boss and I illustrated how we can have similar capabilities to an existing engine that has been produced by the Java teams on the real money side of the business, but with a lot more productivity. I was able to build a simple game inside of a 2-4 hour sprint on my own whereas it takes sometimes up to 3-5 days for a new game to be developed by some of our Java developers, partially because they have some legacy concerns, but also partially because the problem that we are trying to obviously solve is a better match, better solved with functional languages like F#.
Werner: So, how is F# better at these problems, at gaming. I mean I would think that interactive gaming would need imperative code.
For that particular engine, a lot of it is just rules, rules that you can specify ahead of time using math models and our job was to ultimately implement some of the math model and then taking into account the request response formats and stuff like that, that is required by the actual games themselves. So, the UI itself is not developed by us. They are done in the Objective-C and ActionScript depending on whether it is for the web or iOS platform. But for the actual back end itself, it is very much a rule engine, you have input based on some rules, you have deterministic outputs and, by using say C# or Java and using the OO paradigm, also you are just spending time working on the plumbing where with the functional paradigm, in F#, you spend much more time thinking about the problem that it is trying to solve and writing code that has much higher signal to noise ratio and you are spending much more of your time just tackling the core problem. So you end up writing fewer lines of code and the stuff that you write is actually more concise to your problem domain and easier to read and maintain, as well.
3. So F#, as a functional language, is more suitable to data processing. Is that one aspect?
I think F# is suitable for very general purpose programming. It is functional first, but you can also do OO and imperative style as well if it is required. I think one thing that F# is really, really good for is building, say, DSLs. We have worked with Amazon services very extensively, we use something like 30 different services that they provide and sometimes they find complexities with those APIs because of the way the APIs – the APIs themselves may be simple, but the way you interact with them in order to build reliable applications for your particular domain, sometimes that introduced some impedance mismatch and using the API itself becomes a burden and even though you leverage off some of the work that the guys at Amazon had done, but it is complexity that you do not want. So, we built a number of libraries –one library that we built for working with DynamoDB.
DynamoDB is a great database. It is a key-value store with some query capabilities and it can scale to almost as much as you want. The way it works is rather than the pay-as-you-go model that Amazon employed for most of their services, with DynamoDB is that we specified that for this particular table I need to be able to support x number of reads per second, x number of writes per second and Amazon goes away, reserves enough capacity resources to handle the throughput that you want, but in term of query ability the raw construct and abstraction that you are working with is really low level and when you are building say layers so that you can do all your circuit breaker, timeout patterns and things like that in one abstraction layer so that the game developers themselves do not have to be tied to the abstraction as exposed through Amazon SDK you can’t really do that because in order to do queries, you would have to use the raw abstraction form of the SDK. So, we built a SQL like DSL so that you can select attribute names that we want from a table with where expressions, so that we can then allow our game developers to almost think about doing queries as if they are writing SQL. We are also writing a number of other libraries to help us deal with some of these complexities that come up with Amazon web services. One more thing, one more place we are using F# for: it has support for the actor model through this type called MailboxProcessor service, called agents.
So, for one of our games where the player state can be as big as 4-5 megabytes. So, in using the stateless model, it just won’t scale because every call requires loading 4-5 megabytes from data store, retrieving it and then saving 4-5 megabytes back to your No-SQL database. So, we decided to move to a stateful model and with that move we decided to look at actor model because it seemed like a good fit and I have been spending a lot of time with Erlang by that time, in F# you had the implementation of that through agents and ultimately we built an agent-based framework on the server side that we can do a stateful middle tier for our game servers so that we are able to scale – I think that at the end we got something like 500% increase in performance in terms of number of players that a server can handle and something like 60-70% reduction in average latency as well. So, that was to prove to be a big reinforcement to using F#.
That is right. For one of our games, it is a MMORPG called “Here Be Monsters”. For this particular game which is a location-based game where you have a virtual world with 500 different places you can visit and monster trapping is a big part of the game so, for instance, if you want to catch a ruby dragon, he is only available, he only exists in certain parts of the world and to catch him, you need to use a trap that's strong enough to hold him and you need to use a bait that he actually likes. In that game we also have NPCs, quests and most of the RPGs game elements that you find now in other games as well. One thing that we struggled with initially is to understand the impact of change because everything is so inter-connected.
By changing, say, the price of water, you now have to change the price of everything that is made from water. We have something like over 4,000 items in the game and about 1,000 recipes which combine different items to form other items which means that the prices of some of these other item that are crafted – they need to be tied to the things that they are made from. So, if you change the price of water, everything that is made from water needs to have their price adjusted so everything else that is made from those other items and some of those items are used in other things as well - for instance, they are used to catch a monster; so, if the cost of the bait for a monster is changed, it goes up or down, that means that the reward that you get for catching the monster also needs to be adjusted. We have so much content, that it became impossible for any one person to understand the impact of any change or any addition to our data.
So, we built a system as well as a math model to govern the economy so that once the game designer is working on data, in isolation they test it and they can put the data into Neo4J and they are using the math model and the ability to query data very efficiently and easily. In Neo4j we are able to then automate some of this balancing process that we have to do to make sure that when you change something, all the things that need to also change are identified straight away and we use the model and we recalculate the recommended buy and sell prices and if it is something that your character can eat, how much energy and we can also use all this information – the price of an item can give you an indication of how difficult it is to obtain that item. Then you can use that information to then feed into your quest line – early quests should not require expensive items and also they should not give you expensive items as rewards. So, that is one example where we are using Neo4j to help us understand and manage the complexity of the data that we have in the game. Other companies have tried to do something similar in terms of game balancing, but they tend to just throw people at it. I know one RPG who is really big and it has been running for over a decade.
They have a team of 100 QA people, they would test new releases for weeks, sometimes even months and relying on people to judge from their subjective view how balanced the game is and if particular elements feel wrong and if there are any loopholes in the economy and you can, say, buy something for cheaper than if you have to make it. So, we had one QA member for web and one for iPad and they focus mostly on testing the interactive side of things and the client. So, we did not have the manpower and overall we had quite a small team behind the project anyway. So we wanted to find a better way, more systematic and a sustainable way to do this game balancing and that is where Neo4j comes in and that is where this whole process helps us do that.
5. You get the main value out of modeling the logic or your ecosystem, your economy basically, right?
Yes. We get benefit out of modeling all the relationships that exists between places, quests, NPCs, items, recipes. Before, we could not even answer simple questions such as “What are all the different ways you can obtain this item in the game?” because there is no one person that understands all the quests, all the places. We just could not answer that question easily, but with Neo4j it is just one simple Cypher query and there you go, there is your answer. We actually built some tools for our customer support team so that they can easily answer some of these queries that we get from players using data that we already have in Neo4j.
Werner: So, what kind of data size do you get in Neo4j. I guess it is not massive.
No, because we are using Neo4j for data analysis only. It is not used to hold any player state in production. One of the downsides to Neo4j is that you are bound to one box in particular, the demand memory that you have on the that particular box as well so you do need a high end spec machine to deal with a large data set with a lot of nodes and a lot of edges and it does not scale out your writes across different machines, even though you could do replicates, but it offers flexibility that other horizontally scalable solutions do not. So Google, Microsoft both have their in-house graph databases to which are horizontally scalable and I think there is another service that is called Titan – they also have a horizontally scalable graph database, but they place limits on some of the things that you could do, such as how many layers, how many levels or paths you can traverse, whereas with Neo4j you do not have that limit, but then you are limited to how much you can do in one particular box. So for instance if an item – water – is used to make beer and then through some relationships of the beer, it is used to make another item which is used to make another item which is used to catch a monster who drops a loot, you can then use this to craft another item which you can use to trap another monster who drops another loot. So, there are these 10-15 hops between this item and the loot that you are looking for and you can find that relationship really easily in Neo4j whereas in the case of other solutions you may not be able to, because of the limit of the number of hops you can jump.
Werner: That is a very interesting use of graph databases. So graph databases are useful.
They are. They are tremendously powerful and useful and to be fair, a lot of things that we are doing now for analysis for economic balancing – people in finances are doing that already, for sure some guys at Credit Suisse are using Neo4j to do impact analysis the same way we are using it for derivative pricing, because in finances, in derivative pricing specifically, you have the same problem: if something changes, there are a lot of options that are tied to that change, to that one thing. How do you easily identify all those and using some model, the price changes to those things and the things that those things depend on. So, the guys at Credit Suisse are doing very similar things to what we are doing except they are dealing with real economy whereas we are dealing with a virtual one.
Werner: Well, I think our audience has got the taste for graph databases now. Thank you, Yan.
Thank you.