Yes. Hi, my name is Jesper Richter-Reichhelm and I come from Berlin, Germany where I work for Wooga as Head of Engineering and we do mobile games and it’s cool because I am a gamer at heart and I can do software and gaming in one job.
2. That sounds actually wonderful, you are getting paid for playing games?
Yes, sometimes. Playing games is part of that, you need to know what your competitors are doing, what you can learn from them and most of the errors to avoid, but of course it’s a bit more than just playing games.
Ralph: Yes, I was afraid of that. How hard is it to stay in this business? Because we know a few brands out there and it seems that they are not very much out there.
It’s though. We started in 2009, went to Facebook. Facebook changed a lot, so in 2011 we went over to mobile, so basically now we are mobile-only or mobile-first. The competition on the app store market is really, really tough, you have something like 1,000 games launched every week and maybe 10 of them are profitable and one is a hit. We are trying to make hits so that we can also cover the games that are not successful. It’s part of the deal, but overall the competition is very fierce.
It’s basically our way of coping with the fact that it’s very hard to produce a hit. So we have something like 20 games running in parallel and we run them through something that we call a hit filter. So, basically a team starts very small, one person does some concepting, writes down papers, looks at the market, then maybe an engineer joins for some prototyping. That phase can go two months, six months, maybe even a year if necessary. They go through many cycles and throw away ideas all the time, do user tests all the time, always asking ourselves is this game becoming a hit or not. And the moment they start doubting that, they stop and start over. Later on we go to production, the teams get bigger, the biggest team right now we have in-house is something like 25 people and even before we go for launch, we first do soft launch in one country only.
Very often this is Canada or Australia because it’s English speaking. And then we see also in combination to our own gut feeling, we see hard numbers and even at these late stages we very often stop games. Maybe half of the games we put to soft launch we put to stop before we commit to a huge launch budget and launch them. So it’s all about making it normal and without lasting penalty for the teams to cancel their own progress, to start over, so you are not fired if your game stops, you might get an increase in salary if you did a good job and because you did a good decision. But since we have something like 20+ games running in parallel, a few of them make it through, they create a hit and they finance all the rest. So basically this is our risk mitigation strategy: If you have many games running in parallel, the moment we think this will be a lame duck and not successful, we stop the stuff to not invest resources so that we can instead concentrate on the few hits that we have found. And that works quite fine, so it’s all about repeatedly making hits to finance the whole company.
Ralph: That sounds promising, I guess.
It’s hard to do in reality.
Ralph: I think so. Do you have problems actually with stopping games? Because I think every game is somebody’s baby and it’s a hard decision, I guess.
Yes, it’s a very hard decision. The first thing is that no one need to be afraid if they stop the game and that’s a big change. If you look at three years ago, everyone was afraid “my game is going to be canceled, do I lose my job now?” So, that’s a huge mentality change. The second thing is that the final decision always comes from the product lead if the game is canceled. But what you do is that you setup – this is similar with what Pixar has, for example – a review meeting, every milestone or every four to six weeks. People sit together, product lead, head of studio, other product leads, the CEO, and they discuss this. And the first thing they ask themselves is “one, two, three, do we think this will become a hit, yes or no?” And that’s the starting point of the discussion, whether they should continue or not. And if they decide to continue, they set up goals for the next milestone and they will then refer to that. If they have the good trajectory and they still maintain the good feeling, then this will become a hit game.
4. But it’s all based on gut feelings?
It’s all based on gut feelings plus a few things. One thing that we regularly do is usability testing. We get some users from outside who have never played the game before, give them some money, and let them play the games. We also started using a service called PlaytestCloud when the game is a bit more polished and we could send it to users in the US and they would be filmed while they playing the game, maybe over multiple sessions over an hour and then we would analyze these videos. And this is really nice because when no one is in the room there, they are really honest and really blunt and you see where you need to tune and where you need to get better. So this helps. And then we have this test launch and we see hard metrics on retention and monetization and things like that.
Ralph: This is really aggressive agile because you see the reaction of the true customer and you just cancel your project if it’s not the way you think.
Yes, agile is part of it, and lean startup is part of it. It’s also quite hard for engineering folks because what we found out is that the make or break of a game is always the game design. It’s always the product, so we’ve had some technical problems, but we never ruin the game due to technical problems. That makes it easy for me to talk about these problems, because it’s never the ……………. factor. So everything that we do in development need to aim at how can we assist product in making a decision or the game design to come to a decision. So that leads to this extensive prototyping period and at the end of a prototyping period that can take a year if it’s really long we throw away the whole code base and start from scratch because then we shift gear and we start production of the code which is then the usual test driven approach as much as we can. So it has a lot of repercussions this aiming everyone in the company in getting the right product and then it’s mostly game design and game mechanics.
It’s minimized. We might go into a soft launch, see that the numbers are really bad and then bad luck, that can happen. Actually harder are the cases where the numbers are somewhat good. It’s actually good, maybe 50% retention, yeah it’s good, numbers look good, but it’s not really great. But then you have numbers to stop the game. So, I think we prevent complete failure by stopping games early based on gut feeling. In doubt we rather stop than continue with the game. And later on when we have numbers, then it’s actually reasonably secure that it’s not a complete failure anymore. It can still fail, but not totally.
Ralph: You already mentioned that it’s never a technical problem which prevents a game from being published.
So far.
It also depends on the game. So, we are in in the free-to-play market that means you get more users to try out stuff but you have maybe 30 seconds to convince them that they should give this game another two minutes. There is not much you can do with story in these first 30 seconds. You can hint at there is a good story, but I think story is more important - and then it becomes really important - for long term retention. So in our market, for us it is a hit game if the game makes the revenue for maybe a year or two. And to keep players engaged for that long time, then you need a strong story arc. So, story, I think it’s mostly for that, to keep them in the game, but to have some start then it’s visuals and also the game mechanics and what kind of game it is.
Probably. I don’t know, I am a bit skeptical about that any of these things will come next year. So a few years ago location based services and location based gaming was the hype of whatever. Later, it turned out that even if you have 100,000 players playing your game spread over the planet Earth, it’s hard to meet someone. So it’s practical problems like this. And also as long as VR still means you have to put Oculus Rift or something on your face, it will not become pervasive. You will not see, if you take the underground to the airport you will not see people with this. Something like Google Glasses might still have chances, I think we will see something like a blurring, bringing gamification elements to other areas. We already see this when you go to Stackoverflow, you have a high score like how often you could help, that’s a gamification element. So blurring along these lines and then also with the internet of things you have more and more devices that you can actually put games on, think of watches that are upcoming right now for example, then the lines will blur. But it will be a slow gradual thing, it will not be a big bang and oh, now everything is different. Small experiments, some of them will get into action and will start to grow.
Ralph:
Are there different types of games, in the sense of we have for example the people that are gaming in the tube and on the way to the airport, they won’t to use the internet of things and stuff like that, and maybe younger gamers that really are focused on gaming and they are playing at home and there is no problem in using Oculus or using sensors around me and stuff like that.
That can definitely happen, you have different segments. So what we try at Wooga is basically address those players that are not catered for by the console industry for example, but you will have consoles and you will have your VR environment and this will be perfect. And you will have this target male audience of 15- 35 who sit at home and who are able to spend three hours playing your game. You will have that, but this is a minority. If you look at how many people play Candy Crush, then more players play Candy Crush on a single day than consoles have been sold over their whole lifetime - so that’s orders of magnitude. What we also aim at for Wooga is more like the mass audience. We want to bring gaming to everyone. The vision is everyone should play not just watch videos and listen to music, they should play. And when you look now in the underground public transportation I think we’ve gone a healthy dose that way and we want to be part of it and also bring this along, so this is where we are focusing, but there are different niches and for now I think that VR is a niche as console gaming, gaming society as a whole is a niche.
Ralph: So you are going for the casual gamer.
Yes, definitely. So, I am going for my mum or my wife, because not many people do that, not many companies do that. But I think everyone should be able to play! Why only these male stereotypes of half-naked women and crushing someone’s skull? I think there is more to gaming than that.
Ralph: That’s right. You mentioned console games, is there a need for consoles in the future? Because mobiles are getting more and more powerful, but I guess there is still some kind of gap to consoles.
I think they will stay. And I also have a console at home. It’s just that I don’t have that much time to play with it and when I put my games in then I have to put my son somewhere else because he must not see that. So you have this conflicting resource of the TV in the living room like who gets it. So consoles will stay and they will always be more technologically advanced, specialized on gaming than an iPad. But an iPad is pretty cool if you compare it to PlayStation 2, I don’t know, I think you would prefer the iPad. There are also different ways of gaming - it’s very hard to play a console when you are away, but in the airplane, when I came here, yes, I played.
Yes and no. So this fragmentation of catering for different platforms and then especially thousands of devices if you look at Android, that of course is a problem for us as well. But there are tools like specialist game engines like Unity for example or Unreal 4, that allow you to have a true cross-platform development. So in the past we first tried out to do HTML5 for games and it just failed because technically it was not on par and in this competitive market, the games were not good enough, so it was a failure in that sense. With Unity now this looks better, there are a lot of things in Unity I don’t like about it, but it does one thing right: You can have a small team developing a game very fast and you can publish it on different platforms, from consoles, Windows phones, even Blackberries probably, but mostly for Android and iOS. But then comes the thing that I call the operator stuff, especially with Android, these low end devices that then takes on more tooling and I think everyone is in the same boat again, games and app developers.
9. Did you actually build native games, native iOS or native Android?
We started with that. After our first experiment in 2010 with HTML5, we then switched to Objective-C development using Cocos2d and then you can quite easily port it to Cocos2d-x on Android using C++. So we did this in the beginning. But this process of, I would call it chase port, you would have one game on a platform, in the beginning it was Flash, then you have another team that is porting this to Objective-C and iOS, then another team that is using C++ to bring this to Android. This is a very clumsy, slow process and with every port you lose quality. People there are not happy because it’s oh, here is the source code, now we do this change, they are just following. So, we tried out different things and we ended up with Unity because from having one team working on the game it’s so much easier if they work on one code base and it can lead you to things like oh, now we have one code base and actually testing out a new feature on Android is so much easier than on iOS, so we test out features in Android even to publish them on iOS first for business reasons. So if you have this true cross-platform development then you can also leverage the individual strengths of the platforms and make it work for you.
Let’s put it this way: I come from the engineering side, at Wooga we have these independent teams and they can decide on their own technology. So we don’t force a team to use Unity, but what we give them as a constraint is that any given time you need to be able to have same quality on both platforms. As a result of that they choose Unity. If some other technology comes along, we might give it a try. So, from a technical side you want to be on par, in sync, and launch at the same time. With our launch last week, Agent Alice, very successful on both platforms, they did launch at the same time. But you always want featuring from the app store, so from Google and from Apple, and if you would launch it on Google first, the question is would Apple give you the same app or otherwise. So this is business reasons and this is not my domain, so from technical side I want to be able to do whatever is necessary and then someone else decides where we launch first.
Unit testing, of course, system integration testing, is a bit harder on Unity then you would think, but we do it. The testing for fun, this is where the usability testing comes into play, where you give someone your game and they have never seen the game before. Can they work with it? Do they have fun? And after they’ve played the game for half an hour, do they reluctantly give you back the device or is it “oh, fine, now I can go”? If they say “fine, now I can go”, then probably something is wrong. It’s also that we do a lot of internal testing, we use tools like HockeyApp, or an internal app store, so all versions are quite freely available within the company, so we get a lot of feedback internally. But then the ultimate testing is putting into a market and see how users behave. The most important metric is retention. If users don’t come back after three or seven days, something is wrong and you can think whatever you want and production value can be as high as you want, if people don’t come back to play your game, forget about it. So, that’s the ultimate test, how users behave.
12. [...] What about the infrastructure challenges in gaming?
Ralph's full question: So there is some, not some, but very much social aspects in gaming I guess, it’s about sharing your achievements, your high scores and I guess you mentioned if the user doesn’t come back and this includes communication between devices, between devices and servers, and there is a huge base of gamers, I think, so what about the infrastructure challenges in gaming?
Yes, they are quite high. So, one thing is how we develop the mobile client and one thing is how we set up the backend, and backend for us, we usually use a REST approach for our servers, that’s easier to scale and over time we learnt to keep as much stuff in memory as we can, because when you have a game with a few million users a day and they play the game a lot, you easily end up with a few thousand or 10,000 requests per second hammering your API, and if you then go to disk all the time you have a problem. So it’s all about classic ops tactics, of automating your setup, if you are on the cloud or not, again this is something that our games team decide, but to add new capacity, this is something that you need to press a button and then it works, otherwise you’ve done something wrong. And they have a lot of good practice about the architecture and in the end it always boils down to there are players who play the game and they want to persist changes to their game state, so they will change, they just built something or leveled up or whatever, and this change needs to be persistent somewhere in the backend, so we need to do some kind of disk, and even if it’s SSD, it’s still super expensive, so how can you build layers on top of that that you make this change to this less frequent and keep it in memory. And this is also why we ended up doing some backends now using the actor model, so we started out using some Erlang, now we are also doing some backends using Scala and Akka, because for us the notion of this is one user and it has his own memory, this is the actor and the lifecycle of the actor in memory is the lifecycle of the game set in memory and this model of thinking about it makes it very easy to keep the important stuff in memory and model the game logic without going nuts.
Ralph: So you have quite cool techniques in the frontend because you are doing gaming and very sophisticated architectures in the backend to be as scalable as you need to be.
The nice thing about games is every game is anew, so if you made bad decisions technically in one game the next games can learn from that and do it better. If you did something right, in the next game you can take over the same thing and learn from that. It’s kind of like evolution on a large scale. So, five years ago we had PHP and MySQL as backend, so classic lamp stack and now it’s completely separate. So it took us something like three years to completely go through many iterations and now we have a stack that just works, so we are not afraid of these loads anymore. Because of this approach of each game tries out something, it can innovate and if it fails it will be only one out of 20 games and the rest can learn from that. This evolution makes us very flexible and robust and innovative in technical terms.
It definitely is for Sony when you look at credit card theft. For us one thing we don’t have to worry about is financial security because all of our payments are done by Apple and Google and Facebook and Amazon, so we don’t have this data. So, for us security is mostly about cheating, and yes, this is something we have to worry about, it’s a part of following best practices and then being frustrated that they cheat anyway. So we protect our data by usual means, checking how we set up servers, also people from other teams cross-checking so you have to explain to people from other teams how you set up so you get some independent opinion and tips and tricks on how to do stuff. But regarding security our biggest problem every day is cheating and to a degree we just live with that. So if you want to cheat, ok, cheat. As long as your cheating doesn’t spread virally and affects all other players, ok, we just live with that. We are doing games, we are not doing banking transactions or people’s lives depend on that. We are doing games that for 90% of all players are free.
I hope so. We do this game jams every now and then, every studio does it once a year, so it’s three days and everyone just hacks away games. The energy that you have then and you also see the games that come from the indie market, the creativity that is in there, I just hope that there is something. But what we also see is that everyone who uses a paid game approach in the app store is struggling. 95% of all revenue is done by free-to-play, because a free game gets 20x more downloads and you get more ways to monetize it. But if you want to do a free-to-play game you have to be able to maintain it for years and that’s very often outside of the capacity of an indie developer. So at Wooga we basically have these independent teams and their mental model is kind of like they are indie teams but together as a group, they survive. The big problem here is how you get users into a game and with companies like King, Supercell, also Zynga, having a lot of money due to IPOs or being bought or just making a lot of money with their top games, the prices for user acquisition, so you pay for adds and bring users to a game, they just doubled, now are 4x higher than just a few years ago, so it’s getting really hard. I think right now if you want to make a start, you probably go to the advertisement business, there is more money to be made there than in games and that’s a bit sad. But as long as the discovery of games is not changed in the app store or on Google play, it will not change and it will be harder for indie developers than easier in the future.
Ralph: Let’s hope the creative ones will have their chance.
I hope so. I think the founder of Supercell, he also has a VC firm and he stopped funding games companies. He should know what he is talking about.
Ralph: Ok. It was very interesting, thanks for having the time to answer our questions and have fun at QCon.
Thanks for having me here, it’s an awesome conference. Thank you.