This text is a translation of an article originally published on DOU.ua.
My name is Arsen, and I'm a .NET Developer at Plarium, working on the backend of RAID: Shadow Legends, a dark fantasy MMORPG.
This article will be helpful for those who want to know what to expect from working in game development and how it feels to be taking part in a project where the number of daily active users exceeds a million, and every minute of a server being down costs the company reputation and money.
I started my career as a full stack developer in a small IT company where .NET was used for the backend and Angular for the frontend. I took responsibility for the entire development cycle myself and was aware of how things were implemented on the client- and server-side, and how and where everything was stored in the databases.
But there were drawbacks, such as having to deal with a large scope of work and a gazillion daily updated frameworks. You try to keep track of one, you miss something on another, and inevitably something gets lost. I wanted to become a highly focused specialist, so I started thinking about switching to .NET or frontend development.
When I started feeling more confident in the profession, I decided to move to a larger company with more interesting projects. I chose another outsourcing company, where I was only working on the server-side. It was while I was working there that I found myself at a uDev conference and learned more about Plarium. By then I already knew a little about Unity through watching videos on YouTube and some pet projects, so I decided to take a shot at a Unity Developer vacancy they had open, and managed to receive an offer.
Since I like to write pure code without any UI stuff, it didn’t take too long for me to figure out that, having worked at Plarium for a while on the client-side, it was really on the server-side that I belonged. When it was suggested that I build a tool for game designers using WPF, I immediately grabbed the opportunity. It was then that I realized .NET is for me. I moved over to the server-side of gamedev and, three years later, I’m still here.
In product gamedev - as opposed to my previous career in outsourcing - I began to treat projects differently: more as “my own”. During my first week of work, at a daily meeting, I heard the phrase: “we’re doing this for ourselves”, and somehow it stuck with me. In outsourcing, I was used to giving estimates in hours, but here – it’s days: we spend a lot of time on code review and can afford to search for the most optimal solution.
I was lucky enough to be involved in the development of one mobile game from the very beginning. I knew each member of the team and understood how everything worked. I felt that I was part of something big. You can never be sure whether a game will succeed or not, but when you work on it from the very beginning till the very end, it’s still “yours”.
That never happens with a huge outsourcing project. You never know if it’s going to be there tomorrow or not. Even when it’s a long-term project, you still feel that you’re doing it for an external customer, who might suddenly say: "Guys, we aren’t a good match; we’ve decided to work with another company."
What’s important to me is to be able to make a difference. When we were making our mobile game, the game designers, developers, testers, and managers all got together and argued over whether a feature would be a good fit or not. We had some real battle royales, probably like what happens in a startup. Imagine: you, a developer, are standing in front of the game director telling them why something is a good idea. And they say, “that sounds cool, let’s do it.”
I don't think gamedev is suitable for those who believe that game development is all fun and games, though. It’s anything but. This work can be stressful. Production issues lead to disappointed or angry players, complaints to support teams, negativity in chats. Not only do we have to fix problems as quickly as possible, we also have to give players warnings about server maintenance, and offer them compensation if things go wrong.
Once, on one of our strategy game projects, some test code for issuing game resources was uploaded into the game. The server began to give players a weekly bonus every minute. This happened over the weekend, so we didn’t immediately spot the problem. As soon as we noticed it, we wrote a fix and deployed it. But just imagine how disrupted the game balance was for that short period of time, when players were receiving huge numbers of crystals (game currency) in-game.
Urgent changes mid-sprint are also stressful. For example, the release of a feature might be planned to take place in three months but if the analytics team thinks it’s better to move it forward, it gets included in the next sprint in place of something else. Or if app store policy changes, you need to set aside other tasks and update the game ASAP.
When a feature has been released and players give it five stars and call it "cool", that’s inspiring. But if players do like a feature then it's really the game designers who deserve the credit for it. As a backend developer, I’m responsible for ensuring that the feature works correctly and consistently, and that the game performs well and doesn’t crash.
If something crashes outside of normal working hours, I'll still have to look into it. That never happened in outsourcing. In gamedev, though, if you screw something up, it’s impossible not to worry about it. Not only because it can upset players but also just because of knowing that it was your fault. Sometimes, you can go into the team chat at midnight and it’s like a full-blown working day in there.
On one of our projects, the game started to roll back user actions. It was an offline game, so the way it worked was that every user action was saved on their device, then sent to the server every 30–40 seconds. The server would validate and save those actions, then send a response back to the client to say that everything is in order. But every now and then the actions on the client- and server-side would be different from one another. If the server failed to repeat the action on the client-side, it automatically thought that the client was cheating, and would roll back the user’s actions. The client would try again to send a request to the server, and everything would go round in a circle.
After one release, this type of error began to occur more frequently, and complaints from players started pouring in. To figure out what was going on, we temporarily turned off validation on the server. As it turned out, the problem was backward compatibility. We had duplicated logic on the server and client. But when changing code on a server, you have to make sure that an old client with the old code will work correctly on the new server too. We should have foreseen the issue, but in this case we had let it slip by us. I remember only too well that sinking feeling you get as you watch one error after another appear on the server...
Our server has a very narrow stack. It’s built like this: pure .NET, ASP.NET, and MS SQL Server. No complicated and lengthy entity frameworks, everything is simple and fast. Our servers process 16 thousand requests per second and 140 million in-game battles per day.
When you work with a highly loaded system, you need to understand it deeply, know all of its bottlenecks, and understand how everything works inside the platform. For example, on small projects, I never gave a thought to how the garbage collector works. But on projects like RAID: Shadow Legends, it turns out that it can work for a long time - as we found out when we collected analytics. I really didn't even know such a thing was possible - with .NET, the garbage collector is implemented automatically and you don’t even need to monitor it.
But when you’re in the position I’m in, that’s the level of detail you need to be at: you have to understand all the nuances of the systems you work with, and how you can make the most of the limits and opportunities they create.
Predominantly, we use proven technology. On average, our database takes no more than 10 milliseconds to complete a request. It’s not unheard of for modern frameworks to take 10 or even 100 times longer, and that is always a potential risk. Even if at first glance a framework works well, you never know what can happen under a load of millions of users: for example, under the kind of loads that we work with, operation speed can suffer when using a third-party library.
It’s not that new technologies are off-limits to us, it’s just that we first test them thoroughly. In the outsourcing companies I worked for, we often used new frameworks. Sure, it was cool to keep up with the latest trends and be aware of all the latest Angulars and Reacts, but I think that for a developer it’s not really that important to know new frameworks because they can be learned quickly. It’s more important to be able to solve non-routine tasks. The deeper you understand how the system works, the more expertise you have.
For the projects I work on these days, it’s sometimes more effective to write a custom solution. For example, we built our own JSON serializer. We did that because, in terms of performance, it was inefficient to use third-party frameworks. They’re complex and have extensive functionality like data validation, analytics collection, and problem diagnosis. In contrast, we were only interested in one task that needed to be completed as quickly as possible: serialize a data model to a JSON string and back. For that, our simple tool turned out to be many times more efficient.
I think that most of the problems faced by a server programmer in game development are not related to peculiarities of the system, but to business tasks. When I was starting out as a full-stack developer, I would constantly google things like how to add something to a page, how to calculate something on the web, or how to implement something in Angular. But now, it’s different. Yes, I can google how Apple implements some kind of API, but that's it.
For example, I had a task to save the progress of players who didn’t have an associated ID. After all, what if a player at level 100 or 200 accidentally deletes the game? We care about our users, so it was important for us to try to save and then restore their progress in such cases.
The difficulty was not the technical execution, but finding a solution. Our other projects didn’t have any similar functionality, so we had to come up with everything from scratch. Since it was an offline game, we needed to figure out how to save the player’s data until their next internet connection.
In the end, we decided to use Google Drive or iCloud. When the user installed the game, a token was automatically generated in the cloud. When there was no internet connection, their data was stored on their phone. As soon as a connection appeared, the data was synchronized with the cloud. If the game was deleted, the token was saved in the cloud. So, when the game was reinstalled, we checked the token and offered the player a choice: restore their progress or start fresh.
But another problem arose: it turned out that the token might not arrive immediately from the cloud when the game was reinstalled. Google Drive works randomly and while iCloud is more stable, it could still be the case that the token might not come or might not register. We, meanwhile, had a service that was constantly spinning in a loop, checking whether or not a token was present. Since Google synchronized the token every 24 hours, it meant that at some point over the course of a day, a player's progress would be unavailable.
So, a player might download the game again and get to the twentieth level, and only then would the token be discovered and pulled from the cloud. In that case the player would have two progress points, which meant that we needed to show them both, and include information for both on what level the player had reached and how many coins (in-game currency) they had. The player would then decide which progress point to restore, and we would delete the other.
These are the types of technical changes that excite me, and gamedev is full of these examples. In my opinion, if you like games, that doesn’t mean you’ll be a great developer. What’s most important is that you enjoy solving problems like the ones above. If you do, it could be the profession for you.
Several books and courses helped me understand high-load systems, as well as game and mobile development.
If you're interested too, welcome :)
What to listen to and watch:
What to read: