As IT pros, we often feel like we're racing from one problem to another or we never have enough time to deal with all the demands of the job. So it feels great when efforts pay off and something goes right.
One such instance took place when I was working for a company that produced free-to-play browser games for large numbers of players. I was part of a team that constructed a fictitious world where players could roam around continents and oceans and come across many quests to solve. They could also battle each other or band together to go on quests.
Several times each week, our team released updates to this gaming world, so each working day was a busy race toward the next due date.
Our players were very vocal about changes, whether they liked them or not, and after the updates were released, the in-game chats and forums would be especially busy.
The trap is set
One day, we changed some of the financial rules. This was badly received by the players, including highly respected community members who had been playing the game since it had rolled out several years before. The players heatedly discussed the change for several days.
Eventually, they agreed to gather in one spot on the world map and stage a protest, expecting to cause a DDoS attack on our servers. Our community managers immediately alerted us to what the players were attempting.
We felt well prepared technically, but still uneasy. Our system architecture was sophisticated and our server resources quite extensive, but no one had ever put the systems to such a test. To be on the safe side, we decided to batten down the hatches.
The players' attack plan was founded on two assumptions: that we were providing only one server for each translation of the game, and that their gathering in one place of this world would affect this server's performance. We never told them that both assumptions were wrong.
In actuality, it didn't matter if players' characters were in one place or spread out all over the world, or that during log-in players had to choose an entry from a list of server names. Due to the architecture we had devised and assembled over time, behind each single list entry for a "server" stood a load balancer set atop a pyramid of similar components that were founded on clouds of servers floating above globe-covering Content Distribution Networks. Players were served by many more than one machine.
Tech preparation: All hands on deck
But would our architecture endure what the players had planned? We thought so, but took extra precautions just in case. We had five days to prepare and tried anything that came to mind: load tests, renting more cloud servers, networking with CDN suppliers, and even borrowing servers from other games (because our supplier would've taken too long to deliver new ones).
All the while, the players bragged on the forums how they would get back at us for the rule changes.
Eventually, the appointed time for the protest rolled around and we'd be able to see in real time if our preparations would work out. Our team gathered around as one of us logged into the game and went to the place where the protesters had agreed to meet: a deserted void in the center of the world. On another floor, the devops team watche server performance even more closely than usual.
The moment of truth
The appointed time came, and within seconds the place was occupied by tens of thousands of protesters. We could tell from statistics that some protesters had even been playing the game for days and nights on end to have their game avatar traverse seas and continents in order to make it. Very quickly, the entire screen turned to pitch black, as the software wasn't able to render all those characters individually. The in-game chat was flooded with so many protest messages that it flew by in a blur.
Then something happened that no one had expected: The massive gathering attracted more players who arrived on scene. They had been roaming the world as usual, looking to challenge, battle, and plunder. Immediately, they attacked the protesters. The protesters, caught up in the action and not able to tell friend from foe, hit back.
A massive battle broke out—the largest ever witnessed in this game. Within moments, the screen turned white with smoke and dust, then black again with defeated avatars decaying into the ground and into eventual oblivion. The avatars left standing quickly fled the scene.
Shortly after it had started, the protest was over and had vanished. The in-game chat turned to silence as the players went quiet from the shock of what had happened.
The onlooking team members were also shocked—faces turned pale and jaws dropped to the ground. Then someone reached for a phone and called devops. Our preparations had paid off, and our system had balanced the server load so well that it had easily handled both the expected protest and the unexpected battle without any problems ensuing.
We went back to our regular duties. And as time would show, that was the end of anybody debating that particular rule change.
The lesson learned? It will pay in time to have your proverbial tech haystacks tied down thoroughly as there's always a storm coming—though when and where nobody can tell in advance. And when the solid preparations can be beefed up to handle a known, tougher scenario, it's very satisfying when the efforts pay off.