I mentioned last week that we sent a fairly terse set of questions over to Blizzard. They're a lot more direct than developers are used to receiving. The crazy thing is, the answers we got back were a lot more direct than the interviews out there usually receive. Funny, that.
Without further ado:
1. You say that you sold six hundred thousand units. Is the game not capable of supporting this many users?
The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.
We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.
With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load—which, again, we hadn’t planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.
2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?
We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow players to move from their high-population realm to one with a low population is one of the things we’re investigating. We’re exploring this option fully and hope to be able to communicate more detailed information about it to our customers in the coming weeks.
3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?
The player raids often have hundreds of people per side in one area; that area is on a server that is also running the rest of the continent, and that can result in the latency you describe—depending, as well, on the total population of that server. We’re continuing to look into the issues surrounding this dip in performance. Battlegrounds, on the other hand, will run on the instance server, so there should be no such issues. Additionally, players will be unable to "zerg" in Battlegrounds; there will be a limit to the number of players per side.
4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?
The emergency maintenance periods are to restore stability while we continue to narrow down the cause of the problems. Some of them are also to deploy temporary fixes to various in-game systems while we continue to develop a longer term, more stable solution. World of Warcraft delivers many complex features that are unique to MMOGs. Features such as the in-game mail system, auction houses, player inventories, flight paths, quest states, etc. use a lot of server bandwidth, which makes pinpointing problems on the server infrastructure much more complicated.
Recently, the extended emergency downtime for a certain number of realms was needed in order to better accommodate our growing player base. Some of the upgrades that we planned for all of the realms were made to these realms first, as they are among the most populated and thus most in need of aid. We set the realms up on the latest top-of-the-line hardware and made the software upgrades accordingly, but some unforeseen issues cropped up with the database that resulted in the problems players currently see. This is no fun for our player base, of course, and we don't want to keep the realms running in a condition that frustrates our customers when we can attempt to fix things . So, these downtimes have been used to change hardware and apply fixes that will hopefully alleviate the issues. We have not yet resolved the problems, but we're working on this around the clock.
5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.
These types of issues stem from the problems described above. Conflicts occur between some of the internal applications running in the background, and the end result can take the form of temporary login issues. We're working to resolve these conflicts so that they are no longer a factor.
6. When do you expect to have the worst of these problems resolved?
We’ll be constantly working on these issues each day moving forward until they’re resolved, but we don't currently have a set date for when that will be. We're doing all we can to make sure these problems no longer occur -- it's our top priority, and we hope to have the issues fixed as soon as possible. We’ll continue to provide players with regular updates on our progress.
7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?
They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.
8. What would you have done differently?
It would be easy to speculate about what we could have done differently, but that wouldn’t turn back the clock. Right now we’re extremely focused on the issues at hand, and this focus is helping us methodically chase down the problems that are causing frustration for some of our players. The foundation of our company is based on providing a top-notch game experience and an equally top-notch level of customer satisfaction; we won’t be happy until we feel we’re consistently meeting those standards.
