Posted: 23 Jun 2009 18:06

Since the launch of America's Army 3 (AA3) on June 17, America's Army has seen data transfer demands upon our game infrastructure that far exceeded our expectations and capacity to support new game registrations and game play. Data transactions associated with all existing and new players attempting to complete the Basic Combat Training (BCT) portion of the AA3 game over the span of a few days overwhelmed our authentication and database systems. These sections of AA3 include key new features of the game that are particularly data intensive and which we clearly did not adequately load test during our closed and open beta process. As Director of the Army Game Project, I apologize for this gap in service and want to outline below the actions we have taken and are taking to close this gap.

First, AA3 is designed to provide players a great deal of data regarding their performance and key aspects of the online experience. As a result, compared with prior versions of America's Army, AA3 generates about ten times more traffic between game instances and our middleware and backend infrastructure. While we accounted for this level of data exchange when designing our AA3 infrastructure, we did not anticipate the unprecedented demand and data transfer levels we are witnessing as tens of thousands of players are simultaneously establishing accounts and working through the BCT portion of the Game. This was due to the fact that we did not adequately replicate the data transaction loads associated with this game activity during our beta test. As a result, immediately following the launch of AA3, we saw a mismatch between incoming data transactions and our infrastructure capacity. Therefore, we immediately began adjusting our infrastructure resources to maximize capacity and undertook substantial tuning of our database systems. Shortly after launch, we also discovered a hardware fault at our data center that created significant network lag. After addressing these faults, we achieved improvement in our infrastructure performance. However, demand for the game quickly consumed the capacity we created with the foregoing steps. Therefore, while we were able to support additional users, individual players did not see substantial improvement in game performance.

Since last Friday, to further improve system performance, we created a data handling modification for the game designed to moderate data flows during this launch period. This modification was released on Friday evening via Steam and our Deploy Client to automatically upgrade AA3 games installed on players’ computers. Additionally, we began crediting all players for completion of BCT levels so that they could proceed directly to multi-player missions; thereby removing the bottleneck created by the high volume of players engaging in BCT maps and improving the overall player experience.

We also began addressing performance and stability issues on the game servers by implementing memory upgrades and adding more database nodes to carry the load of authentication and player stats traffic between game instances and our infrastructure. We are correcting map rotation reporting deficiencies and issues with the Master Browser System (MBS) that prevent players from identifying servers with available Soldier positions. To improve network traffic, we continue to tune game network code, add more game servers, and correct server administration features so that server providers can better manage their servers. Finally, we are remedying instances where we identified memory leaks and are reducing server memory requirements to eliminate network lag while increasing server stability.

In summary, we are working to add capacity, and correct faults that have significantly impaired player access to America's Army 3. We appreciate player patience as we ramp up our infrastructure to meet the demand that clearly exceeded our capacity on launch day.

Casey Wardynski
Colonel, U.S. Army
Director, Army Game Project (America's Army)

Posted: 23 Jun 2009 18:23

Finally something!

Posted: 23 Jun 2009 18:24

Hooah! Good Job...can't wait for the new Hotfix!

Even with stable servers the game still has a long way to go...so many things im surprised have been missed out or not corrected in the beta.

Step in the right direction!

Posted: 23 Jun 2009 18:25

Thanks for letting us know! I love to see communication from the superiors; all I want are some updates about what's going on. I'm satisfied.

Posted: 23 Jun 2009 18:29

Thank you Sir, for your open and honest comments. We the players appreciate your efforts and understand what a strain AA3 must be on the infrastructure. Just hearing from you in this definitive way will go far in reducing the frustration level among the player base.

We'll keep trying to strain the network while you make it "Army strong"!

;-)

Posted: 23 Jun 2009 18:33

game is promising...very very large upside to the project, just needs some TLC..

please dont let our game die a horrible miserable death!

G

Posted: 23 Jun 2009 18:36

Hooah, [Salute] Kill em all, Sir [/Salute]

Posted: 23 Jun 2009 18:36

Thanks for the letter Colonel it is much appreciated.

Posted: 23 Jun 2009 18:37

Hooah!

Posted: 23 Jun 2009 18:41

Hooah! Great to hear some news after 2 days of nothing. Hope those extra servers come in fast and thanks for all the hard work from the Americas Army team.

Posted: 23 Jun 2009 18:41

Sir, your ongoing and valiant efforts are much appreciated. I applaud the Army Game Project and you have my full support and patience as your team works this through. I'm not a fan of counter strike, I have been a fan of the America's Army game ever since the pre 1.0 recon release, rest assured you'll keep me around even with these glitches. I'm imagining however that many do not understand and they quickly jump to conclusions and want to go play their run frag respawn games that require no tactical skill. It is frustrating however that I can unload a full clip with a saw at an enemy at point blank range and they live to shoot me and i'm dead in a fraction of a second. Aim or not, when they're right there 2 feet away and i've got the obvious tactical advantage they'd be in bodybags not the other way around

Posted: 23 Jun 2009 18:57

Thank you, Colonel, for the open letter. I think all players like to know what the developers are doing to make America's Army 3 better

I finally got to playing AA3 with some friends tonight, and we had the best time!

The map "Ranch" is really, really good

Posted: 23 Jun 2009 18:58

Thank you for the letter.


We appreciate.

Posted: 23 Jun 2009 18:59

HOOAH!

Posted: 23 Jun 2009 19:00

Thank you for the letter. Just hopefully we can get these issues sorted out rapidly fast.

Thanks again.