I’m torn as to the pros and cons of this approach. Speaking as a long-standing community member over at the Jumpgate Evolution forums, one of the biggest complaints I see is that this “random Joe off the street” approach means you’re getting feedback off people that don’t necessarily have the drive or passion to provide meaningful critique of the game. On the other hand, if you want to make a game that’s easy to get into, accessible but ultimately complex to master (which I hope JGE will be…) then it’s a really good approach to take. But where do you draw the line? How much feedback can you get from someone that might not even have heard of your game or that doesn’t even play MMORPGs, let alone after only a couple of hours of gameplay? Maybe that’s where the “complex to master” element comes in; your random, focus testing public are the perfect testers to tell you what’s wrong with your user interface and to identify usability issues. Hermann Peterscheck (Lead Producer for JGE) has often been cited as touting that “first fifteen minutes” as being critical to capturing a potential subscriber in an MMORPG. Can you get the accessibility and polish early on in a project’s lifecycle (which is kind of the idea of agile, iterative development) with just internal focus testing? Is it crazyness to keep things internal (all use of cranial explosive devices aside)?

I’m interested to see how this pans out with a smaller developer like NetDevil. The game isn’t in beta yet but has been “Friends and Family” testing for some time so I’m guessing they’re listening to their testers and…iterating their iterations. Time will tell…