On the discussion of how best to rank and evaluate game development tools I was recently challenged by the May 2009 Gamasutra article: Game Tools Tune-Up: Optimize Your Pipeline Through Usability by Dan Goodman.
I was challenged in particular because I read the article right in the middle of an effort to put together a website called http://IndieGameTools.com that I hope will serve as a ranking tool for commercial game development tools from an Indie developer perspective.
It took me awhile to realize that there are fundamental differences between evaluating tools through usability testing, the premise of Goodman's article, and an attempt to skim tool quality information from transient website visitors. Usability and focus testers are more or less trapped by the testing process. So they have an incentive to provide detailed information and feedback to shorten their involvement and get back to work. They therefore make time to 'pontificate' on the usefulness of a particular tool.
If I were to attempt to ask detailed or nuanced questions about tools of website visitors they would quickly opt-out and go on to another website and I would collect nothing of value. People don't generally think very hard or provide much detail on web surveys. Especially if you are busy as is the case with almost all game developers, instead you just avoid surveys like the plague.
I think the simplest of gut level questions will work best if I am to collect anything of value from visitors... a visitor might be willing to answer something like "I (could/could not) afford Tool X and it (was/was not) helpful, and the tool maker (was/was not) responsive to my needs." but not much more than that. In the end I did choose criteria that were a little more specific than that question, but not much.
Anyway, I discuss this a little more in my personal blog entry Evaluating Indie Game Tools if it is of any further interest. I'd also love any feedback that you are willing to give, since the website is still in its infancy it would be much easier to change ranking criteria now if I missed anything then to try and change it after the database starts filling up with votes. Thanks.
Robc