Grading student game projects

How do you evaluate and grade student-designed games? It is hard, no doubt, especially as there’s rarely enough time to play all the games enough to properly test them. Still, here are a few pointers. This is aimed primarily at non-electronic games, which are much better tools for teaching game design than electronic games, because a much greater percentage of a student's effort goes into design than prototype production, and they take a lot less time to produce a playable prototype.

Any “review”, whether of a book, a movie, a play, music, software, or a game, must answer three questions: what was the creator trying to do, how well did he do it, and was it worth doing? These questions can help guide a teacher grading a student game project, just as they help a reviewer evaluate a commercial game. However, I am not going to discuss these questions, except insofar as what follows indicates how well the student(s) did it.

If you don't play games, all you can do is try to judge effort and professionalism. If you do play games, but are not accustomed to evaluating games without playing them (perhaps talking with people who have), you’ll have a problem, because you often won’t have enough time to play the game enough to really find out what it’s about. (This is why game reviewers must spend a lot of time playing a game, to avoid being misled by first impressions.) Some games just don’t sound (or look) like much until you play them. Some sound or look really good, but fail in actual play.

Fortunately, if you’re very familiar with games and have some design experience, you can judge the important things fairly easily.

Just as the three most important factors in real estate are location, location, and location, the three most important factors in a game are gameplay, gameplay, and gameplay. There are several aspects which I’ll discuss in the next paragraphs.

The most important question about any game is, what does the player DO? What are the challenges the player(s) face? What actions can they take to overcome those challenges? This is the heart of most games. You can often see quickly, when evaluating a student game, that the player just doesn’t have much to do, or that what he is doing is quite repetitive without compensating factors.

Related to this is player interaction. What can players to do affect each other? There are certainly good games (often “race” games) where a player can do little to affect another player, but most good games have a significant-to-high level of player interaction. Or if it’s a solo game as are many electronic games, interaction between the player and the game becomes the target.

Another question related to what the player does is, is the game replayable many times without becoming "just the same" over and over? Perhaps this is less important with electronic games, as players EXPECT such games to become “the same”, and they also don’t mind sameness as much as the non-electronic gamers do. (So many AAA list electronic games are quite derivative of the gameplay of so many predecessors.) Nonetheless, the better a game is, the more replayable it is likely to be.

Some people would add one more thing, is the game fair? Do the rewards seem to match the effort?

So much for gameplay design. What about what we might call the professional aspects of the game? Care of construction (NOT looks--construction of the “rules” or electronic equivalent, game mechanics) is important. Sloppiness leads to imprecision which leads to confusion.

Even more important, how much was the game playtested? Are there records (not necessarily elaborate–I tend to list the date, who played, and anything unusual that occurred, along with changes I decided to make (or decided did not work, so I changed back). That can be just a couple lines of documentation per play.

With very few exceptions, a game with a good basic gameplay design won’t actually play well unless it has been thoroughly playtested to work out the little (or big) obstacles to good gameplay.

While the instructor may not be able to play the game enough to know it well, he can suppose that a poorly playtested game won’t be worth much, and that one that is playtested a lot is more likely to be a decent game. “Playtesting is sovereign”, whether the game is digital or not.

I usually ask students to give me the original playable prototype, and the “final” version. There should be very significant changes in the game, if it has been playtested much. You might even ask students to list what significant changes were made.

Here are some specific comments about what NOT to use as criteria:

"Fun" is not a criterion. We can't generally agree what fun is, and your idea of fun is different from mine. A chess master has a different idea of “fun” than your mother has! Some people like party games, some like silly games, some like perfect information, some like planning ahead (hence tend to like perfect information), some like much that is hidden, some like reaction to circumstances (hence tend to prefer hidden information), etc.

Likely hundreds of thousands of people have played my game Britannia, yet even *I* wouldn't call it "fun". It may be interesting, fascinating, and lots of other things, even educational, but many players would not call it fun.

Story is not a criterion. This is especially important because so many students don't understand this, and think story or something other than gameplay is what's important in a game. People play games, they listen to/watch stories. Yes, you can make the story interactive to an extent, but a great many game players really don't care about story. It is absolutely necessary to pound into students’ heads that story is not important in the design of most games.


What about the marketing palaver–the game concept and treatment and so on? These are not even written for non-digital games, though when the game is “finished” and the designer is trying to persuade a publisher to look at it, and then only sometimes, a description of the game will be written that is something like the game concept.

These documents in the electronic game world are marketing documents that have NOTHING to do with the quality of the game, NOTHING. They represent a simple plan for what the game will be.

Consequently, does it even make sense for a teacher to require students to produce these documents? I would say, it makes sense only insofar as the document might help the instructor evaluate the game, or help the students create the game. In that context, the two to three page game concept is useful, but something longer such as a game treatment may not be.

At the least you might ask the student(s) to briefly characterize the “essence” of their game–and let them decide what “essence” means.

Looks of the Game–not a criterion. This counts for virtually nothing; in fact, for non-electronic games I definitely do not want students to spend a lot of time on looks. As long as the physical components of the game are clear, not confusing, that's what counts. (There's a rule of thumb in the boardgame world, that the better a prototype looks, the less likely it is to be a good game, because novice designers spend far too much effort on the looks of the prototype.) I want students to understand that gameplay is what counts, not graphics, even though in the AAA list electronic world graphics become quite important simply because of the youthful audience.

I heard of a “game design teacher” who severely downgraded a student’s non-electronic game project because the box he supplied wasn’t large enough for the game. “The BOX?” That’s completely irrelevant to design; most experienced designers don’t make a box for their prototypes (I *never* have). This is something only novices who don’t understand what they’re doing think is important.

For electronic games, looks only matter near the end of development. And students will rarely have the time to do the playtesting and incremental, iterative modification necessary to “complete” a game and get to the true end of development.



So what else do you evaluate? Appropriateness for the audience. Is there an appropriate mixture for the audience and game type? If it’s a party game (whether “Apples to Apples” or a Wii party game, is it relatively easy and does it promote interaction amongst the players? Here we might actually ask if it’s “fun” in a party sense.


I am not a person who lists exact grade points and values, because I don’t think you can consistently judge this carefully, nor am I confident that my “rubric” would take everything into account. But I can judge an “A”, or a “B”, or a “C”, or something in between, based on these criteria, without assigning specific percentage numbers.

It's very difficult for anyone to "grade" these games without playing them several times, for which there is no time. My main criterion, aside from what I can see about the gameplay, is whether the students playtested the games and benefitted from that.

For groups, I also give them a peer evaluation sheet to fill out. The idea is that I may find out which people in the group actually contributed most, or least. It has its flaws, but is better than nothing.

Training vs. Education

I've had some thoughts from recent experience about how important the distinction is between training and education in community colleges (and in any educational institution).

Everyone seems to have his or her own definition of "training" and "education". Here are mine.

In general, when you train someone, you tell them a specific way to do (or not do) something. In some cases it can be strictly rote learning, as in how to assemble or disassemble a weapon (there's only one way to do it, by the numbers). In any case, you're not trying to help people make judgments about uncertain situations, you're telling them, "if A, then B; if not A, then C." Many corporate training sessions are of this sort.

In education, you explain to people why something works the way it does or is the way it is, so that they can understand its nature, what's going on. If they lose their way, they can figure out what they need to do to get back to their objective. They can deal with uncertain situations, where there's no clear answer.

In general, the training recipient requires good memory and good organization more than good thinking processes; the education recipient requires application of intelligence, and sometimes critical thinking.

To me, this is the difference between having a set of written directions to get somewhere, and having a map. If you go wrong with the written directions, you may be able to backtrack, but if you get off the path you may be completely lost. If you have a map, you can figure out where you are and get back to where you need to be even if you've strayed far from the proper path. The first is analogous to training, the second to education.

In K12, we now have many schools that are training institutions. The teachers know what is required in the "end of class" test (EOC) , and they know that their job security depends largely on how their students do on those tests. So they drill into their students what they need to know to pass those (usually, multiple choice) tests, and that's all. The students, too, know this is the score; they know they can do next to nothing during the semester, as long as they pass the EOC. Even the smart ones tend to do little, then cram from the book (which, of course, is supposed to contain everything they need to know to pass the EOC), then forget it after the test. (Yes, there are many exceptions: this is the trend, not in every school or every class.)

I've seen the results of this time and again in college. Students expect to be told exactly what's on a test, and what they need to regurgitate, and are dismayed when I require them to actually think. They don't have any idea how to think, because they haven't been required to. In classes I try to impose the "educated" attitude from the beginning, but it's hard for kids to adjust.

Unfortunately in colleges of all types we have teachers who think their job is to "convey the material". I had one teacher say to me, when I was discussing a difficult class, "but you covered the material, didn't you"? That's a dumb question, I'm afraid. That's not what education is about, but it is what training is about. A corollary of the "training mentality" is that if you present the material and the students have the chance to absorb it, and some don't, then oh, well, that's the way it goes. In education, you're trying to find ways to convey what you mean to each class (and each class is different). You've not only conveying information, you're conveying an attitude, a way of doing things. If you can't cover "all the material" because a particular class is having problems, oh, well, your job is to choose (to judge) what's best for the class, and do it, not necessarily "cover all the material".

This is something like sports team coaching, but we don't have as many hours with the students, and we have a lot more students than, say, Coach K at Duke has basketball players (and he has three assistants). Many "trainer" types don't even know the names of their students, let alone understand them as individuals, and don't think that's a problem! How can you judge what the students need, when you know so little about them that you don't know their names?

The problem is, many "teachers" aren't interested in this more complex kind of thinking and understanding of what is needed. It requires more effort, more thought. And unfortunately, school accreditation people also aren't interested, because they can't measure it.

See my post about "educated" people (http://teachgamedesign.blogspot.com/2007/11/game-industry-wants-educated-people.html). I'm not talking about degrees here, I'm talking about a way of thinking and approaching life.

Unfortunately, the US "education" system is becoming more and more training oriented, and less education oriented, every year. Perhaps one indication of this is the very strong tendency of accreditation organizations to emphasize (in teacher qualification) degrees and classes taken rather than actual ability to do something. We are more and more finding "teachers" who have never done what they're teaching in the real world, and it shows. But if all you're doing is conveying material, telling students what to regurgitate on multiple choice tests, how much will the experience of a person who's actually done the work professionally matter? It's a system geared to turn out people who cannot do complex work in the real world--people who have been trained, not educated.