Alexis, The Encomium TOEIC Test Preparation System
(Continued from Page One)

Reviewed by Maggie Sokolik, Ph.D.
UC Berkeley, College Writing Programs
sokolik@socrates.berkeley.edu

Another item in this section is culturally biased: students who don't know the difference between a basketball and a soccer ball won't be able to answer the question about the picture. Their inability will have nothing to do with their listening skill, and everything to do with experience with sporting equipment and vocabulary.

Unfortunately, this is only a sample of the weaknesses in the Picture Modules. There are many more.

The Listening Modules

The listening modules are less problematic. Although they suffer from the same navigational problems, the sound quality is excellent. After completing a module consisting of 10 questions, the user is told how many were correctly done and in what time. This is especially useful for students trying to improve their time management skills in testing situations. Again, though, the user cannot choose which items to review, but must page through and answer each question correctly before being allowed to move on.

Short Conversations

The Short Conversations are also well done in terms of sound quality and of the content of the conversations. Some attempts at humor, however, fall flat. In item 3.14, there is a question about a calendar date of a restaurant reservation. One of the distractors is "Friday the 13th." The feedback says, "No. This is an unlucky day. They might choke on their steak tartare." This seems a gratuitous attempt at humor, and presupposes a superstitious viewpoint. What's worse is that many users may not understand the cultural reference, and thus not understand the humor.

Short Talks

There are six short talks, each followed by a series of questions. Again, while the sound quality is good, the quality of the test items is less consistent. In Item 4.2 (Weather Report #1), one question requires that the user listen for two different reported temperatures, and then perform a subtraction exercise to get the correct answer. Although a clever task, it requires more than listening skill to complete. Granted, it is a simple subtraction exercise, but it turns the question into a memory and math item, rather than a listening one. In Question 4 for this same listening passage, the user has to select the correct Fahrenheit temperature related to a Celsius temperature. This, in fact, can be done without listening at all if the student has a basic understanding of the two temperature measurement systems.

One solution to all of these problems is to ensure that the test items do indeed test the skill intended. If a listening item can be done without listening to the passage, for example, then it should be revised.

Reading

The reading exercises are the single sentence, fill-in-the-blank type favored in this type of testing. The navigation for these items is a little smoother than in the listening items, since there are no unlinked "hotspots" and the "hint" button doesn't stay illuminated after use. The feedback is generally good, although a few feedback items don’t supply enough information for the user to understand fully why a choice is right or wrong.

Error Recognition

This is also a type of test item familiar not only to TOEIC takers, but TOEFL takers as well. In fact, the Alexis system uses the same interface as the TOEFL practice found online at Students International (formerly part of the Okanagan website, now at http://www.studint.net). This isn't surprising, since the same writers and designers at Okanagan University College created it. An example from the Students International site is below:

TOEFL Test Prep available at Students International. click on image to see a larger version

These types of items are particularly complicated to write, and the authors did a fairly good job with most of them. But, they slipped in a few places. For example:

Nobody, in spite of the warnings, didn't wear protective clothing while sterilizing the equipment.

Which underscored item "must be corrected or rewritten," as per the program instructions? Although it appears that the answer should be (C)"didn't", in fact, there is no easy way to rewrite or revise that one word to come up with a correct sentence. It is the entire verb phrase, "didn't wear," that needs correction. Thus, given the strictures of the test, I would answer (A) "Nobody", since it can easily be corrected or rewritten to produce a correct sentence, such as: "My friend, in spite of the warnings, didn't wear protective clothing while sterilizing the equipment." The feedback, however, indicates that C is the answer.

There are a few other items with similar problems. Given the difficulty of creating this type of item, it is especially important that each distractor be fully considered.

Reading Comprehension

In these activities, the user must interpret information presented in advertisements, signs, and posters. Unfortunately, these few readings are again filled with culturally-specific ideas, such as barn dances and St. Patrick's Day. Given the context of the TOEIC examination, these kinds of questions seem to miss the mark. They have little to do with international business, and are deeply embedded cultural constructs that are not contextualized for the user.

Additional Comments

Other features about this CD-ROM are notable:

1) The entire CD-ROM uses only 16 MB of space (a standard CD-ROM will hold up to 660 MB). The largest file is 332K, and there are only a very few files over 100K. The vast majority are smaller than 50K. It is unclear why this material is presented on a CD-ROM rather than a website with a CD-ROM option. (The CD-ROM option is a good feature for those who must pay local telephone charges for Internet access.) Given the expense of the CD-ROM, much more could have been done with the CD-ROM technology and available space.

2) A great number of the right answers are also the longest answers. This is a common flaw in distractor-writing. In fact, test-taking workshops tell students to be on the lookout for this feature.

3) There is a credit screen at the beginning of each item. However, it scrolls by so quickly that it is difficult to read it before the next screen comes up. There isn't a way to get to those credits other than to reload the page and read fast. If you read fast enough, you'll see that the items were written and designed by Brian Rhodes and his colleagues at Okanagan University College.

4) The focus of TOEIC questions is consistently on business and workplace English. This is not the case with the Alexis system, which more than not presents general English items more suitable for the TOEFL.(See the sample test at http://www.toeic.com.)

Conclusion

Test item writing is a lot harder than one might think. What are often dismissed as mere "multiple guess" tests are in fact extremely difficult to design from a number of standpoints. They require:

• distractors that tap into users' likely confusions,
• items that test specific skills and not cultural knowledge,
• items that test the skill intended, and not other skills (such as memory or mathematics), and
• answers that are clear-cut.

This is why organizations such as ETS have large groups of item writers and thorough systems of testing to determine item validity and reliability in testing.

Unfortunately, the Alexis system fails on a number of accounts, in ways that many "home-grown" practice test systems do: They don't appear to have taken into account the rigor that is required by good testing design. They don't seem to have done the type of validity and reliability tests on these items that one would hope for in published software.

For the number of activities and high quality sound, this CD-ROM will no doubt be useful for test-takers looking for practice. However, the responses and the feedback to many of the items, as well as the navigation, will probably confuse and frustrate them.

All "big tests" and test preparation books undergo constant revision. Items that are deemed insufficient are replaced. A companion website that published updated modules and offered additional content would improve the Alexis system greatly.

References

Gilford, S. (1996). A review of TOEIC. The Internet TESL Journal, Vol. II, No. 8, July.
http://www.aitech.ac.jp/~iteslj/Articles/Gilfert-TOEIC.html