RGA Project Part 4
Results of the evaluation
Evaluation Exercise 1
Cooperative Evaluation
Criteria Being Evaluated
The criteria being evaluated by the cooperative evaluation are familiarity and recoverability.
Description of Evaluation Exercise
The evaluation is performed using participants that are representative of a typical user. They are either college students are business professionals who have experience with handheld devices. Specifically, the cooperative evaluation is performed on 5 users. The user is asked to say anything that they are thinking while performing the task. The user's task is to start the restaurant finder application, enter search criteria for cuisine, price range, and location. When the user has finished, they are to search, and then select a restaurant. For the selected restaurant, the user is to get directions and make reservations. Special attention is paid to the user's ability to recover from mistakes, and the learning curve for using the application (due to familiarity with other interfaces and processes).
Results of Exercise
Many of the users were delighted to try the restaurant finder application due to the novelty of the device. None of the users had a good understanding of how to write Graffiti characters, so most of them used the virtual keyboard to input characters. All users were able to easily search by their criteria and find directions. The users frequently had problems selecting the right button to make reservations on their first guess. The developer's reasoning for placing the "Make Reservations" function inside of the "Wait Time" function is that making reservations would almost always only occur after seeing a lengthy wait time. Plus, the developer believes that making reservations would be less frequently used feature, therefore it should not be placed on the main results screen. Users felt that the search interface was familiar since it represented a conventional search engine, and all users were able to correct mistakes that they made. Overall, the users felt that the restaurant finder application was a useful tool.
Evaluation Exercise 2
Heuristic Evaluation
Criteria
Each of our usability criteria were evaluated in this evaluation exercise. The Heuristic Evaluation and the six guidelines we selected allowed us to evaluate the familiarity of the system, the recoverability of the system and the customizability of the system.
Description
Each of our group members performed a Heuristic Evaluation on our system. The design team performed this evaluation due to time constraints and constraints of resources. Additionally, it was appropriate for the design team to perform the heuristic evaluation since each member is representative of the target user population. Each member received a form with the six chosen guidelines and a copy of the storyboard. They ran through the storyboard while assessing the six guidelines set out on the form. The guidelines chosen were as follows:
· Match Between System and the real world
· User Control and Freedom
· Consistency and Standard
· Error Prevention
· Flexibility and efficiency of use
· Help users recognize, diagnose and recover from errors
Results
After each sections results are compiled a severity rating based on frequency, impact and persistence will be assigned to any violations of the heuristics that are found ranging from not severe at all to extremely severe. The overall results to the examination of the guidelines in the context of the storyboard went as follows:
1) Match Between System and the real world
Each group member agreed that the system appropriately resembled systems in the real world. Everything from the search fields to the menus that were presented by the system were presented in such a manner that would be familiar to all potential users. This guidelines shows familiarity with the system that aids in learnability
2) User Control and Freedom
The evaluators found no dead ends based on the storyboard and felt they were able to move about the system without surprises. One evaluator saw a problem arise of if the search produced a list of restaurants that did not contain a desired restaurant. This violation could be considered to have moderate severity. It could result in user frustration with the system and essentially means that the purpose of the system is not fulfilled. However, with a large database of results it will most likely occur quite infrequently This guideline shows recoverability of the system.
3) Consistency and Standard
All of the evaluators found that all buttons, fields and menus, etc, were consistent and standard throughout the system. However, one evaluator pointed out that in the search fields all the information was not entered in the same way. For example, some were text lines and some were pull down menus. This problem is considered not severe at all. Uers will be familiar with entering information into empty fileds in search engines as well as clicking on buttons and icons. This guideline shows familiarity with the system.
4) Error Prevention
The evaluators each found that the storyboard did not appropriately show how the system might accomplish this task. This problem has minor severity.
5) Flexibility and efficiency of use
The system is flexible and efficient in that it allows experienced users to maintain a favorites list or possibly have the system predict what the user would like based on a history. This guidelines shows customizability of the system.
6) Help users recognize, diagnose and recover from errors
Error messages were not strictly present in the storyboard. Lack of error message presents a problem that is moderately severe. However, the evaluators found that the ubiquitous presence of back buttons and the search again button allowed the user to go back to a previous screen or start over if they perceived that they made an error. This guideline shows recoverability of the system.
Evaluation Exercise 3
Cognitive Walkthrough
Criteria:
The main criteria of this evaluation was that of familiarity. The cognitive walkthrough is designed to explore learnability through exploration and for this reason it is most likely to detect usability problems that relate to the familiarity of the system.
Description:
Each of our group members performed a cognitive walkthrough on the limited functionality prototype. The cognitive walkthrough was performed in this manner due to limited time and resources. Each evaluator had to answer the 4 cognitive walkthrough questions on a list of the following seven actions (an action sequence that would be common for users):
A1. Fill in criteria of price using pen to write price.
R1. Screen displays acceptable price range.
A2. Select a cuisine type from a pull-down menu.
R2. Screen displays selected cuisine type.
A3. Use pen to select search icon.
R3. Search results are listed, along with several options on the right portion of the screen, ncluding details, menu, directions, wait time, and order.
A4. Select restaurant of choice.
R4. Restaurant of choice is highlighted.
A5. Select menu.
R5. The restaurant's menu is displayed.
A6. Select "back."
R6. The restaurant list is again displayed, along with the above mentioned options.
A7. Select "directions."
R7. The screen shows a map, along with text directions to the restaurant.
Each evaluator used the evaluation forms as provided in Appendex "X". These forms provided a place for the user to answer the four cognitive walkthrough questions for each each action. It also provide a place on a separate form to discuss any negative results that were obtained.
Results:
The results of the cognitive walkthrough provided the group with several potential usability bugs mostly stemming from the inherent knowledge of a PalmOS based handheld that was required to use the prototype. The results for each action are as follows:
Action 1: The evaluators agreed that the users will be able to determine how to fill in the criteria of a price except that if the user has not been exposed to the palm before they may not know that they need to select the price field before they can enter a price in via graffiti. This was rated to be a mild severity problem as the target group is expected to have experience with PDA's and even a novice user could probably quickly figure this out.
Action 2: The evaluators all agreed that selecting a cuisine from the pull down menu was completely obvious in both action and results.
Action 3: The evaluators all agreed that clicking on the search button to initiate a search was obvious in both action and results.
Action 4: The evaluators all agreed that selecting the restaurant of choice was obvious in both action and results.
Action 5: The evaluators found two minor problems with the select menu button action. One is that it is not completely obvious that you need to have a restaurant selected before clicking on the menu button. This is of minor severity as a restaurant is always selected by default and the user would quickly be able to determine that they needed to select the restaurant before clicking menu.
The other problem is that some users could believe that they have to click on the menu button and then the search again button. Since the search again button is in the same place as the search button on the previous page it suggests that you make some modifications on the results screen and search again to display more information about the restaurant. This is of very minor severity since it is obvious in any case to press the menu button first which will immediately bring you to the menu screen for that restaurant. This would be allow novice users to determine the correct action after only trying it once.
Action 6: The evaluators all agreed that selecting the back button was obvious both in action and results.
Action 7: The evaluators found three minor problems with the select directions button action. One is that it is not completely obvious that you need to have a restaurant selected before clicking on the directions button. This is of minor severity as a restaurant is always selected by default and the user would quickly be able to determine that they needed to select the restaurant before clicking directions.
The next problem is that some users could believe that they to click on the directions button and then the search again button. Since the search again button is in the same place as the search button on the previous page it lends itself that you make some modifications on the results screen and search again. This is of very minor severity since it is obvious in any case to press the directions button first which will immediately bring you to the directions screen for that restaurant. This would be allow novice users to determine the correct action after only trying it once.
The last problem is on the directions screen the map doesn't indicate the the red line is a route or which end is the stop/start. In a familiar situation this is okay as the user will know where they are and can determine which end of the line is the beginning. In a foreign city they will not have this luxury and could easily be confused as to which end of the line was start. This is a fairly serious problem and could defeat the purpose of having directions in the first place.
Overall evaluation and Recommendations
The cooperative evaluation brought to light a few issues in the button placement and naming on the search results screen. The users had trouble determining how to make reservation as this feature of the system is under the Wait Time screen. All the users were unfamiliar with the Grafiitti handwriting recognition system an as a result used the virtual keyboard but in this case they were still able to use the full functionality of the system.
The heuristic evaluation showed that there were a few minor consistency issues in the interface. Overall it showed that the storyboard was lacking in a complete discussion of the system that could possibility be built. This is both a problem based on the fact that the storyboard is not a functional prototype and also in that some areas of the system needed more detail.
The cognitive walkthrough made the designer very aware of how dependent using the prototype and system is on knowledge of the PalmOS. The system is dependent on the grafitti handwriting system and on the UI elements that make up the palm interface as some of these are fairly different than a desktop UI. The main difference is that text fields are lines instead of boxes, something that could confuse a novice user. The cognitive walkthrough also showed how familiarity with web based systems would be a major plus in using our system as it is based on the UI of a web based search engine. One final point is that the cognitive walkthrough discovered a few minor usability bugs in the search results screen which would mainly be a problem for new users but could be improved upon.
In terms of the usability criteria in question, (familiarity, recoverability, and customizability) the evaluations generally show that the prototype performs well. All three evaluation techniques indicate that the system is extremely familiar to users. It is similar to internet search engines and has familiar concepts like entering text into an empty field and selecting parts of the display that have the look and feel of buttons. It also has similarities to other PalmOS applications in the use of the stylus and Graffiti. The only problem that these evaluations indicated for familiarity would occur with those not acquainted with PDAs. However, those individuals are not part of our targeted user population. The cooperative evaluation and the heuristic evaluation addressed the recoverability of the system. This recoverability is more than adequate in terms of back buttons and the option to search again. However, the system fails in that it does not address the need for error messages and it does not allow the user to recover if no acceptable restaurants can be found. The heuristic evaluation addressed the criterion of customizability and found that there is adequate ability to customize the system through offering manual entry of favorites and a memory of frequent searches and search styles.
The main changes that we see to the system would be in the realm of some simple UI improvements that would help the user determine the next action required by the system. These center around possible improvements to the search results screen to remove any ambiguity, and possible changes to the search screen to help new users who are not familiar with the PalmOS interface conventions. Specifically, the search results screen should include a renaming of the "search again" button to "modify search." Additional improvements would be to rename the detail button to possibly restaurant info, as this would provide a clearer picture as to what was on that screen. Also it may be required to move the make reservation button to a more noticeable location to ease the process of making a reservation. Additionally, error messages should be developed for situations in which users enter information outside the parameters of the search criteria. A better direction screen also needs to be developed. This screen could offer a clearer map indicating starting location and destination and also should have a zoom feature so that the user can have a better view of their route. To aid in the directions and in solving other usability issues, several features could be explored for this system. For example, to provide more accurate information on directions, the integration of GPS technology should be explored and evaluated. To alleviate the possibility that an acceptable restaurant will not be identified by the system, wireless internet technology should be considered for the importation of a larger number of restaurants for cities that are frequently searched.
Critique of Evaluation Plan
The Evaluation Plans were each defined very clearly and quite thoroughly. Each of our evaluators found no problems in following the plan when they were asked to perform an evaluation on a specific representation of our prototype. Our knowledge of the evaluation techniques and our prototypes were more than sufficient for this part of the project. However, a problem that was encountered would have been with the Heuristic Evaluation being based on the storyboard of the system and it was only a small problem. The storyboard did not or was unable to show some features that were important to answering the guidelines outlined in Heuristic Evaluation form. For example, it was difficult to evaluate the customizability criterion with no access to this feature. It was impossible to adequately evaluate the heuristic of error prevention using the storyboard. It might have been more informative to run the Heuristic Evaluation on the functional prototype of the system. If this had been the case, all the features would have been present and not even minor inconveniences would have been encountered. There were also possible confounds in the evaluation based on the fact that the design team performed two of the evaluation techniques. There is a possibility that using one evaluation technique biased the second evaluation. We attempted to correct for this problem by counterbalancing the evaluations, with two design team members performing the heuristic evaluation followed by the cognitive walkthrough and with the other two members performing the evaluations in reverse order. However, we realize that some bias still exists. Ideally, with more time, the evaluations would have been performed by two different groups of HCI experts.
For the most part, each of the evaluators found a clear and easy path between the evaluation tasks and the criteria we were attempting to measure. We were able to confidently assess the usability criteria of familiarity and recoverability, but we had more difficulty in determining the customizability of the prototype. The only evaluation of this criterion was one of the heuristics and as previously mentioned, it was difficult to perform the heuristic evaluation on the storyboard. Customizability would have been easier to determine if this aspect of the system had actually been included in the functionality of the functional prototype and evaluated as part of the heuristic evaluation and possibly even the cooperative evaluation. Additionally, it would have been beneficial to perform the cooperative evaluation and the cognitive walkthrough on more than one action sequence since there are a myriad of action sequences possible with this system. Increasing the action sequences evaluated would most likely display more usability problems, especially in terms of recoverability. Additionally, there was the problem of not being able to evaluate the problem of a user not finding an acceptable restaurant choice. However, this problem is extremely difficult to evaluate because it would essentially require a fully funstional prototype with all available restaurant choices included, something we did not have the resources to develop. It would also require us to have participants in an evaluation that could not find restaurants that they found acceptable. With a large number of restaurants included in the system this would likely be a rare occurrence.
Appendices
Cooperative Evaluation Result Data: CooperativeEvalData.doc
Cognitive Walkthrough Evaluation Form: CognitiveWalkEvaluation.doc
Cognitive Walkthrough Result Data: CognitiveWalkthroughData.doc
Heuristic Evaluation Result Data: HeuristicEvalData.doc
Link to this Page