version 1.0 (12.2.01)

[Instant Environment]

Part 4: Evaluation of prototype

Fall Semester 2001

Team 10

Results of the evaluation

Evaluation Exercise 1

Evaluation Exercise 2

Evaluation Exercise 3
Overall Evaluation and Recommendations
Critique of Evaluation Plan
Appendices

Results of the evaluation

Evaluation Exercise 1 - Cognitive Walkthrough

Criteria being evaluated:

Our main goal with using the cognitive walkthrough evaluation is to examine the overall learnability and usability of the interface. More specifically, however, the cognitive walkthrough allows us to examine the observability and predictability of the interface.

It is important that the sequence of actions for necessary for invoking a desired action is obvious to both novice and expert users. That means that the proper inputs must be clear and visible, and the current state of the system must be clear to the user. Humans have a short attention span and can easily be distracted. In the event that the system is left idle in a particular state or the user is distracted during a task, the user must be able to return to the interface and recognize the current state of the system.

In addition to making the interface observable, the system should also afford the particular desired commands and their output must be one that the user expects. For example, the user wants to add a light to a macro. The user must know that he/she must select the button “ADD DEVICE.” Then the user expects some way of telling the system which particular light he/she would like to add. The system must provide this type of feedback to the user.

Description of evaluation exercise:

3 HCI experts were chosen from CS 4750 HCI class. The project, problem domain, system, and its capabilities were described to the evaluators. The evaluators were then given a description of task in which they are to perform and evaluate. Each evaluator was then given a questionnaire detailing each step in the task that they are going to perform. After the completion of each step, the questionnaire calls for the user to answer the four usability questions of a cognitive walkthrough:
1. Will the user be trying to produce whatever effect the action has?
2. Will the user be able to notice that the correct action is available?
3. Once the user finds the correct action at the interface, will she know that it is the right one for the effect she is trying to produce?
4. After the action is taken, will the user understand the feedback given?
However, instead of answering the questions with a simple “yes” or “no,” the evaluators were asked to select a number on a Likert Scale, 0 being absolutely “no” and 5 being absolutely “yes.” The evaluators were also encouraged to give comments and feedback regarding particular usability problems.

Results of exercise:

Summary of Results:

The most common usability problem that the evaluators uncovered involved the naming and semantics used in the interface. Example: a novice user may not understand what a “macro” represents. Also the interface uses common computer lingo as the theme for menu selections such as “Run” and “Edit.” Though these menu choices are familiar to computer savvy users, they may not be so clear to others. Another problem arising from the naming of the menu functions results in ambiguity. The names or context in which the menu selections are presented may not provide the user with enough information in order to invoke the intended action. “Do I hit ‘Edit’ to edit a macro?” Another common usability problem that the evaluators seemed to agree on in general had to do with menu actions and feedback. Many of the actions result in responses that are subtle and may not be obvious to the user. The interface does not make any attempt to notify the user as to recent changes made to the screen. Example: A user adds a new macro and the name of the new macro appears in the menu, but does the user notice it has been added right away?

Discussion of Results:

With regards to the given task that was evaluated, the interface scored low in observability. The evaluators found the various indicators, menus, feedback, and labels to be misleading as to informing them which state they were currently in. After performing an action, it was sometimes not clear if the action was performed correctly, and the next step to be performed may also have been ambiguous. The observability issues influenced the system’s predictability. When a user is not sure of the state of the current system, it is difficult what to expect next. This relationship between the two criteria makes it difficult to analyze only the system’s predictability. However, the evaluators have provided some insight into this particular issue. The main problem with the predictability of this particular interface arises from the naming and labeling of the buttons. The evaluators were not sure of the exact meaning of the buttons. However, after making a determination of the correct button to press, the feedback given by various menus was one that the evaluators generally agreed were appropriate with a few exceptions. One issue was the label “Current Macro.” This label was intended to provide the user with the information of the “currently running macro,” but confused the evaluators when they were editing one macro while the label indicated another. Another evaluator made a strong argument regarding the sequence of screens that appear while adding a new macro. The evaluator recommended that the system automatically go into “Editing a macro” when a new one is created (i.e. ‘New Macro Wizard’). This would reduce the number of steps the user would have to perform and also provide a more predictable series of feedback. In conclusion, the interface needs much improvement in terms of labeling and naming and the behavior stands fairly well with only a few minor changes.

Evaluation Exercise 2: Heuristic Evaluation

Criteria being evaluated:
We intend to perform a heuristic evaluation on our system in order to discover potential usability problems. While our system is not in the early stages of development, we still feel this type of evaluation is appropriate because uncovering usability problems will benefit future projects like ours.

A heuristic evaluation examines multiple usability criteria including:
1. Observability
2. Responsiveness
3. Familiarity
4. Recoverability
5. Consistency
6. Generalizability

In addition to these basic usability criteria, a heuristic evaluation also examines other advantages or disadvantages of the system. The evaluation also encourages recognition over recall, a minimal design, and help and documentation.

Description of evaluation exercise:
A heuristic evaluation is an excellent way to uncover bugs in a user interface. In addition to discovering bugs, a heuristic evaluation provides a method for assessing the severity of each bug so that the development team can decide what to issues require resolution. A heuristic evaluation allows evaluators that are unfamiliar with the system and requires few evaluators. According to Neilson’s research, five evaluators provide the optimal results. The development team devises a set of heuristic guidelines and each evaluator independently tests the system using the guidelines. After they are finished the problems are discussed amongst the evaluators and the severity of each bug is determined.

For this evaluation, there were five evaluators from outside the team of developers. All are expert computer users and familiar with many programs and user interfaces. The prototypes used for the evaluation where the storyboards and the limited functionality prototype. The heuristics used were those determined in part 3. They were taken from Human-Computer Interaction page 414.

1. Visibility of system status
2. Match between system and real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help user recognize, diagnose and recover form errors
10. Help and documentation

Also we used Nielson’s severity rating in the assessment of our system.

0 = I don't agree that this is a usability problem at all
1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority
4 = Usability catastrophe: imperative to fix this before product can be released

Results of exercise:

The problems discovered and their severity rating is listed below.

	Description of Problem	Severity
1	Symbols for lights, blinds, and other appliances are not obvious, until the user becomes familiar with the system.	2
2	It is not clear what the buttons on the top right are for. It seems like you should be able to touch on the devices on the diagram to change their status.	2
3	It is not clear what the buttons on the bottom right are for. Once you use the system, it becomes obvious, but initially their effect is unclear.	2
4	There is no undo button anywhere, ever!	3
5	If a user wishes to manually turn off a Conventional device, they are required to manually turn it back on before the system has control of it again.	1
6	There is no help button!	4
7	There is no way for the user to determine if a device is under the control of the system.	4
8	There is no way to turn all the lights, appliances, or blinds on or off in a room, on a floor, or in the whole home, without selecting each of them.	4
9	Does not accommodate the blind or visually impaired.	1
10	The sliders on the temperature and dim controls have no arrows at the top or bottom, which could make them potentially difficult to operate.	3
11	The diagram is unclear how many devices are in one location. For example, a TV, VCR, and Stereo system could all be in the exact same place, and it is not clear if they are all on, or which ones are on.	3
12	If a user inadvertently exits the edit or new macro modes, they should be prompted to save their changes.	3
13	There are no accelerators for expert use.	1
14	Buttons for typing in new macro names and for selecting devices are too small to be precisely touched.	4
15	When a user creates a new macro, they might assume that the initial settings would be the same as whatever old macro was highlighted at the time.	2
16	When the user is selecting devices, there is no way to select more than one. This will cost the user valuable time having to select many devices individually.	3
17	In the edit macro mode, adding a device does not make much sense. It seems like all devices in the house would be included and each would need to be set to on or off.	3
18	If a device has not even been added to a macro, there needs to be some sort of visual indication on the diagram.	3
19	When the user is manually controlling the devices in the home, once they have reached their goal, they should be able to “save as a new macro” their settings.	4
20	When in the manual mode, the current macro should not display.	2

Summary of major usability problems (severity rating of three or higher) and possible solutions

4. Lack of undo functionality – This is fixed by simply adding an undo button to the static portion of the menu. It will require additional code work, but is a useful feature.
6. There is no help button – This is fixed by simply adding a help button to the static portion of the menu. It will require additional coding and careful design, but is a useful feature.
7. There is no way for the user to determine if a device is under the control of the system. – Lights and appliances that have been turned off through the conventional means provided by these devices should display on the diagram so that the user will recognize that these devices are not under their control.
8. There is no way to turn all the lights, appliances, or blinds on or off in a room, on a floor, or in the whole home, without selecting each of them. – In the lights, appliances, and blinds menus in addition to buttons for each individual device, there should be buttons to turn off all devices, turn on all devices, and close or open all blinds.
10. The sliders on the temperature and dim controls have no arrows at the top or bottom, which could make them potentially difficult to operate. – Simply add arrows at the top and bottom of the sliders allowing the user greater precision and easier use.
11. The diagram is unclear how many devices are in one location. – Overlap the symbols for the devices so that it is obvious where multiple devices are.
12. If a user inadvertently exits the edit or new macro modes, they should be prompted to save their changes. – Add code to prompt the user to save their changes when the exit the edit macro or new macro modes.
14. Buttons for typing in new macro names and for selecting devices are too small to be precisely touched. - The size of buttons that are to be touched should be increased so that they are all large enough to precisely touch.
16. When the user is selecting devices, there is no way to select more than one. This will cost the user valuable time having to select many devices individually. – The user should be able to select multiple devices, and remove them by touching them again. Then the user should be taken through the settings for each device. This will cut the time needed to create or edit a macro.
17. In the edit macro mode, adding a device does not make much sense. It seems like all devices in the house would be included and each would need to be set to on or off. – When the user creates a new macro, add all the devices to the macro, and then allow the user to turn them on or off.
18. If a device has not even been added to a macro, there needs to be some sort of visual indication on the diagram. – If a macro is not controlling one of the devices in the home, it should appear differently from the devices that have been set to off by the system.
19. When the user is manually controlling the devices in the home, once they have reached their goal, they should be able to “save as a new macro” their settings. – Simply add a button and a little code to accomplish this.

Evaluation Exercise 3: Think Aloud Cooperative Evaluation

Criteria being evaluated:

With the cooperative evaluation we were trying to see how the system would function when the users were put in control. We wanted to gather data on the learnability and robustness of the system. Predictability and synthesizability of the system were the two main focuses of learnability. This information was gathered from watching the user interact with the system and from talking to them about what helped them make choices on what to do. Robustness was examined by paying attention to observability, recoverability and responsiveness of the system while the user executed their task and if an issue was discovered, it too was talked about.

We desire the system to be as easy to learn to use as possible and robust enough to keep using. By watching the user we were able to understand how they learned to use the system and what past experiences and visual cues aided them in learning the system. We also looked for signs of frustration if the system responded too slowly or if they got stuck at a point and could not recover from it.

Description of evaluation exercise:

{The procedure was changed a bit from the phase 3 turnin.}
10 participants were chosen from people who, from a short description of the project, were interested in the technology. The users were asked to sit down at a computer running our system with one of project team. The team member again gave a short description of the system and explained that he was going to ask the user to perform 10 tasks and give feedback during and after each one and then answer a few follow up questions at the end of all the tasks. The examiner then asked the user to perform each of the following tasks:
  1. Look at various floors and look at the various appliances
  2. Click on the different types of item functionality
  3. Run different macros and notice changes to the system
  4. Create and name a new macro
  5. Modify an existing device in the macro you just created
  6. Add device Lobby Blinds to the newly created macro
  7. Save the new Macro
  8. Run the new Macro
  9. Edit the newly created macro again and delete the Lobby Blinds device
  10. Delete the newly created macro

During each of the task the evaluator took notes on the participant’s ability to navigate the system and perform the tasks. He also asked questions of the user about why he chose the procedure he did to perform the task, how easy the user thought it was to perform the task and whether or not the steps needed to perform the task were intuitive.

At the end of all of the tasks, the user was asked the following questions:
  Do you currently use any home automation systems?
  Do you have any problems w/ your system?
  What is your living environment?
    Roommates:
    Type of home:
  Was our system easy to use and learn?
  Was the overall system intuitive?
  What problems if any did you have interpreting or using the system?
  What are the advantages / disadvantages of this system?
  Did you find the system to be useful?
  What did you like best about the system?
  Any additional comments on the system design?

Results of the evaluation:

The evaluation went very successfully. The users, overall, seemed to be very impressed and intrigued by our system. The users were able to complete most of the tasks without any guidance at all and were able to do the task very rapidly. When asked why they chose what they did to complete the task, most stated that the correct path was obvious. The biggest problem area that was found was in the layout of the menus and the naming of the buttons that existed in the menus. Users mentioned that the definition of macros should have been defined better in the pre-task discussion and that the macro button should be accentuated from the rest of the buttons in the main menu (be it, move the button or change its color, etc.). There were no observable problems with system response time and the users said that it was very easy to recognize the state of the system by the colors and shapes. They did recommend that the colors could contrast a bit more and that some form of labeling for the icons on the screen could be introduced so as to know what the actual device was. A sound played in response to user actions would have been an especially useful method of delivering feedback. They also mentioned that drag and drop devices would be nice for placing new elements. On this comment it was explained that the system was not a complete representation so that aspect (of adding new items) was not built into this prototype, but may make it to a final production if there ever was one. One user mentioned that the system was easy enough for anyone who could program a VCR but an instruction manual would make the system easy even for a person who couldn’t program a VCR. Many users commented that they liked the graphical user interface and mentioned that they system got even easier to use after you played with it for a couple of minutes (which many of them continued to do after the evaluation).

To sum up via the criteria, the evaluation results came out like this:
Learnability:
  Predictability: The users pretty much all agreed that the system seemed to perform the way they would have anticipated it to perform, thusly very intuitive and predictable.
  Synthesizability: The user was able to very quickly learn how to manipulate the system and understand the constructs of its design. He was then able to apply that knowledge in order to progress through the system. One user even stated that it got easier to use the more you used it.
Robustness:
  Observability: The users were able to recognize the different icons and textual labels on the screen and interpret both the state of devices and the state of the overall system without problems.
  Recoverability: Two users made errors, one clicked the wrong button (knowing the correct button, just having the mouse slip) and the other clicked ‘ADD’ when told to edit a macro. By the noticing the state of the system he quickly realized (without any prompting from the evaluator) that he was in the wrong subsystem and was able to easy find his way back to the main menu in one click.
  Responsiveness: The were no observable problems with system response time and no user comments on it either.

Overall evaluation and Recommendations

Usability criteria:

The cumulative effect of the evaluation exercises is that we discovered a great many usability enhancements that would not have been obvious to our own group members had we evaluated the prototype internally. Each evaluation method provided us with highly relevant and important insight into the success of our original design criteria, and offered practical suggestions for future changes in the prototype. Though there was disagreement between the results of some of our evaluations, the process did delineate several common shortcomings of the prototype (such as the lack of help or undo functionality). The evaluations did prove especially useful in creating a list of changes that would be necessary before any further version of the program was released for testing.

While the evaluation techniques are designed to test a range of usability principles, the following sections discuss our results in regard to our original and most important criteria drawn from our assessment of our goals and the user population.

Primary criteria:

Predictability:

There were conflicting results from the evaluations about how well the prototype fulfilled this requirement. The heuristic evaluation and cognitive walkthrough gave results that indicate severe shortcomings with predictability; evaluators often commented that, even after using the system for some period of time, it was difficult to tell what the result of certain actions would be due to confusion semantics and a lack of feedback. The labels for showing the current macro and the current mode, while designed to provide useful information, in actuality ended up confusing evaluators by making it unclear how there actions were really changing the system. In short, the heuristic and cognitive walkthrough evaluations seemed to show that users were somewhat unhappy with the predictability of the interface, primarily due to confusing labeling and a lack of certain information on the floor plan display.

The think aloud cooperative evaluation gave somewhat different results, however. Users, even those with minimal home automation or computer experience, needed little to no prompting to predict the effects of their actions in their prototypes. Some users even attempted to perform many of the tasks without being directed to do so, and while some did comment about the confusing nature of the labeling and semantics, most did not seem to encounter any major problems in understanding and predicting the path through the prototype.

With mixed results, we conclude that further testing for predictability is necessary. The fact that significant problems did arise in the heuristic and walkthrough evaluations suggests that perhaps the cooperative evaluation created a situation favorable to overlooking major shortcomings or allowed the user to predict results based on the verbal dialog with the collaborator rather than the prototype. In conclusion, we feel that specific issues in terms of predictability should be corrected, but additional evaluations should be performed to confirm that end users indeed find the prototype acceptable in this regard.

Responsiveness:

While the prototype was not implemented in the intended environment, the interface was designed to simulate a wall mounted touch panel display as closely as possible. The fact that we received almost no feedback about responsiveness is both positive and negative. Our prototype’s interface apparently responded with appropriate timeliness, but in the actual environment there may well have been delays while interfacing with the central computer system in the house and further delays in seeing the actual changes performed in the environment. The prototype was mainly a shell of the interface and did not perform any manner of meaningful tasks in regard to automation of a system. Hence, we feel that occasional random delays simulating slowdowns in a fully implemented would make the prototype more useful for testing this criterion. Further prototypes should be better matched to the actual delays involved with a touch panel interface and a networked system.

Dialog initiative:

While this original usability criterion was deemed of less importance by later work in developing the system, our evaluations did shed light on this issue. The users did not always feel completely free to initiate communication with the system; often they commented that it was unclear what state they were in, what devices were being controlled, and what the symbols on the interface meant. This led to a situation in which the system and not the user primarily dominated the flow of communication. Users, who had no undo button in the prototype, were left to interpret the responses from the program rather than pre-empting it in a fashion that would be more realistic to an actual home automation environment.

Recoverability:

There was agreement among all evaluation methods that there were severe shortcomings in the area of recoverability. There was no undo function at all in prototype, a fact which nearly all reviews commented needed to be corrected. In addition, there was no confirmation before completing actions such as deleting a device or macro. Since there were no user authentication or password controls included, it is conceivable that a random user may accidentally bump into the interface or touch an area he did not intend to, thereby deleting a macro which may have taken significant time to program. This is a serious shortcoming in the prototype, and is the highest priority for any further work on the interface.

Secondary Criteria:

Familiarity:

All evaluation methods received distinct suggestions and feedback about the familiarity of the systems. Several reviewers mentioned that the items such as “Run” and “Edit” were familiar from other Windows applications, while other reviewers commented that those same labels were confusing within the current context. Many evaluators had difficulty understanding the idea of a macro in home automation; nothing they had commonly used in their own experiences seemed similar, and most homes do not have anything even remotely equivalent. The symbols used on the floor plan for the lights, blinds, and so forth were not immediately clear and there was not enough relation between their chosen shape and coloring and their real-world counterparts. The naming and semantics chosen for the labels and buttons did not always seem naturally intuitive to the users, and bore little relation to other concepts they were familiar with. Finally, there was no help feature included, a fact with most reviewers also commented on; this made the task of becoming familiar with the system even more difficult.

However, most reviewers (especially those in the think aloud cooperative evaluation) did comment that the visual floor plan was a very beneficial and useful aspect of the interface. Assuming it corresponded directly to the layout of their actual house, most users agreed that this was not only a useful feature but conducive to learnability and manipulation of the interface without previous experience.

Recommended Changes:

Based on the results of our evaluations, we feel the following the feedback warranted the following changes to the interface:

  1) Better labeling of buttons, labels, menu items, and popups to convey a more accurate sense of what its function is
  2) Addition of a help system or context-sensitive help to provide information about the functions of interface features
  3) Addition of confirmation messages whenever deleting a device or macro
  4) Clearer portrayal of the appliances, lights, etc. in the visual floor plan. Possibly have the device labeled or have a small bit of text appear when the device is touched
  5) Random delays in performing certain tasks to better simulate the actual environment our project would be deployed in
  6) Authentication of users, possibly by a password or via biometrics if feasible. This would keep small children from turning dangerous appliances on and prevent visitors to the house from corrupting the macro setup.
  7) Better contrast of colors
  8) When creating a macro, have all devices automatically added to the new macro and simply allow the user to turn them on and off
  9) Individual control of devices by touching them in the floor plan diagram and editing them
  10) Larger keypad buttons and a more accurate keyboard display for entering new macro names
  11) Allow users to save a macro based on the current settings in the floor plan display

Critique of Evaluation Plan

The empirical data gather from the questionnaires in the cognitive walkthrough evaluation was almost completely useless. The Likert scale for each question only made the data more ambiguous and difficult to parse. The evaluators found the numbers to be somewhat arbitrary. For example, what is the difference between a 4 and a 5? Between a 3 and a 4? What also contributed to a relatively bad data sample was allowing each evaluator to evaluate the interface on their own. One evaluator would rate one question at a 2 while another would rate it at a 5. Another attribute that damaged the validity of the evaluation task was having only one piece of functionality to evaluate. Lastly, the questionnaire was a bit too tidious and time consuming. The evaluators were subject to losing their focus and arbitrarily choosing answers for each question. Relatively invalid data and not a large enough data sample made it impossible to formulate any conclusions from the empirical data. Luckily the evaluators were encouraged to make constructive comments regarding any usability issues pertaining to the task and questions at hand. These comments were supportive of one another and could be logically justified. This particular evaluation was based purely off of the comments that the evaluators provided. In the future, this evaluation would best be performed with the evaluators simultaneously commenting on each step. The evaluators could concur with one another and agree on specific issues. This method would make the data more conclusive and meaningful. Also, the data would be easier to gather given that it was coming from one direct source and one could simply record it.

Where the cognitive walkthrough failed in providing meaningful empirical data, the heuristic evaluation prevailed. The data was gathered in the form of a collective agreement between several HCI experts rather than separate collections of data. The data therefore is more conclusive since it stands as something the evaluators discussed among themselves and agreed upon. The scale given was also precisely defined with little ambiguity whereas the scale in the cognitive walkthrough was hardly defined at all. The data gather is meaningful and provides insight into usability issues. However, the scope of the evaluation was far too broad examining too many usability criteria. The problems that were discovered represent constructive criticism, but are somewhat sporadic jumping from one criterion to another. No one criterion was deeply examined and no concrete conclusions were made regarding the criteria that were to be examined. This evaluation technique promises great potential for the interface used in this project. Perhaps using a heuristic evaluation while focusing in depth on a few criteria would prove very effective. In the end the heuristic evaluation served its purpose in uncovering many unforseen usability bugs.

The cooperative evaluation concluded in results that conflict those of the cognitive walkthrough. It is believed that one reason could be the design differences of the two evaluations. In the cognitive walkthrough, the evaluators were forced to comment on issues directly regarding the criteria specifically outlined by the four questions or the “believability story.” However in the cooperative evaluation, the subject may be victim to subconsciously overlooking issues or troubles. Still the results must be respected and further investigation must be performed as to why the two evaluations derived different results. In general, it is good that there exists some overlap among the evaluations while each should still have its own distinct focus. The cooperative evaluation also falls victim to many different data sources. One subject may comment on the difficulty of one feature while another may revel its ease of use. Who do you believe? Each subject has their own opinion and the resulting data gather may not reveal any one or few particular issues. Perhaps a better-designed experiment would call for only a few tasks to be performed during the evaluation. These tasks would then be more concretely outlined, for example, numbering and labeling each subtask. The users would then be asked specific questions about specific usability criteria in response to their actions.

Overall the evaluations provided an excellent way of discovering usability issues. However, the evaluations were not designed to work in synergy and did not point to specific conclusive evidence. The evaluations can be better designed in the future to provide less overlap and more depth into the desired criteria. Rather these evaluations uncovered a new set of many possibilities that need further exploration. Only having a partially functioning prototype both in function and in form also hurt the evaluation results. A touch screen could not be obtained so the evaluations were done on a PC with a mouse as the pointing device. This model was not representative of how the user would actually interact with the invisioned system. The interface's limited functionality inhibited the user from freely exploring the interface's possibilities. At times they were told, "That part doesn't work" or "Don't use that part." This limitation constricted the results that could be obtained. With a tighter design and a fully functional interface prototype, the evaluations would have resulted in more specific and precise conclusions. The system could then be redesigned to prove more usable to the users.

Appendices

1. Cognitive walkthrough questionnaire

2. Cooperative Evaluation Results: hciteam10ques.htm

Link to this Page

Team 10 last edited on 29 November 2001 at 9:30:41 pm by adsl-81-152-225.asm.bellsouth.net