InformalPrototype:Power Ranger

From IEOR 170 Spring 2007

[edit] 1. Introduction and Mission Statement

In this work we evaluate a low-fidelity prototype of a distributed power-consumption monitoring system, the Power Ranger 2000. The purpose of these tests is to be able to determine the best way of presenting data to users in an organized, easy-to-find manner. By performing tests on real subjects with low-fidelity paper prototypes, we are able to discard bad designs and refine our interface without incurring much cost or wasting time with bad high-fidelity prototypes or final products. The motivation for our project falls in two categories, economics and the environment. In terms of economics, power prices are increasing and while consumers can immediately see aggregate effects of power consumption on their monthly utilities bill, they have no way of knowing exactly what has used most of the energy billed to them. Along with the economic costs, the environmental cost of running appliances is generally also unknown to the average person. We hope that by providing relevant, easy-to-understand, information about power usage, we can influence users into conserving more energy.

[edit] Mission Statement

We are committed to educating consumers and households about energy conservation awareness and providing them the tools to initiate positive change.

[edit] Member Roles

Sumeet Patel – Write and present out demo script and consent forms, write test measures.
Fenelia Kosasih – Write and present interview instructions and task list to interviewees, write procedure.
Joshua Funamura – Time the process and take notes during tests; write introduction, participant, and environment sections.
Shwan Kazzaz – Prototype construction and presentation during testing, write discussion.

[edit] 2. Prototype

The prototype we are using is a software design. This interface is designed to accomplish simple tasks associated with checking, monitoring, and analyzing one’s power usage. The main screen and program window background is the Navigation Bar (Fig. A-6.1), from that menu you can navigate to one of 4 functions (Bar Graphs, Consumption, Power Bill, and Calendar). The Bar Graphs (Fig. A-6.2) link displays a bar graph with power usage on the vertical axis, each item is displayed and its power usage is portrayed by the height of the line. One can return to the Navigation Bar by either closing, minimizing, or moving the Bar Graphs sub-window. The second link in the Navigation Bar leads to the Consumption window (Fig. A-6.3). Here users can view consumption for any single device, which can be selected from the drop-down menu, as well as total consumption for all devices. Total power use for a given device or all devices can be ascertained by selecting a time frame from another drop-down menu. The third link in the Navigation Bar leads to the Power Bill window (Fig. A-6.4). This window displays the predicted power bill for the month, based on entered parameters during setup. The Calendar window (Fig. A-6.5) is the fourth link on the Navigation Bar and it allows users to view automatically saved usage graphs and data for any day or time period in the past (since product installation). The menu bar at the top of the main program window allows users to navigate directly to certain aspects of the program. Transmitter information, the Navigation Bar, system preferences, power price rates, and more, can be directly navigated to from there. The transmitter list (Fig. A-6.7) is a list of each transmitter on the network with its Transmitter ID # and its alias (name). The names can be edited and changed directly from this window, the Transmitter ID # is a characteristic of each individual transmitter, similar to a serial number, and cannot be changed. Once a new transmitter has been installed, an alert pop-up appears (Fig. A-6.6), which prompts the user to visit the transmitter list to configure the transmitter. The user environment is designed such that multiple windows can be opened from the Navigation Bar. These windows can be moved around, minimized, and manipulated in size, such that they can be placed side by side, for example, and used for comparison and the best user experience in evaluating one’s power use patterns.

[edit] 3. Method

[edit] Participants

Three participants were selected based on their availability, willingness to be tested and observed, and experience with computers. They were all acquaintances of the group members, but not familiar in any way with the project or the class. It was attempted to obtain results from users with a variety of self-identified computer competence and confidence levels. In general, all users were young but in our target demographic of billpayers. More detailed demographic information has been recorded in the appendix, section A-5.

ID #1 – 28 years old, Male, Computer Programmer

ID #2 – 23 years old, Male, Process Engineer

ID #3 – 22 years old, Female, Student - Biology

[edit] Environment

The test was set up in a moderately noisy computer laboratory. All of the interviews were given in a structured, professional manner with interviewers spread out so as not to stress the participant, but still close enough to observe and run the test. On average, the tests took less than 10 minutes each, so they were low-stress for the interviewee. Most of the time spent on formalities: introductions, the consent form, reading the instructions and performing the demo. The tests were given in the evening time, when the interviewers and participants were available and most-likely to use the actual product.

[edit] Tasks

We chose three tasks to have our participants demonstrate. They were based on our contextual inquiry and task analysis work previously done. The tasks were intended to be arranged in order of complexity such that the first task would be the easiest to complete and the last would be most difficult, with a moderate task in between. We wanted to demonstrate the main functionalities of our product: checking total power consumption as well as per-appliance power consumption. Our third task would also be a crucial step because we allow for easy expansion of our power monitoring system. We need to verify all these crucial procedures can be done by the user in the easiest and fastest way possible. Appendix A-4 shows the tasks in the form presented to the participants.

Easy Task: The first easy task is to check the total energy consumption of a household for the entire month. The most direct way to do this is to select Consumption from the Navigation Bar. When the Consumption window appears, the user should view the line that says Total power use for entire home over last [time frame]. The user should select Month from the drop-down menu, and then view the value at the end of the line.

Moderate Task: The moderate task chosen is to check the breakdown of consumption levels for each appliance. In order to do this, the user must select the Bar Graphs link from the Navigation Bar. The Bar Graphs window that then appears shows each appliances power use on a vertical axis. Each appliance can be singled out in this manner, and its power consumption can be evaluated.

Difficult Task: The difficult task chosen is to add a new transmitter. This is done by plugging in a transmitter to a wall outlet, plugging in a device to the transmitter, and pressing the On switch on the transmitter. When the program is opened, a message will appear asking the user to click a link to the Transmitter List this list gives the user the option to double click on the dash marks in the Name column, and edit this text to contain a name for the transmitter.

[edit] Procedure

In order to test out our product in its early stage, interviews are carried out to observe how the user interacts with the product to find out any difficulties that they are facing when using the products. This test requires at least three persons. The first will act as a greeter. The greeter will also read out the instructions to the user and answer any questions he might have prior to the interview. The second person will demonstrate the role of a participant. A demo script will be carried out with the greeter being the interviewer and assigning the task and the second person acting as the participant and demonstrating how to “think aloud” when performing the tasks. The third person will act as the recorder, and will observe and records down the user’s actions and comments. The test should cover at least three tasks that are frequently used in the interface. The three tasks should range in difficulty level, from easy, to medium to difficult. The three tasks can be found in the appendix, section A-4. Instructions should be read off to the interviewee prior to each interview. These points should be noted in the instructions: 1. Emphasize that the test is for the products and not the user. 2. Encourage the user to think aloud. 3. Remind the users that no help will be provided during the interview as per realistic situation. In addition, a demonstration of an example task should be done. The instruction and the demo script can be found in the appendix.

[edit] Test Measures

We want a concrete performance measures in order to make sense of our test results. For this we use the number of clicks, the amount of time, and the user-specified level of difficulty to perform this task. In addition, we wanted the users to rate how comfortable they are with computers.

1. The number of clicks it takes to finish the task: This test measure is very important because we decided successful software minimizes the clicks a user would have to do to finish the task. The lower the number of clicks, the easier the task and consequently, the easier the software is to use.

2. The amount of time it takes to finish the task: This is an easy test to measure because it requires a group member to start and stop a watch. This test measure is important to our prototype because we will be trying to minimize the amount of time a user spends performing each task. The shorter the time to complete the task, the easier the task is to perform.

3. User rating of how easy it was to complete the task: This is to get a quantitative value from the user himself on how easy the test was. Sometimes a test may take numerous clicks but in the users mind it was simple and straightforward. If this is the case than our task has been accomplished successfully. Our goal would be to have the user feel like every task is easy and straightforward.

4. User rating on their computer usage comfort level: This metric is solely for distinguishing how the test results relate to our metric. A user who is highly proficient in using computers would naturally have fast times and less clicks but if they begin to falter than there is a problem with our software. Similarly, a user with little computer experience is expected to have a slow time and the faster they complete the task, the better our prototype is. After the observation, ratings are given to incidents experienced by the user on the following severity scale from our instructions: 1. I don’t agree this is a usability problem 2. Cosmetic problem 3. Minor usability problem 4. Major usability problem: important to fix 5: Usability catastrophe: imperative to fix Our incident log and ratings can be found in the appendix, section A-5.

[edit] 4. Results

The full details of the interview data are shown in Appendix section A-5; the following results are a summary. Table 4.1 shows the numerical results of our testing.

Table 4.1: Summary of test measure results

We aimed to have three participants of different computer experience, and we had a 5, 7, and a 10 (interviewees #3, #2, and #1, respectively) when self-declaring how comfortable or confident they felt with computers. We thus got a wide experience range. Interviewee #1 was an experienced computer scientist whose comfort level with computers was a 10. We expected his interview to reveal the quickest possible time a user could have to navigate through our software. Interviewee #2 is a process engineer at a biotech company. Even though he is not an expert in the use of computer, he is fairly comfortable with various computer interfaces and often has to deal with various simulation programs in his work field. Interviewee #3 is a biology major, generally familiar with computers though not an expert. She might represent an average demographic for younger users who aren’t particularly tech-savvy. When confronted with the first task, generally the interviewees were able to associate the monthly consumption with the graphical consumption button and then read top down to the current month drop down menu. This task was the easiest and took the shortest amount of time. Some confusion, however, arose with the ambiguity of “energy” and “power” given in colloquial or engineering contexts. All three users went back to the consumption screen for the second task and attempted to look for “all devices” in the drop-down menu. Only through trial-and-error were two of the users able to find the right screen. The third participant completed the task by cycling through all of the possible appliances and had to assign a time frame, resulting in many clicks, but arriving at the correct information. Non-descriptiveness of the “bar graphs” label led to this confusion. The task involving the connection of a new transmitter is performed with ease. This is because there is a message prompting for the next action as soon as the new transmitter is plugged in. Hence, little confusion can arise from this, but when given the task, it was hard for the user to explicitly know whether the transmitter should be configured or plugged in first.

[edit] 5. Discussion

The main issues with our prototype were revealed through our interviews, while there were some issues that we were able to notice and the interviews did not reveal, since the tasks did not use those features in the prototype. The first issue noticed, and the most obvious one as well, was that the graphical representations of each of the four options on the Navigation Bar were not entirely clear as to where the link lead. Even with the textual title beneath each graphic, the users still revealed that this was confusing and not intuitive. Of course, this is mainly just a first-time user pit-fall, as users who have used the software many times will have memorized the functions associated with each graphic. Regardless, it is important to be clear to any user, and current solution we expect to include in the next design cycle will be for a small description of the associated function, two sentences long at most, to display at the bottom of the window when the mouse rolls over each graphic. The next problem that was revealed is that the consumption screen requires better instructions and descriptions of what the user is actually looking at. Longer descriptions explaining the difference between total power usage for the home, total power usage for the device, and current usage for the device are necessary. It may be useful to divide the window more clearly, using strong dividing lines (graphics) and large titles explaining each section. Graphs of consumption over time are also useful at this screen, as this is a useful function our device can offer. This conclusion was determined by our group members, rather than the actual users in the interview. The Calendar screen was also decidedly confusing. Since it was not involved in any of the three tasks, it was also not revealed to be confusing through the interviews. However, it was something that upon further inspection as a group, we decided was unnecessarily confusing. The idea is to allow the user the ability to view their consumption graph for any one day in the past, and to compare it to another day. The user can also view their consumption over a period of time, using a selection of days across the calendar. This will be better accomplished by allowing the user to select the days from a drop-down menu, and allowing them to open a separate window where days can be selected from a calendar should the drop-down menu be insufficient. Currently, seeing the calendar first is confusing and overwhelming, freeing the window of clutter will also allow more room for instructions. We have also decided that each window should have a short description of its functions at the top, so that users can understand what the window does. Lastly, the Power Bill screen needed a link to a popup window where the current power price rate could be modified and saved. Overall our design was not entirely intuitive and easy to use for a first time user, in our next design cycle we aim to improve that, having taken the information from our interviews into account. The general conceptual flow of the software will remain the same, as it seems to be easy to work through for the experienced user.

[edit] 6. Appendix

[edit] A-1: Statement of Informed Consent

A group of students in the IEOR 170 class at UC Berkeley are conducting studies to assess the effectiveness of the Power Ranger 2000.

If you volunteer to participate in this study, you will be asked to use the Power Ranger 2000, and to answer some questions. The questions will be asked orally before, during and after the design activities. There will also be written questionnaires before and after each activity. Your interactions with the system will be recorded on video, audio or with still photographs.

There are no benefits to you for participating, other than what may be an educational experience in using some new technologies for the Power Ranger 2000. We hope that the research will benefit society by improving the Power Ranger 2000. This research poses no risks to you other than those normally encountered in daily life. All of the information from your session will be kept confidential and be referred to by an ID number. The correspondence between your name and ID number will be kept confidential and treated with the same care as our own confidential information. We will not name you if and when we discuss your behavior in research publications. After the research is completed, we may save the notes for future use by ourselves or others. However, this same confidentiality guarantees given here will apply to future storage and use of the materials.

Your participation in this research is voluntary, and you are free to refuse to participate or quit the experiment at any time. Whether or not you chose to participate will have no bearing in relation to your standing in any department of UC Berkeley. If you have questions about the research, you may contact Sumeet Patel at ***-***-****, or by electronic mail *****@berkeley.edu. You may keep a copy of this form for reference.

If you accept these terms, please write your initials and the date here: ________________

[edit] A-2: Instructions

First of all, thank you for your time and willingness to test this product to facilitate our design process. The objective of this observation is test out the product in its early stage of design. The observation is intended to test the product, not you. We are looking for areas where the product might be difficult to use. If you have trouble with any of the tasks, it is the product’s fault and not yours, and that is exactly what we are looking for – to locate trouble spots and improve the products.

Remember that this survey is totally voluntary. In any case you become uncomfortable, please feel free to quit anytime.

We will be able to get a great deal of information from the observation if the people “think aloud” as they perform their tasks. “Thinking aloud” involves speaking your thoughts as you go through each steps of your work. It may feel awkward initially, but it is easy once you get used to it. Don’t worry. If you forget to think aloud in any steps, we will remind you to keep talking as you work.

Before we move on to describing the exercise that you are supposed to do, we will give you a brief introduction of the system. The purpose of this system is to monitor and analyze the power consumption in the household. It is able to monitor the power consumption down to the individual appliances. The way it works is that each appliance will be plugged in into a transmitter before being plugged in into an electrical socket. Each transmitter will transmit the electrical information to a receiver which is connected to the computer. After you install the software on your computer, you would be able to monitor your power consumption from your computer.

Sumeet will now demonstrate an example task and how it is carried out. Here, you will also get an example of “thinking aloud”.

>>>> Demo script performed (see Demo script page, section A-3) <<<<

Now that you have seen a demonstration of one example task being carried out, there are three different tasks that we would like you to try out. We would like you to note that as you are working through the exercises, we will not be providing any help or answer questions. This is because we would like to create a realistic situation where you usually will not be able to get immediate help on using the interface. However, please still ask your questions anyway. We will note down your questions and answer them afterwards.

You can find the lists of tasks and instructions on the handout. Let us know of any questions you might have before you begin the exercise.

[edit] A-3: Demo Script

Interviewer: Hi, thank you for helping us out with completing this survey. This survey will be interactive and we want to know what you are thinking while you are performing the tasks given.

Interviewer: The product you are working with sends a wireless signal from each outlet of your house to your computer. Software on your computer reads the signal and interprets how much power you are consuming.

Interviewer: I will demonstrate to you how a sample question in this survey could be answered. This can also be used to familiarize your self with the product.

Interviewer: Do you have any questions before I begin a sample run through?

Interviewee: No.

Interviewer: Okay well than let us begin. Let us say my task is to view my power consumption in a graphical manner. If I were being interviewed, here is how I would answer the question.

Interviewer: I am now going onto my desktop. I am double clicking on the Power Ranger icon. I am waiting for the software to load.

Interviewer: The software has now loaded and I see a menu with 4 buttons. I find the bar graphs button. I click on the bar graphs button. The bar graph showing my power consumption in last few months appears.

Interviewer: As you can see we would like to see what exactly you are thinking and doing while you perform the task. Follow-up questions could be asked.

Interviewer: For example when finding the bar graphs button. I may ask you, “How did you recognize it was the correct button?”

Interviewer: Now that you have seen a sample question, are you ready to proceed with the survey?

Interviewee: Yes.

[edit] A-4: Task Description Handout

Please complete the tasks in the specified order while thinking aloud and asking relevant questions.

As a reminder, we cannot answer these questions or offer assistance during this process, but we are happy to do so afterward.

Task 1: Imagine that the product and the software have been installed, and all the setup steps have been completed. Let’s say you would like to check the total energy consumption of your household. Demonstrate how you would do this.

Task 2: Now, you want to find what the consumption level is of each individual appliance in your house. Demonstrate how you would do this.

Task 3: Imagine this situation: You just bought a new appliance, and would like to monitor the consumption of this new appliance. You have a spare transmitter and decided to use it for this particular appliance. Demonstrate how you would monitor the consumption of this new appliance.

[edit] A-5: Interview Data

[edit] Interview ID#1

Age: 28 Sex: Male Occupation: Computer Programmer Level of comfort using computers (1 = beginner – 10 = expert): 10

Task 1: Checking total energy consumption The number of clicks taken to complete the task: 3 The amount of time it took to complete the task: 10 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 10

When confronted with the first task, the interviewee was able to associate the monthly consumption with the graphical consumption button and then read top down to the current month drop down menu. This task was the easiest and took only 10 seconds and was rated as easy.

Task 2: Individual energy consumption The number of clicks taken to complete the task: 5 The amount of time it took to complete the task: 22 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy):6

The second task was to find how much each appliance was using. The user went back to the consumption screen and then went through the device drop down menu and selected a device. After realizing that this was the wrong screen, he then went back to the main menu and clicked on bar graphs, which was then the right screen. This task was the hardest task as the user was confused on where to find a list of all the appliances and how much power they were using. The time is much longer and clicks are more because of the mistake initially made.

Task 3: New transmitter setup How easy was it to complete the task( 1 = easy – 10 = difficult): The number of clicks taken to complete the task: 3 The amount of time it took to complete the task: 25 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 10

He was able to see the alert box that appeared on the screen when the transmitter was recognized. He then was able to see transmitter numbers and corresponding names. He then was able to fill in the name for the transmitter and complete the task.

[edit] Interview ID#2

Age: 23 Sex: Male Occupation: Process Engineer Level of comfort using computers (1 = beginner – 10 = expert): 7

Task 1: Checking total energy consumption The number of clicks taken to complete the task: 3 The amount of time it took to complete the task: 30 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 7

The word “consumption” in the task prompted the user to click on “consumption” icon on the navigation bar. Once in the “consumption” page, the user mentioned that he would click on the “select timeframe” drop-down menu and choose March 2007 to view this month’s consumption. And then he would read off the values under “current total home power use”. However, he reread his task and realized that the task asks for “energy” consumption while the screen displays “power consumption”. Being an engineer as he is, he over-thought the problem, and stated that power and energy are different and tried converting from power to energy.

Task 2: Individual energy consumption The number of clicks taken to complete the task: 7 The amount of time it took to complete the task: 40 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 5

Again, upon seeing the word consumption, the interviewee immediately clicks on the “consumption” icon. He clicked on the “select device” drop–down menu on the “consumption page to select the devices that he would like to view. Later on, he realizes that this method only allows him to view the appliances’ consumption one by one and will not allow him to compare the consumption of the appliances all at once. He then went back to the navigation bar and tried the icon “bar graphs” instead. This brought him to the page with the bar graphs, comparing the consumption of different devices.

Task 3: New transmitter setup The number of clicks taken to complete the task: 2 The amount of time it took to complete the task: 30 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 10

The interviewee mentioned that he would plug in the transmitter to the socket and the appliance to the transmitter. As soon as he does that, a message will be displayed indicating that a new transmitter has been found. In the message, there is a prompt “click here.” The interviewee follows this prompt, and clicking that will bring him to the “transmitter list page.” He quickly recognizes the last entry, where there is an unnamed transmitter. He clicks on the unnamed transmitter on the table to rename it.

[edit] Interview ID#3

Age: 22 Sex: Female Occupation: Student – Biology Level of comfort using computers (1 = beginner – 10 = expert): 5

Task 1: Checking total energy consumption The number of clicks taken to complete the task: 3 The amount of time it took to complete the task: 8 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 7

The first task she went right away for the “clock-thing” because it said consumption. It was fairly straightforward as she chose the time interval and completed the task in a short period of time.

Task 2: Individual energy consumption The number of clicks taken to complete the task: 32 The amount of time it took to complete the task: 2 minutes How easy was it to complete the task ( 1 = difficult – 10 = easy): 1

For the second task, she also used the consumption window since it seemed logical that when looking for consumption information that she would find it there. This led to many clicks to see the power consumption of each individual device over a specific time frame. When it was pointed out that the bar graph was the intended window, she remarked that the “bar graphs” name should be changed and/or there should be a button on the consumption page that would lead to “all devices”. Because of the lengthy process of checking each device, this task was very slow and laborious.

Task 3: New transmitter setup The number of clicks taken to complete the task: 7 The amount of time it took to complete the task: 50 seconds How easy was it to complete the task ( 1 = difficult – 10 = easy): 8

The third task of setting up a new transmitter was performed easier than, though requiring a little extra time to figure out what to do with the transmitter without any additional information. The pop-up notification and on-window link made it easy to be driven to the correct setup window. It was also noted that she thought it was unclear whether the transmitter should be plugged in first or set up first.