Luna was an ongoing research project at CMU’s Human Computer Interaction Institute on how a teacher dashboard could assist middle school teachers’ use of an adaptive tutor. I designed wireframes and conducted interviews, working with graduate students Françeska Xhakaj and Kenneth Holstein while under the direction of Vincent Aleven and Bruce McLaren.
Base level Dashboard
This was an early prototype created before I joined the project. The two screens: "Class Level" and "Individual Level", are a macro and micro approach to showing student data to a teacher.
An adaptive tutor is unique in that it reacts to student performance, moving students at their own pace. This presents a complicated challenge to teachers, as they must interact with students who may be at vastly different places. Schools which use these tutors often use them two times a week as a supplement to teacher's lectures.
Through research which was already done and interviews I conducted, I learned that teachers have a fair amount of skepticism towards the dashboards. Information had to be presented as the slightest of suggestions in order to avoid offending teachers.
Class Level Dashboard
Just from this dashboard, a teacher can adjust their lecture plan based on the skills students are working on, address class-wide misconceptions which students still have, and identify students who may be "stuck". I visually weighted the "Skills" section in order to encourage teachers to rely on it more compared to the "Errors" section.
The error chart shows patterns of occurrence, allowing the teacher to make the judgement of whether a certain error is relevant enough for the whole class or if an error is only classified as such due to low number of opportunities. The dashboard also helps teachers understand the language the adaptive tutor uses through displaying actual student questions ("Combine unlike terms to make variable" means a student combined 2x and 3 into 5x).
Individual Level Dashboard
My focus was primarily on the bottom graphic titled, "How fast are students progressing?". Showing direct progress of students over calendar days in the tutor helps teachers more accurately understand how the tutor, mastery, and errors work.
Various filtering methods show: students who are "stuck", students learning the same concepts, students abusing the hint button, and students in alphabetical order. These filters address either a need teachers requested or positively augments a behavior already practiced by teachers which I learned through research.
I planned the interview protocol with Kenneth Holstein, but designed almost all of the interview materials myself. Our objective was to understand what needs a dashboard must fulfill, our overarching goal to augment and enhance a teacher's ability to circulate effectively. I interviewed three different middle school teachers.
"If you could have any super power to help you teach, what would it be and how would you use it?"
In beginning the interviews with a question about super powers, I intended to frame the interview as being exploratory; many of the teachers interviewed were recently shown high fidelity versions of dashboards. Moving the teachers expectations towards participatory design with a simple, open-ended question encouraged participation and creativity.
"While drawing on the computer lab seating chart, how did you usually spend your time?"
Given contextual inquiry restrictions due to the time of year (summer), I resorted to directed storytelling. Some teachers had never taken their students to computer labs, so I improvised, asking them to describe how they supervised students working in activity stations.
"What are your gut reactions to the idea behind these hypothetical 'teacher dashboards'?"
Encouraging teachers to respond to the idea, versus focusing on details, was an essential part of the process. Another key was to have intentionally vague scenarios and use teacher's clarifying questions as an opportunity to ask what they would ideally want.
One of the biggest challenges I had was knowing enough about how the data was being generated in order to make accurate visual representations for each metric. Quickly seeing which students are "stuck" is a necessity for teachers. Being able to change the threshold for being "stuck" is also essential due to class performance variability between classes, schools, and states.
Sketches of different graphics and how they were intended to be used was important to my process. My process for constructing the final screens revolved around creating dashboard metrics with a specifically intended teacher action.
Vincent Aleven designed these two graphics.
I used this graphic as inspiration for the class-level page metrics in the Next-Day Dashboard.
I read literature published on teaching, dashboards, and research methods to better understand how to design dashboards in educational environments. Information Dashboard Design by Stephen Few was especially helpful, I adapted many of his principles and a few of his graphics into the Next-Day Dashboard.