Adaptive tutors, which move students at their own pace, are often difficult to understand and effectively utilize. My job was to work on the development of two different dashboards, Luna and Real-Time, which would help teachers make better use of the information from adaptive tutors.
Middle school teachers who used adaptive tutors to supplement their teaching a couple times a week were our primary users. The main challenge with communicating adaptive tutor information was making the data actionable while respecting teachers' autonomy and gaining their trust.
I joined an ongoing research project at CMU’s Human Computer Interaction Institute led by graduate HCI students Françeska Xhakaj and Kenneth Holstein, while under the direction of Vincent Aleven and Bruce McLaren.
Final Concept: Luna
This interface was meant to be used by teachers the night after a computer session in order to adjust their lesson plan for the next day.
Class Level: Skills
From this dashboard, a teacher can make quick adjustments to their lecture plan while having an idea of which students may be struggling. The “skills” data has been more visually weighted compared to the “errors” data in order to reflect the varying strength behind their statistical models.
Class Level: Errors
Errors are less actionable for planning lectures and thus are displayed on a problem by problem level. I made an effort to help teachers understand the tutor by including pop-ups with actual student problems (2x+3=10) next to the language used by the tutor (combine unlike terms to make variable).
Individual Level Dashboard
I primarily redesigned the bottom graphic titled, "How fast are students progressing?". Showing direct progress of students over a calendar helps teachers understand the tutor better and what the student experience is like. The filters included for the bottom graphic support either a requested feature by teachers or a current teacher practice.
Final Concept: Real-Time
These interviews were for a different type of dashboard, one which a teacher would use in real time while students were working in the computer lab. This dashboard was in a much earlier stage than Luna.
I designed almost all of the interview materials myself, as well as conducted three interviews. My objective was to understand what needs a dashboard must fulfill in order for a teacher to circulate effectively throughout the class.
"If you could have any super power to help you teach, what would it be and how would you use it?"
In beginning the interviews with a question about super powers, I intended to frame the interview as being exploratory and build rapport. Moving the teachers expectations towards participatory design with a simple, open-ended question encouraged participation and creativity.
"While drawing on the computer lab seating chart, how did you usually spend your time?"
Given contextual inquiry restrictions due to the time of year (summer), I resorted to directed storytelling. Some teachers had never taken their students to computer labs, so I improvised, asking them to describe how they supervised students working in activity stations.
"What are your gut reactions to the idea behind these hypothetical 'teacher dashboards'?"
Encouraging teachers to respond to the idea, versus focusing on details, was an essential part of the process. Another key was to have intentionally vague scenarios and use teacher's clarifying questions as an opportunity to ask what they would ideally want. Finally, creating scenarios teachers would likely disapprove of was a necessity.
My first step after joining the team was listening to teacher interviews which already occurred, trying to understand how the dashboards related to each other, and mapping how each dashboard might be used in a school week.
Notification Threshold Feature
Given the variability in student performance between classes, schools, and states, I believe it is essential to be able to change the tutor’s threshold for notifications, whether for positive or negative student performance.
A notification threshold would also help the dashboard be more accepted by the teacher, if it more closely aligned with their sense of who was doing well vs who was not.
Sketches of different graphics and how they were intended to be used was important to my process. My interfaces were designed with the intent of being directly connected to a specific teacher action.
Levels are actually not a necessary part of the tutor or dashboard and exist purely so students and teachers can better understand them. Designing truthful, but user-friendly graphics was a challenge I faced.
In this prototype, “Errors” appear to be more accurate than “Skills” due their higher level of detail. However, the statistical model behind “Skills” is actually much stronger; the visuals of the dashboard needed to actually reflect this difference.
Based on teacher interviews and contextual inquiries conducted by my colleagues, I sought to test out potential ways to help minimize “bad teacher behaviors”. Addressing these without being overly paternalistic was difficult.
Speed-dating concepts for general teacher preferences proved to be helpful for the Luna dashboard as well. Some of the infographics used, such as the one below, were far too unintuitive.
I think my greatest takeaways from this project were how to manage working on multiple projects at the same time and how to integrate into a team which has already been working on a project.
Vincent Aleven designed these two graphics.
I read literature published on teaching, dashboards, and research methods to better understand how to design dashboards in educational environments. Information Dashboard Design by Stephen Few was especially helpful, I adapted many of his principles and a few of his graphics into the Next-Day Dashboard.