How might a teacher dashboard help realize the full potential of an adaptive tutor?
Luna was an ongoing research project at CMU’s Human Computer Interaction Institute on how a teacher dashboard could assist middle school teachers’ use of an adaptive tutor. I designed wireframes and conducted interviews, working directly with graduate students Françeska Xhakaj and Kenneth Holstein and presenting my progress each week to Vincent Aleven and Bruce McLaren.
Base level Dashboard
This was an early prototype created before I joined the project that I was tasked with improving. The two screens: "Class Level" and "Individual Level", are a macro and micro approach to showing student data to a teacher.
An adaptive tutor is unique in that it reacts to student performance, moving students at their own pace. This presents a complicated challenge to teachers, as they must interact with students who may be at vastly different places. Schools which use these tutors often use them two times a week as a supplement to teacher's lectures.
Dashboard Concepts and Uncertainties
"Skill or Mastery" is defined as the probability that a student will correctly answer a question on a topic if tested on it, however, a threshold for the percentage of mastery implies mastery to be an predetermined binary measure.
"Levels and Opportunities" are essentially groups of problem sets. Different skills are tested within levels, often times being repeated in multiple levels. Opportunities are the number of times a skill will be tested for in a level. Each student's number of opportunities varies from level to level, making tracking difficult.
"Errors or Misconceptions" are specific types of errors that the tutor detects are indicative of a misunderstanding, however, only about a third of these errors are categorized. While teachers loved to see these misconceptions, the metric itself is much weaker than the analysis being performed on skills and mastery.
Student performance varies from class to class and school to school, making any form of notification or threshold for individual or class progress difficult.
Time investment by the teacher is the final, arguably most difficult, uncertainty to plan a dashboard around given the range of teaching styles. The separation of class and individual metrics helped address this issue by separating use cases and hiding complexity.
Through research which was already done and interviews I conducted, I learned that teachers have a fair amount of skepticism towards the dashboards. Information had to be presented as the slightest of suggestions in order to avoid offending teachers.
Class Level Dashboard
Just from this dashboard, a teacher can adjust their lecture plan based on the skills students are working on, address class-wide misconceptions which students still have, and identify students who may be "stuck". I visually weighted the "Skills" section in order to encourage teachers to rely on it more compared to the "Errors" section.
The error chart shows patterns of occurrence, allowing the teacher to make the judgement of whether a certain error is relevant enough for the whole class or if an error is only classified as such due to low number of opportunities. The dashboard also helps teachers understand the language the adaptive tutor uses through displaying actual student questions ("Combine unlike terms to make variable" means a student combined 2x and 3 into 5x).
Individual Level Dashboard
My focus was primarily on the bottom graphic titled, "How fast are students progressing?". Showing direct progress of students over calendar days in the tutor helps teachers more accurately understand how the tutor, mastery, and errors work.
Various filtering methods show: students who are "stuck", students learning the same concepts, students abusing the hint button, and students in alphabetical order. These filters address either a need teachers requested or positively augments a behavior already practiced by teachers which I learned through research.
I planned the interview protocol with Kenneth Holstein, but designed almost all of the interview materials myself. Our objective was to understand what needs a dashboard must fulfill, our overarching goal to augment and enhance a teacher's ability to circulate effectively. I interviewed three different middle school teachers.
"If you could have any super power to help you teach, what would it be and how would you use it?"
In beginning the interviews with a question about super powers, I intended to frame the interview as being exploratory; many of the teachers interviewed were recently shown high fidelity versions of dashboards. Moving the teachers expectations towards participatory design with a simple, open-ended question encouraged participation and creativity.
"With drawing and annotating the computer lab seating chart, how do you usually spend your time?"
Given contextual inquiry restrictions due to the time of year (summer), I resorted to directed storytelling. Some teachers didn't take their students to computer labs, so I improvised, asking them to describe how they supervised students who rotated between different academic stations.
"What are your gut reactions to the idea behind these hypothetical 'teacher dashboards'?"
Encouraging teachers to respond to the idea, versus focusing on details, was an essential part of the process. Another key was to have intentionally vague scenarios and use teacher's clarifying questions as an opportunity to ask what they would ideally want.
One of the biggest challenges I had was knowing enough about how the data was being generated in order to make accurate visual representations for each metric. Quickly seeing which students are "stuck" is a necessity for teachers. Being able to change the threshold for being "stuck" is also essential due to class performance variability between classes, schools, and states.
Sketches of different graphics and how they were intended to be used was important to my process. My process for constructing the final screens revolved around creating dashboard metrics with a specifically intended teacher action.
Vincent Aleven designed these two graphics, which I modified and applied to the Next-Day dashboard.
I used this graphic and its zooming feature, as inspiration for the class-level page metrics in the Next-Day Dashboard.
I read literature published on teaching dashboards and research methods to provide context as to why dashboards are an up and coming tool as well as specific implementation for educational environments. Information Dashboard Design by Stephen Few was especially helpful, I adapted many of his principles and a few of his graphics into the Next-Day Dashboard.
"Cross-disciplinary Participatory & Contextual Design Research: Creating a Teacher Dashboard Application"
- Troy D. Abel andMichael A. Evans
"Learning dashboards: an overview and future research opportunities"
- Katrien Verbert, Sten Govaerts, Erik Duval, Jose Luis Santos, Frans Van Assche, Gonzalo Parra, and Joris Klerkx
"An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment"
- Roberto Martinez Maldonado, Judy Kay, Kalina Yacef, and Beat Schwendimann
Information Dashboard Design
- Stephen Few
Storytelling with Data: A Data Visualization Guide for Business Professionals
- Cole Nussbaumer Knaflic
Universal Methods of Design
- Bruce Hanington