How might a teacher dashboard help realize the full potential of an adaptive tutor?
Luna was an ongoing research project at CMU’s Human Computer Interaction Institute on how a teacher dashboard could assist middle school teachers’ use of an adaptive tutor. I designed wireframes and conducted interviews, working directly with graduate students Françeska Xhakaj and Kenneth Holstein and presenting my progress each week to Vincent Aleven and Bruce McLaren.
Base level Dashboard
This was an early prototype created before I joined the project that I was tasked with improving. The main issues I had with the prototype was repetitive information and a lack of strong hierarchy in the dashboard. The split between a "Class Level" dashboard and "Individual Level" dashboard was the best part of this early interface, as it is a logical first layer of information hierarchy.
An adaptive tutor is unique in that it learns and adapts to student performance, moving students at their own pace. This presents a complicated challenge to teachers for planning lectures and activities, as they must teach students who may be at vastly different places in the curriculum.
Dashboard Concepts and Uncertainties
"Skill or Mastery" is defined as the probability that a student will correctly answer a question on a topic if tested on it, however, a threshold for the percentage of mastery implies mastery to be an predetermined binary measure.
"Levels and Opportunities" are essentially groups of problem sets. Different skills are tested within levels, often times being repeated in multiple levels. Opportunities are the number of times a skill will be tested for in a level. Each student's number of opportunities varies from level to level, making tracking difficult.
"Errors or Misconceptions" are specific types of errors that the tutor detects are indicative of a misunderstanding, however, only about a third of these errors are categorized. While teachers loved to see these misconceptions, the metric itself is much weaker than the analysis being performed on skills and mastery.
Student performance varies from class to class and school to school, making any form of notification or threshold for individual or class progress difficult.
Time investment by the teacher is the final, arguably most difficult, uncertainty to plan a dashboard around given the range of teaching styles. The separation of class and individual metrics helped address this issue by separating use cases and hiding complexity.
Class Level Dashboard
Just from using this dashboard, a teacher can ideally adjust their lecture plan based on the skills students are working on, address class-wide misconceptions which students still have, and identify students who may be "stuck". The dashboard also helps teachers understand the format and language of the adaptive tutor through displaying the tutor's structure in the form of the math questions themselves.
The error chart shows patterns of occurrence, allowing the teacher to make the judgement of whether a certain error is relevant enough for the whole class or if an error is only classified as such due to low number of opportunities.
Individual Level Dashboard
My focus was primarily on the bottom graphic titled, "How fast are students progressing?". Showing direct, literal progress of students in the tutor helps teachers more accurately understand how the tutor, mastery, and errors work.
Various filtering methods show: students who are "stuck", students learning the same concepts, students abusing the hint button, and students in alphabetical order. These filters address either a need teachers requested or positively augments a behavior already practiced by teachers.
I planned the interview protocol with Kenneth Holstein, but designed almost all of the interview materials myself. Our objective was to understand what needs a dashboard must fulfill, our overarching goal to augment and enhance a teacher's ability to circulate effectively. I interviewed three different middle school teachers.
"If you could have any super power to help you teach, what would it be and how would you use it?"
In beginning the interviews with a question about super powers, I intended to open the minds of the teachers, many who might already have fixed ideas for what a dashboard is and does. Moving the teachers expectations away from me providing all the ideas with a simple open-ended question encouraged participation and creativity.
"With drawing and annotating the computer lab seating chart, how do you usually spend your time?"
Given contextual inquiry restrictions due to the time of year, I resorted to directed storytelling. Some teachers didn't take their students to computer labs, so I improvised, asking them to describe how they supervised students who rotated between different academic stations.
"What are your gut reactions to the idea behind these hypothetical 'teacher dashboards'?"
Encouraging teachers to respond to the idea instead of the exact details of the scenario was an essential part of the process. The key was to have scenarios that I knew teachers would not like, and to use teacher's clarifying questions about the scenarios as an opportunity to ask what they would ideally want.
One of the biggest challenges I had was knowing enough about how the data was being generated in order to make accurate visual representations for each metric. Quickly seeing which students are "stuck" is a necessity. Being able to change the threshold for how many students are classified as "stuck" is also essential due to class performance variability.
Vincent Aleven designed these two graphics, which I modified and applied to the Next-Day dashboard.
I used this graphic and its zooming feature, as inspiration for the class-level page metrics in the Next-Day Dashboard.
I read literature published on teaching dashboards and research methods to provide context as to why dashboards are an up and coming tool as well as specific implementation for educational environments. Information Dashboard Design by Stephen Few was especially helpful, I adapted many of his principles and a few of his graphics into the Next-Day Dashboard.
"Cross-disciplinary Participatory & Contextual Design Research: Creating a Teacher Dashboard Application"
- Troy D. Abel andMichael A. Evans
"Learning dashboards: an overview and future research opportunities"
- Katrien Verbert, Sten Govaerts, Erik Duval, Jose Luis Santos, Frans Van Assche, Gonzalo Parra, and Joris Klerkx
"An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment"
- Roberto Martinez Maldonado, Judy Kay, Kalina Yacef, and Beat Schwendimann
Information Dashboard Design
- Stephen Few
Storytelling with Data: A Data Visualization Guide for Business Professionals
- Cole Nussbaumer Knaflic
Universal Methods of Design
- Bruce Hanington