Class Engagement Metrics

🏢 Minerva Project

⏱️ ~2 months (2021)

🧑🏻‍💻 Designer (me), PM, and
Engineer

Forum - an online learning platform - classroom data for tasks like tracking attendance and student talk time. It is also used to show talk time per student via green red and yellow indicators, accessible to the instructor at any time via a hot key.

However, we wondered how else we might leverage this data for educational purposes.

Class Metrics

🏢 Minerva Project

⏱️ ~2 months (2021)

🧑🏻‍💻 Designer (me), PM, Engineer

The ‘Talk Time’ view for an instructor in the classroom.

The ‘Talk Time’ view for an instructor in the classroom.

The Problem

How might we better use Forum’s classroom data to inform instructors and administrators of student engagement, throughout a program?

Research + Exploration

Research & Exploration

I began by discussing existing reports with multiple partner program managers and examining the classroom data collected by Forum. I explored ways to summarize key engagement metrics such as reactions, talk time, and chat activity to provide insights on student participation.

Sketches exploring different ways we might display the data.

Two Directions

I created two potential dashboard designs, each displaying engagement data in different formats and styles. A survey of 14 instructors helped guide design decisions:

The two mockups presented the same information in different ways due to the different ways I thought instructors might use the data based on previous discussions.

Feedback Summary

  • 92% of responders found some part of the reports useful

  • The majority would prefer seeing this data over time ("sometimes students just have off days")

  • Key engagement metrics included talk time, chat messages, and polls.

  • Many responders voiced concern about how this info could be used or interpreted in unintended ways

  • 92% of responders found some part of the reports useful

  • The majority would prefer seeing this data over time ("sometimes students just have off days")

  • Key engagement metrics included talk time, chat messages, and polls.

  • Many responders voiced concern about how this info could be used or interpreted in unintended ways

  • 92% of responders found some part of the reports useful

  • The majority would prefer seeing this data over time ("sometimes students just have off days")

  • Key engagement metrics included talk time, chat messages, and polls.

  • Many responders voiced concern about how this info could be used or interpreted in unintended ways

Iteration

In response to the survey feedback, I explored program, section, and class level overviews, as well as including technical or internet connection issues.

I collected feedback from 16 instructors and administrators on the test prototype.

Feedback

Overall, instructor feedback was overwhelmingly positive. Helpfulness ratings for each section improved in almost all areas, and the section overview was rated as the most helpful page.

"The engagement chart is 🔥"

"These would be super helpful for personal reflection and faculty reviews. I cannot wait!"

"The engagement chart is 🔥"

"These would be super helpful for personal reflection and faculty reviews. I cannot wait!"

Quotes from the survey.

The Final Designs

The final dashboard provides instructors with:

  • 3 key metrics: student talk time, instructor talk time, and attendance

  • Additional metrics such as polls, chat, breakouts, and reactions

  • Student talk time history for the previous 4 classes

The final design used color to indicate engagement 'health' in an easily glanceable way.

Results & Looking Ahead

The new dashboard was able to replace the unscalable practice of creating custom reports for every partner three times a term, saving over 400 hours annualy.

In addition, instructors are already starting to utilize this data to better inform their decisions during class.

Future improvements could include using this data to better understand student engagement trends and predict changes, though more data would be needed for a high level of confidence.

© Cody Morrow 2024

Connect:

© Cody Morrow 2024

Connect:

© Cody Morrow 2024

The two mockups presented the same information in different ways due to the different ways I thought instructors might use the data based on previous discussions.

Sketches exploring different ways we might display the data.