Carnegie Mellon’s METALS program is a MS that engages industry partners in 9 month engagements for specific education technology projects with real contracts and non-disclosure agreements. I was project manager over a six month collaboration between Carnegie Mellon University and our client, a rapidly growing online institution.
Our client used extensive online systems to align learning goals and instruction in their online platform, and our goal was to design a technology solution that enabled the school to reduce the amount of curriculum provided “off the record” by instructors. These materials are well-intended but they can deviate from the official curriculum at the data-driven university.
I led a team that included a designer, content developers, programmer, and data scientist; managing internal and university communication while acting as point person for my team. We held interviews so we could listen to the experience of instructors, designers, and engineers. We issued surveys, and observed students engaging with each other in online forums. Together our team prioritized stakeholder needs and developed a project schedule, design strategy, and prototype testing plan. The deliverables included a project report and a prototype that our client’s engineers could integrate into proprietary learning system(s).
Early prototypes were tested using proto.io, and the final was created in Sketch
and tested with UsabilityHub.com
Our team began by researching stakeholder relationships and internal communication methods, devised methods of measurement, and designed a scalable software solution designed to improve student performance. We gathered requirements using design exercises and worked with primary stakeholders to analyze features for our prototype. Our interdisciplinary team employed interviews, cognitive task analysis, human-centered design methods, and user testing to help an innovative organization address challenges resulting from their scalable model of instruction.
Our client university had an interesting business model: students pay a fixed fee for a six month term, completing as many courses as possible because a competency-based curriculum allows them to test out of courses at their own pace. However, the final exam is the only grade for the course, putting pressure on students and support staff. In trying to support students course instructors engage with a familiar tool- they create quizzes. The university would like to capture the learning science behind these quizzes and potentially include them in official class curriculum.
I ran weekly meetings with clients and faculty, managed requirements and team tasks, served as the client contact, and conducted stakeholder interviews. I also helped design and test multiple prototypes.
Through surveys, interviews, and collaborations, we engaged with about 80 faculty and staff from the university in one way or another. Our report provides the business case, stakeholder feedback, and user research to support the design, as well as a detailed framework for engineers to build the system. Despite this moderate number of interactions, it feels like our project only scratched the surface of our challenge. Our research uncovered a number of opportunities for the university to address issues and behaviors related to classes and instruction.
After 3 months of user research we were able to pinpoint the challenges that encouraged instructors to create additional instructional materials for their courses. Most instructors were accustomed to traditional classroom responsibilities and they felt the university’s instructional model limited their ability to support student progress.
We spent the remainder of the capstone creating low- and high-fidelity prototypes and presented them to stakeholders: instructors, course designers, assessment designers, and data scientists. We narrowed down seven separate ideas (some of them complementary) and focused on a quiz creation framework, with research-based support for the creation of questions and learner feedback.
Ultimately, we designed a quiz creator and question bank that would allow instructors to publish quizzes that were visible to colleagues and university learning designers. We used Proto.io to build our prototype and automated preliminary testing with UsabilityHub and Mechanical Turk. We presented our rationale, prototype and next steps to high-ranking leaders from the client university.
Our plan defined the technical steps and back-end infrastructure to support features that are linked to the client’s organizational goals.