Evaluation Research
Evaluation research can be defined as a type of study that uses standard social research methods specifically for evaluative purposes, perhaps to assess the results of an intervention. Did the intervention meet its goal? Were there any unanticipated consequences? Some research methods are designed to be used as evaluation tools and employ dedicated techniques to this end. These include input measurement; performance measurement; impact assessment; service quality assessment; process evaluation; benchmarking; standards; quantitative methods; qualitative methods and methods drawn from Human-Computer Interaction (Powell, 2006).
Evaluation: GO-GN Insights
Johanna Funk performed a developmental evaluation of 4 research projects that created learning resources. These learning outsources were all in one way ‘open’ and online.
“I think it was so highly reflexive that it could be interpreted as circular; so a disadvantage was the cycles and circles of evaluation; I was answering the research questions each time with the criteria set filters; this resulted in me writing a LOT about what the resources did according to the three set of criteria; in three cycles of evaluation and interrogation. Pedantic
is the word I would use. It did have a feel of luxury to it, though; being able to really concentrate on the processes in the resources down to a granular level, to see it from a number of perspectives and try to get right down to the mechanisms that helped make the resources different and more collaborative. This ‘search for the things’ was a bit circular and I had to find the things that we also not collaborative; that’s the thing about looking for best practice; you also have to compare it to what’s ‘not good’ in the resource but also know that there are relativity issue with what ‘good’ means, and to whom. So having a bird’s eye view on who the stakeholders are is helpful; as ‘knowledge management tools,’ learning resources have
agenda-pushing potential we might not recognize.”
Francisco Iniesto devised an accessibility audit and then used it to evaluate the current accessibility of MOOCs from 4 major platforms: FutureLearn, edX, Coursera and Canvas. This evaluation comprised 4 components: technical accessibility, user experience (UX), quality and learning design; 10 experts were involved in its design and validation.
“The combination of qualitative studies through interviews with MOOC providers and learners and the quantitative information provided by the MOOC survey data has provided an in-depth and multi-faceted insight into accessibility needs of MOOC learners. The MOOC accessibility audit has helped to identify accessibility barriers and the audit provides a tool that can be used and iteratively developed further to support the design and evaluation of MOOCs for accessibility. Interviews have involved MOOC providers and MOOC researchers. The aim was to explore the perspectives of platform and course developers on the importance of accessibility of the MOOC environment. The data from this study was useful to understand how to approach the next steps in this research. Interviewing individuals involved in MOOC development helped to understand how they cater for disabled learners, and the approaches they use to design accessible MOOCs. Additional evaluation involved disabled learners who had participated in learning via MOOCs. Learners were a useful source of data to explore the accessibility barriers and their solutions in using the technology and the learning designs they come up against when interacting with MOOCs. The data from the interviews helped to understand their motivations, the current accessibility barriers they have found, how they reacted to them, and their suggestions for desired solutions. Qualitative methods can help to explore a new area of research, the use of surveys in my cases helped to identify students to be interviewed to develop an understanding of their perspective on MOOCs.”
Useful references for Evaluation Research: Chang & Little (2018); Patton (2010); Powell (2006); Rutman (1977)
