Open educational resources afford flexibility in teaching that are difficult or untenable with commercial resources. These affordances of open education in teaching are known as open educational practices (Cronan & MacLaren, 2018; Tlili et al., 2021). One promising open educational practice is social annotation, also known as collaborative annotation (Brown & Croft, 2020; Kalir, 2020). In social annotation, students have a course resource, such as a textbook, in which they share their highlights and notes about the content. This can be done through a variety of tools such as Perusall, Hypothesis, and Google documents. Social annotation provides opportunity for engagement and connection as students connect with other peers with the content. However, when I first became aware of the tool, there was little evidence indicating that students were actually benefiting from social annotation. This prompted me to conduct a series of studies to investigate the issue and examine evidence to see if social annotation was beneficial for students. In this chapter, I will share about what I found in these studies including “behind the scenes” details of my reasons for conducting these studies and the logistics of carrying them out. In this way, instructors may be informed about the potential effects of social annotation as well as receive practical advice on carrying out scholarship of teaching and learning studies in their courses.

This journey to collect and analyze data on social annotation began with a study on student perceptions of social annotation. I was invited to collaborate on this by Julie Lazzara, a fellow in the Open Education Research Fellowship. This fellowship is a program to mentor and support early-career researchers in open education in which I serve as a primary researcher. Dr. Lazzara surveyed students in two of her online psychology courses (introductory psychology and developmental psychology) at Chandler Gilbert Community College in the American Southwest. In this study, we found that students had generally positive perceptions of social annotation and participated more than was required (Lazzara & Clinton-Lisell, 2024). This was a good sign, but didn’t address how social annotation related to course grades as the survey responses were anonymous. Also, we didn’t compare social annotation to anything else as it was a descriptive study (see Table 1). In the analyses, we compared the average number of comments by students to the required number for the course. As students reliably wrote more than required for most chapters, we inferred perceptions were positive.

In the next study, I examined social annotation in my face-to-face Child Development course at the University of North Dakota (UND) in the great plains region of the United States. I designed a correlational study in which I examined associations among student perceptions of social annotation, analytics from the platform, representational justice (opportunities to speak from unique experiences; Fraser, 2008; Lambert, 2018), and exam grades (Clinton-Lisell, 2023). For student perceptions, I adapted an established motivation measure grounded in self-determination theory, which is a longstanding theory in psychology (Deci et al.,1994). According to self-determination theory, student motivation is optimized when they experience autonomy, relatedness, and competence (Ryan & Deci, 2020). My thinking was that social annotation would support each of these three areas. Students would have choice over what they comment on, which could foster autonomy. They would be sharing their notes with their peers and able to reply and react to their peers’ comments, which could bolster relatedness. Finally, they would gain better understanding of the reading and course content, which could help with competence. I asked students to compare their motivation for social annotation with their motivation for two other reading accountability techniques: quizzes and individual note taking (comments are not shared). Students generally were more motivated for social annotation than quizzes (effect sizes ranging from d = 1.01 to d = 1.49), but findings for individual note taking were mixed. Students expressed more choice with individual note taking (likely because it usually is not a course requirement; large effect size of d = 1.03), but higher levels of relatedness with social annotation (large effect size of d = 1.37).

I examined representational justice based on previous scholars’ commentaries of the opportunities with social annotation for students to share their experiences and offer viewpoints counter to the text (Brown & Croft, 2020). This could potentially empower students and provide opportunities for representational justice (Kalir & Dean, 2018). Conversely, peer interactions could be unsupportive, contain microaggressions, or even be openly hostile (Brown & Croft, 2020). To quantify representational justice, I adapted a measure my colleague and I developed for a different open pedagogy study (Clinton-Lisell & Gwozdz, 2023). I had a difficult time envisioning how representational justice could be fostered through individual note taking or quizzes and subsequently did not have a comparison to social annotation regarding representational justice. Instead, I compared responses to the midpoint as was done in an earlier study to see if students had positive experiences regarding representational justice. Based on the findings, students reported moderately high levels of representational justice (average of 3.87 rating on a scale of 1-5), which was reassuring.

Using correlational and regression analyses, I found that comments in social annotation appeared to increase reading time of the course textbook, which in turn was positively associated with grades. Competence and relatedness (both components of motivation) correlated with grades when they were examined on their own (r = .35 and r = .38, respectively). However, competence and relatedness did not appear to relate to grades (i.e., there were no significant correlations) once the number of comments and active reading were considered. In other words, the roles of these components of motivation in social annotation could likely be explained by more comments leading to longer active reading times. This could be interpreted that students who were motivated spent more time reading and made more comments, thereby boosting learning. This provided correlational evidence supporting social annotation as a means to encourage students to read the content and engage with the textbook to support their learning. However, the question remained of whether social annotation was better than other methods of encouraging reading, such as taking quizzes or individual note taking, for course grades.

In the following study, I compared social annotation to quizzes on assigned readings again in my Child Development course at UND (Clinton-Lisell, in-press). In this crossover, within-subjects designs, students completed quizzes for one exam and social annotation for another exam with random assignment determining which reading assignment for each exam. In this way, half of the students were completing quizzes for the one exam and then switched to social annotation for the next exam. The other half of the students had social annotation for the one exam then switched to quizzes for the next exam. This allowed for comparing exam scores within the same student and the randomization of order allowed for causal claims to be made (this cannot be done with correlational designs). I also examined motivation using the same measure as from the correlational study. There were no differences in exam grades, but students generally reported more motivation for social annotation compared to quizzes (effect sizes ranging from d = .15 to d = .54). This supported the correlational findings on motivation found in the previous study (Clinton-Lisell, 2023). However, it was puzzling that I did not see any connection with exam grades. I should add that students also learned the exam content in class, although I had thought social annotation would help prepare them to learn in class as a pre-class reading exercise. One thought was that the exams were too broad to really capture any learning benefits from social annotation. Another possibility is that students may have been motivated to engage with the content through social annotation more than quizzes, but that motivation did not transfer to exam performance.

One question about the effectiveness of social annotation is if positive results are simply due to the act of writing annotations or the sharing of annotations with peers if necessary for benefits. To examine this, my colleague and I randomly assigned students in online courses to engage in social or individual annotations of course readings. The design is between subjects and differs as students only experienced one condition (unlike in the earlier, within-subjects study in which they experienced both quizzes and social annotations).

To dig into why social annotation may have additional benefits due to sharing notes, we examined sense of belonging. We expected that sense of belonging, which is how much students feel that they are supported and connected with others in a learning environment (Goodenow, 1993), would be greater for students who did social annotation because of the opportunities to connect with peers. We found that students in an introduction to psychology course who socially annotated reported higher levels of sense of belonging compared to their peers who individually annotated (large effect size of d = 1.83). We did not find a difference in sense of belonging between annotation types for a graduate-level educational psychology course (p = .80, d = -.11). This is likely because the sense of belonging scores for students in the educational psychology course were rather high, meaning there is little room for a notable increase. Students in that graduate-level course tended to be in cohorts and knew their peers prior to the course. In contrast, students in the introductory psychology course likely did not know their peers given the large undergraduate enrollment at the institution. No differences in grades based on annotation type were noted, but most of the grades were in the “A” range making it difficult to observe any changes due to annotations. The high grades in the courses are also likely why a correlation between sense of belonging and final course grades was not found, despite being noted in another study (Yust et al., 2021).

Conducting studies on my own classes involves special ethical considerations as I was both instructor and researcher. Students may have felt pressured as I was their instructor, and their names needed to be included with their responses in order to connect them to grades. To mitigate these risks, I explicitly informed them that choosing to participate or not would not affect their grades or their student-instructor relationship with me. I offered bonus points as an incentive for participation, but students had an option to earn these points through non-research means, usually writing a reflection about a reading I shared with them. It’s important to note that a non-research option for grade incentives is necessary to follow guidelines for research ethics. Anecdotally, students usually preferred to do the study activities than the writing reflection despite similar time investments. Using bonus points as an incentive could potentially lead to biased participation for students as those who are already high performing or content with their grades would logically be less motivated to participate.

However, in my experience, around 80-90% of my students opt to participate in studies, which indicates results are fairly representative of the students in my courses. Not all study activities were optional, though. For my social annotations studies described in this chapter, students were required to complete annotation assignments or quizzes (depending on the study and condition). These were graded activities–not bonus point activities. However, students were not required to have their data from these assignments used in research. Students were provided with study information at the beginning of the term and told if they did not want their data from these assignments to be used, they needed to opt-out by informing me of this. Students were also told that opting out would not affect their grade in the course or relationship with me, but that they still needed to complete the required assignments.

Another ethical consideration is when to look at student data for the study. I would wait until final grades were entered for the term before looking at their responses to the surveys (students were informed of this in both the invitation to participate and the informed consent document). To award bonus points without viewing the data, I asked a colleague, who was on the IRB (Institutional Review Board) protocol for each study, to send me a list of names of students who completed the survey. Once data collection was complete, I deleted the student names, so the dataset was deidentified. If I have a class survey and don’t need to connect responses to grades, then I set up a separate link to another survey at the end of the survey to collect names. This way student names are not linked with their responses and there is less risk of confidentiality breaches.

Anecdotally, students have had positive or neutral reactions to me, as their instructor, conducting studies with them. When I request participation, I explain that the reason I am asking them to complete the questionnaire is because I want to better understand student needs and how to best address them. I also explain that sharing the findings allows for other instructors to learn what is effective for teaching. I have conducted dozens of scholarship of teaching and learning studies in my courses over the past ten years and have never received a student complaint. Students who expressed opinions have generally stated appreciation that I care enough to really examine and test what works for them. It may help that I cover research methods in some form in all of my classes given my field (educational psychology), so they have some familiarity with what research is and how it is necessary for knowledge acquisition. Based on their responses on various questionnaires, students also appeared to be unafraid of expressing honest opinions about what they disliked when asked. I’ve noticed this is the case even when student names are connected with their responses, which is reassuring as a researcher that the data are likely valid and as an instructor that students felt safe being honest with me.

Throughout these studies, I have based terminology choices on what is used in my field. Because my background is in educational psychology, I examined motivation through that lens. The components of motivation, such as interest, choice, pressure, autonomy and relatedness, were those previously defined in self-determination studies and measures (e.g., Deci et al., 1994; Howard et al., 2021). This was also the case with sense of belonging (Goodenow, 1993). In terms of learning outcomes, the measures have been exams or final course grades. Throughout these studies, only one had an effect related to grades (Clinton-Lisell, 2023). Generally speaking, grades are coarse, broad measures with numerous complicating factors making it difficult to see a reliable effect from a single intervention or assignment. For example, the exams contain content from the readings, but material was also covered in class activities and videos. In retrospect, to really examine social annotation’s potential influence on learning, I need to design future studies to specifically examine learning from the assigned readings and how that learning varies depending on social annotation.

References

Brown, M., & Croft, B. (2020). Social Annotation and an Inclusive Praxis for Open Pedagogy in the College Classroom. Journal of Interactive Media in Education, 2020(1). https://doi.org/10.5334/jime.561

Clinton-Lisell, V. (2023). Social annotation: What are students’ perceptions and how does social annotation relate to grades? Research in Learning Technology, 31. https://doi.org/10.25304/rlt.v31.3050

Clinton-Lisell, V., & and Gwozdz, L. (2023). Understanding student experiences of renewable and traditional assignments. College Teaching, 71(2), 125–134. https://doi.org/10.1080/87567555.2023.2179591

Clinton-Lisell, V. (2024). Comparing Quizzes and Social Annotation for Pre-Class Reading Accountability. Teaching of Psychology, 0(0). https://doi.org/10.1177/00986283241275218

Cronin, C., & MacLaren, I. (2018). Conceptualising OEP: A review of theoretical and empirical literature in Open Educational Practices. Open Praxis, 10(2), 127–143. https://search.informit.org/doi/10.3316/informit.559671315718016

Deci, E. L., Eghrari, H., Patrick, B. C., & Leone, D. R. (1994). Facilitating internalization: The self‐determination theory perspective. Journal of personality, 62(1), 119-142.

Fraser, N. (2008). From redistribution to recognition? Dilemmas of justice in a “postsocialist” age. In, Seidman, S. & Alexander, J. C. (Eds.). The new social theory reader. Routledge, 188–196.

Goodenow C. (1993). Classroom belonging among early adolescent students: Relationships to motivation and achievement. The Journal of Early Adolescence, 13(1), 21–43. https://doi.org/10.1177/0272431693013001002

Howard, J. L., Bureau, J., Guay, F., Chong, J. X. Y., & Ryan, R. M. (2021). Student motivation and associated outcomes: A meta-analysis from self-determination theory. Perspectives on Psychological Science, 16(6), 1300–1323. https://doi.org/10.1177/1745691620966789

Kalir, J. H. (2020). Social annotation enabling collaboration for open learning. Distance Education, 41(2), 245–260. https://doi.org/10.1080/01587919.2020.1757413

Kalir, J. H., & Dean, J. (2018). Web annotation as conversation and interruption. Media Practice and Education, 19(1), 18–29. https://doi.org/10.1080/14682753.2017.1362168

Kelly, A. E., & Clinton-Lisell, V. (2025). Strengthening online psychology students’ sense of belonging with social annotation: an experimental study. Psychology Learning & Teaching, 24(1), 60–73. https://doi.org/10.1177/14757257241295302

Lambert, S. R. (2018). Changing our (dis)course: A distinctive social justice aligned definition of open education. Journal of Learning for Development, 5(3), 225–244. https://eric.ed.gov/?id=EJ1197463

Lazzara, J., & Clinton-Lisell, V. (2024). Using social annotation to enhance student engagement in psychology courses. Scholarship of Teaching and Learning in Psychology, 10(4), 605–611. https://doi.org/10.1037/stl0000335

Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860. https://doi.org/10.1016/j.cedpsych.2020.101860

Tlili, A., Burgos, D., Huang, R., Mishra, S., Sharma, R. C., & Bozkurt, A. (2021). An Analysis of Peer-Reviewed Publications on Open Educational Practices (OEP) from 2007 to 2020: A Bibliometric Mapping Analysis. Sustainability, 13(19), Article 19. https://doi.org/10.3390/su131910798

Yust P. K. S., Liu J., Hard B. M. (2021). Course belonging and engagement in introductory psychology: Why they matter and what predicts them. Scholarship of Teaching and Learning in Psychology, 7(3), 206–227. https://doi.org/10.1037/stl0000295

Overview of study details

About the author

name: Virginia Clinton-Lisell, Ph.D.

institution: University of North Dakota

Dr. Virginia Clinton-Lisell began her career in education as an ESL teacher in New York City. She then obtained her PhD in Educational Psychology with a minor in Cognitive Science at the University of Minnesota where she was trained in educational research. She has published over 60 articles in education research and teaches courses in education research, program evaluation, and psychological foundations of education. Her current research focuses on the psychology of reading comprehension and open education. She is a primary researcher for the Open Education Group, which sponsors a fellowship for early career researchers.