8 Assessment Techniques for Higher Education and Student Affairs Professionals

The assessment course taught me so much about some of the most important efforts within higher education and student affairs that primarily take place behind the proverbial curtain. I was able to work with Dr. Carol Powers in the graduate college to conduct an assessment project to define the impact of the 360 Critical Skills for Professional Development program that is offered to all current graduate students at Oklahoma State University. The results demonstrated that participants wish to see more online options available for those who may be remote students or students who work during typical business hours.

360 Critical Skills for Career Success

Overview

This assessment project concerns the graduate college and their existing program for professional development. The program is an optional, supplemental tool that all current graduate students at Oklahoma State University are eligible to participate in. The completion of said program will provide students with micro-credentials that they can display on their LinkedIn profiles, as well as a medallion to commemorate their success within the program. The program is designed to allow graduate students at Oklahoma State University the opportunity to enhance career development skills and professional development skills within six competencies: communication, instruction, leadership, research services, wellness, and management. Although the program has been in operation for the last six years, very few participants have seen it through to completion. This executive summary discusses the need for assessment, methodology, findings, and recommendations.

Need

There was a need for more information regarding the impact of the 360 Critical Skills program. The graduate college needs to design a basic assessment strategy to better track how well the program is working for those who participate. If the graduate college had a better understanding of the program’s impact on past and current participants, then the graduate college could potentially redesign the program to better serve future participants and potentially increase their overall participation and completion rates. If the program could be redesigned to better serve its participants, then perhaps more students would be interested in participating in the program. Since the program is still fairly new and no assessment has been conducted concerning its impact, formative assessment was needed to better understand its impact as well as how to improve the program.

Client and Stakeholders

The client for this assessment was Dr. Carol Powers. Dr. Carol Powers is the assistant dean in the graduate college, and she is the program coordinator for the 360 Critical Skills program.

Dr. Carol Powers

Assistant Dean, Oklahoma State University

carol.powers@okstate.edu

The stakeholders who may be affected by the outcome of this project are those who work within the graduate college. This includes the administration, staff, and research assistants within the graduate college.

Purpose

The primary purpose of this assessment project was to define the impact of the existing program and potentially frame a means to improve it. Participants in the program within the last six years were studied through their exit surveys for the graduate college to determine how to better design the methods in which they track this information moving forward to improve the outcomes of the 360 Critical Skills program as well as the participation rates and completion rates for future participants. A new survey was also circulated to the students within the graduate college to ask for feedback concerning the program and ways to improve it. The survey asked more specific questions concerning the students and what they would like to see offered at Oklahoma State as well as what professional development means to them. The Graduate College sent out the survey link in their Monday Memo on November 7, November 14, and November 21.

Assessment Questions

• How can the graduate college define the impact this program has on its participants?
• How can the program be redesigned to increase participation?
• How can the graduate college boost advertising of this program to increase participation?
• How can the methods utilized by the graduate college be improved to track progress and completion records for participants?
• How can the graduate college improve survey collection to ensure that students are completing said surveys?
• How can the graduate college utilize existing data to better understand how well the program is working?

Learning Outcomes

• Students will be able to define in words what personal development means to them.
• Students will discuss which programs they would like to see offered at OSU.
• Students will be able to decide how they prefer to engage with campus programs and organizations.

Evaluation Approach

The assessment approach for this project falls under the program-oriented evaluation approach. The assessment relies on the utilization of surveys and participant feedback and focuses primarily on the evaluation’s end results.

Assessment Design

The assessment design is cross-sectional. One aim of the assessment was to break down the data into subgroups and to note any trends that can be found between these subgroups. Both existing data and new data were utilized, and the results were compared during the data analysis portion of the project. Six existing data surveys from past events were utilized for the existing data portion of the project via Qualtrics. The graduate college also circulated a new survey to all students within the graduate college to collect new data to better understand students’ knowledge and opinions concerning the 360 Critical Skills program.

No demographic information for the respondents of the existing data was available, but the new survey that was sent to the students within the graduate college asked for their degree level as well as their primary college within the graduate college. One student reported that they were earning a graduate certificate, four reported they were master’s level students, and four reported they were PhD level students. When asked what academic college their primary degree was associated with, one student responded with the college of arts and sciences, one student reported with the college of engineering, architecture, and technology, one student reported with the graduate college, three students reported with the college of education and human sciences, and three students reported with the college of veterinary medicine.

Methods

This assessment project utilized existing survey data while it also collected new data with surveys that were administered and collected through Qualtrics. The data analysis portion of this assessment project was quantitative and qualitative. The goal of the data analysis was to convey the statistical significance of the results, to discover any underlying themes or trends within the data, and to answer the assessment questions. I will send the data analysis results to my project partner so that they can forward them to the stakeholders as needed.

The quantitative analysis portion consisted of analyzing differences in outcomes among the groups to consider how the differing results potentially answer the assessment questions. For the qualitative analysis, the responses received were utilized to better understand what current students would like to see more of within the program and to also frame a better idea of how to potentially improve the program to meet the specific needs of current participants.

Results

The existing data sets had a total of 399 responses to the eight proposed questions. They have been added into a total number for all six surveys that were utilized for this assessment project. In total, there were 341 “strongly agree” responses, 51 “somewhat agree” responses, 5 “neither agree nor disagree” responses, 2 “somewhat disagree” responses, and 0 “strongly disagree” responses. Some of the questions are as follows:

Q1: “The session outcomes were clearly defined”

Q3: “The content was organized and easy to follow”

Q8: “Overall, I was satisfied with this session”

Overall, 85% of survey respondents noted that they strongly agree with the survey while 12% stated that they somewhat agreed. Only 2 respondents noted that they somewhat disagreed, and it was with questions 1 and 8. I have no identifying information, so I cannot determine if it was the same respondent who chose “somewhat disagree” for questions 1 and 8. These results suggest that the overall objectives or goals of the event were not clear to everyone in attendance and that there was at least one respondent who stated they were not satisfied overall with the session they attended. I am uncertain of whether the events took place in-person or via Zoom. The results also show that there were no unanimous responses for any of the questions given.

One question that was asked in the new survey was how students preferred to engage with on-campus groups and organizations. 61% of respondents stated that they preferred to engage online while only 36% responded that they prefer in-person. The majority of respondents also stated that they prefer to engage with organizations and campus groups during the evenings. Many of the options that students can choose to participate in for credit towards completion of the program are only offered in person and during the day.

When asked what programs they would like to see offered at OSU, one respondent stated they would like to see “internal workshops available on zoom to commuter and nontraditional graduate students who are not on the main campus in Stillwater, so they feel included.”

Discussion

The findings from this assessment project suggest that a majority of participants in the 360 Critical Skills program wish to see more options provided to online students and nontraditional students who may work during the day. More online options could potentially allow more online students to participate in the program, thus increasing participation rates. The lack of online options could also explain why many students begin the program but fail to see it to completion. Students may begin working on the program, but when they find that many of the workshops and events occur during the day, they may not be able to attend enough to earn full credit.

Recommendations

More online options

The current program does not provide online students with much opportunity to participate, so designing more online events is ideal for increasing participation. The majority of respondents reported in the survey that they prefer online engagement over in-person.

Online wellness option. For the “wellness” competency, participants can satisfy one of the requirements by visiting the Colvin 15 times in one semester. An option to increase participation could be to allow students to visit any gym 15 times in one semester. Most gym franchises such as Planet Fitness or 10Gym have mobile apps that track when members check in for a workout session. Student could easily provide evidence to show when they visited the gym for credit towards a level 1 wellness badge.

Online sessions and weekend sessions. Another option to increase participation could be to host more online events for remote students unable to commute to campus or events on weekends where more working students could come and participate.

Online program overviews. The graduate college should consider scheduling drop-in sessions, online as well as in-person, to provide more information to prospective students about the program and how to complete the necessary requirements. Partnering with faculty within the graduate college could also potentially open opportunities to drop into lectures in person or via Zoom to give a quick presentation about the program and how to get involved.

Limitations

The limitations of this study concern the data utilized. The quantitative data sample was small, and the sample utilized for qualitative data was even smaller. For future assessment projects, the graduate college should consider interviews or focus groups with current participants to better understand their thoughts on the program and to gather suggestions on how to improve it.

Conclusion

The 360 Critical Skills Program allows OSU graduate students to gain additional professional skills while earning their respective degrees. The program is optional, but students who successfully see it to completion receive micro-credentials that can be displayed on their personal LinkedIn profiles as well as a medallion to commemorate their success with the program. The program has been operating within the graduate college for approximately six years, but there has been an ongoing need for assessment regarding the impact of the program on its participants.

A sample of existing surveys were analyzed to better understand the success of the program since its inception and a new survey was also circulated to all current students within the graduate college to collect more qualitative data pertaining to students’ opinions about the program as well as any suggestions for means to improve it.

The results from this assessment project determined that the stakeholders should consider providing more online options for remote students as well as more options within the evenings for those who may work during the day. If the program offered more variety with their event scheduling, then more students could potentially participate, thus increasing participation and completion rates.

References

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th). Pearson.

Graduate College. (n.d.). 360° CSCS Program: Developing the person, the scholar and the professional. Oklahoma State University. https://gradcollege.okstate.edu/professional-development/index.html

 Appendix A

Existing survey data questions

The following questions were on the exit surveys that participants were asked to complete after attending an event hosted by the graduate college. Attending these events can satisfy certain requirements for the 360 Critical Skills program and completing the exit survey is necessary to gain the credit. Survey links are sent to participants via email by Dr. Carol Powers after the event has taken place. The survey links redirect students to Qualtrics where the surveys are administered.

Q1: “The session outcomes were clearly defined”

Q2: “The session outcomes were achieved”

Q3: “The content was organized and easy to follow”

Q4: “The facilitators were useful and helpful”

Q5: “The facilitators were well prepared”

Q6: “The facilities and equipment were favorable to learning”

Q7: “The registration for this session was user friendly”

Q8: “Overall, I was satisfied with this session”

Appendix B

New survey data questions

The following questions were on the survey created for the purpose of assessing current students’ opinions regarding the 360 Critical Skills for Career Development program as well as what they would like to see offered by the graduate college or by the university in general. The survey was created and administered in Qualtrics, and the survey link was circulated to all current graduate students via the graduate college Monday Memo.

Q1: What is your primary degree program?

• Master’s
• PhD
• EdD
• EDS
• Graduate Certificate
• Non-Degree Seeking

Q2: Which academic college is your primary degree associated with?

• College of Arts and Sciences
• College of Engineering, Architecture, and Technology
• College of Education and Human Sciences
• Center for Health Sciences
• College of Veterinary Medicine
• Ferguson College of Agriculture
• Graduate College
• School of Global Studies and Partnerships
• Spears School of Business

Q3: Had you heard of the 360 Critical Skills for Professional Development program prior to this survey?

• Yes
• No

Q4: What days and times allow you allow for you to participate in student organizations or campus programs? Select all that apply.

• Morning
• Afternoon
• Evening
• Weekdays
• Weekends

Q5: How do you prefer to engage with on-campus programs or organizations? Select all that apply.

• In-person
• Zoom

Q6: What does professional development mean to you? (Open ended)

Q7: What are some topics within professional development that interest you? (Open ended)

Q8: What are some programs that you would like to see offered at Oklahoma State University? (Open ended)

We were also able to create a poster which we presented at a virtual poster fair that our class hosted.

ACPA and NASPA (2015) Competency Statement:

The competency concerning assessment, evaluation, and research encompasses the skills I gained from the assessment techniques course last fall. During the class, we worked on assessment projects alongside a member of the staff or faculty to better understand how much work goes into assessment research and to better understand the importance of this work. Now in my creative component I have been working on my own assessment portion with a research study concerning stigma and its effects on the academic success of students in recovery. The work has not always been easy, but I feel like seeing the responses to my survey has been quite rewarding and I look forward to utilizing my findings in my practicum next fall.