EXECUTIVE SUMMARY

Meints, Kimberly

School of Educational Foundations, Leadership and Aviation, Oklahoma State University

HESA 5343: Assessment Techniques

Dr. David Mariott

December 2021

Cowboy Strong Assessment

Oklahoma State University (OSU) prides itself in the identity of the “Cowboy Family, a culture that cares for each other” (Mariott et al., n.d., p. 9). One way that OSU practices a culture of care is through the Cowboy Strong program, which grants one-time funding for unexpected, emergency needs to OSU students. The maximum award per student is $500, which is provided by a variety of donors. The Cowboy Strong program was originally managed by the Vice President for Student Affairs office but was moved to the Office of Student Support and Conduct (SSC) in Spring 2020 shortly following that office’s role expansion (formerly known as the Office of Student Conduct).

SSC staff believe that awareness of this program comes primarily through word of mouth, as students tell their friends after receiving an award or when faculty and staff make referrals. Links to more information about the fund have been added to the Basic Needs Resource Center webpage[1] hosted by Leadership & Campus Life. Students can also access that webpage through the Canvas web portal (which hosts many of their class materials) through the Student Resources “course.” There are plans to add a similar course for faculty and staff. Canvas can be accessed via a web browser or smartphone app.

The intent of this assessment is to learn more about which students are applying for aid through the Cowboy Strong program, what emergencies prompt their applications, and for what purposes they intend to use the award. The results of the assessment will be used internally by SSC to gain clarity on current program utilization and to help them define specific steps to assist more students in an equitable manner. The results may be shared with others on campus, such as student affinity groups, faculty, and staff, to improve awareness, or with campus leadership to guide decision-making. It may also be shared with donors to expand the reach of the program.

Need for Assessment

OSU conducted four campus-wide assessments during 2019 and 2020, which were then analyzed by a working group within the Division of Student Affairs. The director of SSC was a member of this group (Mariott et al., n.d.). These assessments measured student mental health; student habits and behavior related to current areas of health-related interest; basic and additional needs; and sexual violence. Particularly salient to this assessment are the student responses to the Basic Needs survey (p. 7) and their responses related to the Perceptions and Utilizations of Services (p. 9). Of students who completed the assessment, 51 percent reported at least one form of basic needs insecurity (BNI), including food or housing insecurity, with 15 percent reporting homelessness (p. 7). Of those who received emergency aid (though not necessarily through the Cowboy Strong program), 75 percent said it helped relieve their stress, while 68 percent indicated that the funding helped them avoid dropping out (p. 7). This clearly indicates the importance of providing emergency funding to students. The report also details the student groups who reported the highest rates of insecurity, including sexual and gender minorities (LGBTQIA+), Pell Grant recipients, historically marginalized racial and ethnic backgrounds, and first-generation students.

More concerning, and indicative that there is room for the program to grow, only 19 percent of students who said they had experienced BNI used emergency funding to meet those needs (Mariott et al., n.d., p. 9). More than half of the students with BNI were unaware that there were support services available at OSU; more than a quarter of students said they were too embarrassed to ask for help from campus support services (p. 9). The finding that students “are most likely [to] utilize their friends for informal help and support” (Mariott et al., n.d., p. 9) reinforces SSC’s intentions to provide outreach to student groups.

Between May 2020 and August 2021, 328 applications were submitted to the Cowboy Strong program. I analyzed applications submitted between January 2021 through July 2021 to attempt to exclude the emergency needs caused by the onset of the COVID pandemic and to eliminate the most recent applications that had not yet been reviewed at the time the data was gathered.

Client/Stakeholders

The main contact within the Office of Student Support and Conduct is Ross McClure (ross.mcclure@okstate.edu, 405-744-5470), a coordinator. Other stakeholders include:

1. Student Support and Conduct staff: SSC staff review Cowboy Strong applications and decide when to award aid and for what amount. They make referrals for students to the application as well as to other resources as needed. They educate faculty and staff at OSU and will develop an outreach campaign to various groups based on the results of this assessment.
2. OSU Students: Cowboy Strong funding is only available to OSU students in Stillwater, Oklahoma. This group makes up the pool of people who are eligible to apply for funding, but they may also make referrals within their personal networks.
3. Vice President of Student Affairs and staff: SSC is housed within the Division of Student Affairs at OSU. Not only are the SSC staff ultimately accountable to the Vice President for the success of the program, but the Cowboy Strong program was originally administered through that office, so they may have ongoing personal interest in its success. Campus-wide, many faculty and staff refer students in need to the Vice President’s office. The Vice President also administers the budget for the division and could help with future fundraising efforts for the Cowboy Strong program.
4. OSU Donors: The Cowboy Strong emergency funding program is entirely supported by external donors. These donors may be interested in who, generally, their gifts are helping and in what ways. Accurate and interesting information can illustrate for them the benefits that OSU students experience as a result of their generosity and encourage them to continue their support in the future. This information could also help recruit new donors in order to expand the program.
5. Student Support Entities: On a campus as large as OSU, there are many organizations and departments that seek to provide student support. Examples include student clubs, the Department of Housing & Residential Life, advising offices, academic departments, health and wellness departments, and others who may make student referrals. Individually, it is helpful to share memorable information with these entities to assist them in making timely referrals to the program. As a collective, keeping these organizations informed also helps create a culture of care, as campus partners remember that we look out for one another.

Overview

This assessment closely analyzed the data from 75 applications to the Cowboy Strong emergency funding program submitted between January and July 2021. SSC awarded 47 students out of Cowboy Strong funds or as a Meal Plan deposit for a total amount of $22,925.00. While every academic college, student classification, and racial and ethnic group tracked by OSU was represented, the applications were not equally distributed according to the make-up of the student body. Individual demographic subgroups were too small to make a determination regarding equitable award practices between them, but a few points of interest are discussed.

Focus of the Evaluation

Currently, each Cowboy Strong application is reviewed individually, so there is no existing aggregate data on overall program trends. Gathering and analyzing data about the program since it has been housed in the Student Support and Conduct office will give staff a better understanding of who is using the program and why and will help guide staff in outreach efforts to student groups and colleges. Because this program is intended to help students meet immediate, emergency needs, timely evaluation will better serve the OSU students who can most benefit from it. Further questions may be developed for future assessment projects based on the results of this evaluation. A few recommendations are included at the end of this report.

The purpose of this assessment was to determine for what reasons students at OSU utilize the Cowboy Strong emergency funding program and which student demographic groups are currently represented. The questions that were addressed by this assessment were:

1. When applying for assistance, what causes do students indicate as their emergency?
2. When applying for assistance, what needs do students identify?
3. Are there trends between student demographics and reasons for applying?
4. Do certain demographic groups of students receive assistance through the Cowboy Strong application at a higher rate than others? Are some groups denied at a higher rate?

Assessment Methodology

Approaches Used

This assessment followed a decision-oriented evaluation approach, particularly utilization-focused. The Office of Student Support and Conduct is relatively small and invested in the Cowboy Strong emergency funding program. Therefore, the “personal factor” is available and is a necessary substitute for the lack of expertise in the evaluator and the limited data used within the study. The staff will provide context to my analysis. The office is prepared to make decisions based on the results of the assessment.

Assessment Design

The assessment followed a cross-sectional descriptive design. Because the assessment was limited to existing data, the design was constrained to less complex questions and results than would be available through a case study or time-series design. This assessment may act as a pilot for developing future studies and set baseline data for future comparison. It is for this specific purpose that the data from 2020 was excluded, as the onset of the COVID-19 pandemic may have influenced the number of applications and reasons for applying in ways that are less typical.

Sample

Sampling was not used in the current assessment due to the small population size within the existing data.

Data Collection Method

The Office of Student Support and Conduct (2021) provided the application data for the application submission date range of January 2021 to July 2021 (Appendix A). There were 75 records after removing one duplicate. The data set included the students’ responses to the electronic application questions (Appendix B) as well as some demographic information provided through OSU’s Office of Institutional Research and Analytics (IRA).

Data Analysis Techniques

When analyzing the data for this assessment, I began by identifying broad categories. The demographic information in the source data included fields such as gender, academic major, classification (i.e., freshman, sophomore, etc.), race and ethnicity, and the state from the student’s permanent address. I further consolidated majors into academic colleges and classification into student level (i.e., undergraduate, master’s, and doctoral). Additionally, IRA provided birthdates, which I used to derive age at time of application. With the demographic categories defined, I compared the applicant population demographics to the OSU student body demographics using data from OSU’s Institutional Research and Analytics (2021) department to determine to what degree the applicants reflected the student body.

Then, I reviewed the application data using emergent coding to identify the emergency causes for the application (illness and injury, wage loss, transportation, etc.) and needs (rent, books, medical bills, etc.). The application data contained several questions regarding the student’s need and attempts to resolve the need on their own, as well as a question regarding their inclusion in specific population groups (i.e., single mother, former foster youth, and veteran). The application is included as Appendix B. Within the dataset, this particular data was all contained within a single field, so I parsed the data into eight separate fields to better analyze student responses. It is clear from the free responses that some of these students may have applied more than once at different times, but I do not have enough data to confirm their identities or if their previous applications are included in this dataset. Therefore, I am treating each entry as an individual student for the purpose of this assessment.

Finally, I considered the reasons offered by particular sub-groups to determine if some reasons are more common among certain students. I also compared award amounts and denials for these specific groups.

Discussion

Applicant Demographics

OSU has five academic colleges and an administrative unit called “University College” (UC) which oversees undecided students, high school concurrent students, exchange students, and pre-engineering students, among others. The three colleges with the highest number of applicants were Education and Human Sciences (EHS; 20 applicants); Engineering, Architecture, and Technology (CEAT; 18 applicants); and Arts and Sciences (CAS; 17 applicants). The Spears School of Business (SSB; 11 applicants), Ferguson College of Agriculture (AGRI; four applicants), and UC (one applicant) were all underrepresented among the applicants. Nevertheless, with the highest rate of applicants from a college at 0.6 percent (EHS), it is clear that all colleges could benefit from additional information about the program. Additionally, no doctoral students from SSB or any graduate students from AGRI applied for assistance. One student from OSU’s Center for Health Sciences (CHS) in Tulsa, Oklahoma, applied but was ineligible because they were not enrolled on the Stillwater campus. Further details may be seen in Figure 1.

Next, I compared the make-up of the applicants by race and ethnicity and compared the rate within the applicant population with the OSU student body (Figure 2). Overall, two subgroups were overrepresented within the applicant population, Black/African American (nearly four times higher) and Nonresident Alien (about three and three-quarters times higher). White students were the only subgroup that were significantly underrepresented at about half of their typical rates. This indicated that knowledge about the Cowboy Strong program is likely distributed well across racial and ethnic groups on campus, although further outreach would still be encouraged as smaller populations may be more susceptible to losing institutional knowledge when particular members graduate or otherwise leave the institution.

Overall, more women applied for aid than men, with 56.0 percent reporting female as their gender. When disaggregated to include student level (Table 1), it is clear that undergraduate women applied for the most aid (50.7 percent), followed by undergraduate men (21.3 percent) and then master’s level men (14.7 percent). Senior undergraduate students made up over a third of applicants, followed by master’s students at 18.7 percent and sophomores at 17.3 percent.

Special populations did not make up a significant number of students, with only ten students indicating that they were single mothers, former foster youth, or veterans. However, one applicant noted that they were taking care of a sister through the foster care system. Some students may not know if they qualify as a single mother or former foster youth. As an example, one student indicated that she was pregnant and engaged and did not identify herself as a single mother. Former foster youth may not know if this application refers to the Free Application for Federal Student Aid (FAFSA) definition or some other requirement.

Figure 1

Applicant College by Student Status

EHS = Education and Human Sciences; CEAT = Engineering, Architecture, and Technology; CAS = Arts and Sciences; SSB = Spears School of Business; AGRI = Ferguson College of Agriculture; UC = University College; CHS = Center for Health Sciences

Figure 2

Applicants vs Student Body by Race and Ethnicity

Table 1

Applicants by Gender, College, and Student Level

UG = undergraduate; MS = master’s; DS = doctoral

Application Insights

1. When applying for assistance, what reasons do students indicate as their emergency?

The most common reason mentioned by students as an emergency is something that I have grouped together as “unspecified financial difficulty” (34.7 percent). These students explained that they were unable to pay their bills but did not explain a specific reason why. More than half of these students did not mention any other reason, and only 50.0 percent of these applications were awarded.

The second most common reason was lost wages by the student (32.0 percent). Half of these also indicated COVID as a reason for their emergency, as their hours were reduced, places of employment closed, or they were required to quarantine. Most of these students (62.5 percent) indicated at least two causes for their emergency. Some had concerns about transportation, which was another frequently mentioned need (16.0 percent among all applicants).

Illness, injury, and death of family members was the last significant group of needs identified at 13.3 percent. Students reported traveling to take care of sick and dying parents and grandparents, looking after siblings, and navigating the process of burying a parent. All reasons are listed in Figure 3.

Figure 3

Applicant Causes for Submitting Application by Award Status

2. When applying for assistance, what needs do students identify?

Students indicated many different ways they intended to use the funding award. Rent (44.0 percent), groceries (40.0 percent), and other bills (40.0 percent) were the top three needs students mentioned on their applications. “Other bills” included items such as utilities, cell phone plans, and car insurance. While the application explicitly states that students will not be awarded funds to help pay for tuition, 17.3 percent of applicants indicated this need. Less than half of those applications were awarded, although all of those students indicated at least one another need.

Other needs mentioned were books and school supplies, dependent expenses, and medication or medical bills. All needs are listed in Figure 4.

Figure 4

Applicant Needs Indicated on Request by Award Status

3. Are there trends between student demographics and reasons for applying?

When reviewing the reasons for applying and student race and ethnicity categories, there were not enough applicants in certain groups to identify a trend. The three most highly represented groups, White, Nonresident Alien, and Black/African American, indicated the widest spectrum of reasons, indicating 10, 8, and 8 respectively. White students indicated lost wages by the student as the most common reason (46.2 percent), but other groups did not have significant differences among reasons. A similar review of reasons by student level and classification revealed similarly inconclusive results, although seniors did report lost wages by the student as the most common reason (54.2 percent).

4. Do certain demographic groups of students receive assistance through the Cowboy Strong application at a higher rate than others? Are some groups denied at a higher rate?

Overall, there were not enough students in each race and ethnicity category to calculate significant rates of approval or denial (Figure 5). However, when comparing the rate of awarded White students with all other categories, less than two percentage points separate the groups (White, 61.5 percent; All Others, 63.3 percent). Results were similar when comparing students by classification. However, sophomores had the highest award rate at 84.6 percent, and freshmen and juniors had the lowest award rate at 50.0 percent.

There is a little bit more of a difference among awards when students are considered by college. Arts and Sciences had the highest number of applicants (13) and one of the highest award rates (76.5 percent). The next two largest colleges by applicant numbers, Engineering, Architecture, and Technology and Education and Human Sciences, each had 12 applicants, but CEAT students were awarded at 63.2 percent, while EHS students were only awarded 54.5 percent.

Another interesting way to look at awards by college is to consider the average award amount (Table 2). While students could receive “Cowboy Strong” awards or “Meal Plan” awards, for this purpose, I only looked at the former. Again considering the largest three colleges by applicants, CEAT students received the highest average award at $562.50. CAS students received $450.00 on average, and EHS students received $392.50. Although only five students from the Spears School of Business received Cowboy Strong funds, their average award was $700.00.

Figure 5

Applicants by Race/Ethnicity and Award Status

Table 2

Cowboy Strong Award Amounts by College

AGRI = Ferguson College of Agriculture; CAS = Arts and Sciences; CEAT = Engineering, Architecture, and Technology; EHS = Education and Human Sciences; SSB = Spears School of Business; UC = University College

Recommendations

Faculty and Staff Awareness

The recent assessments conducted by OSU (Mariott et al., n.d) identified specific populations of students who may be more vulnerable to Basic Needs Insecurities. These include LGBTQIA+, Pell Grant recipients, historically marginalized racial and ethnic backgrounds, and first-generation students. Some of these student groups are considered “hidden populations” because they may not make a formal declaration of their status and may not be identifiable in the data that OSU currently collects on students. Outreach to student groups and support offices for these students may be especially effective in helping meet urgent student needs. A few departments that would be a good connection point would include the Office of Multicultural Affairs (LGBTQIA+ and historically marginalized racial and ethnic backgrounds), the Office of Scholarships and Financial Aid (Pell Grant recipients), and First Year Success (first-generation students).

The low application rates for certain colleges suggest that an awareness campaign for faculty and staff would also be appropriate. The administrative structure for each college will require different strategies for communicating (i.e., colleges with centralized advising versus colleges with faculty advising). It is imperative to include all types and levels of staff and faculty, as students may develop a relationship with anyone and could disclose their need.

Student Awareness

The previous assessments also indicate that any awareness campaign will also need to incorporate messaging to reduce the stigma of applying for aid. Messaging should include information about the purpose of the funds to improve utilization. Students should be encouraged to help their friends apply when they have a need, in addition to taking advantage of the program for themselves.

The nature of a university is continual turnover in students as new students arrive and others graduate. It is important, therefore, to ensure that outreach to student groups is consistent every year to support continuity of awareness.

Application Suggestions

There may be ways to optimize the application to make it easier for students to complete. Like all writing, completing an application for assistance is its own genre, and it was clear that students were not familiar with how to answer some of the questions. Many students did not include enough information, while others gave too much. A checkbox list of causes and needs with “other” fill-in options might make the application clearer for the student and for the reviewer. Collecting referral information would also help SSC staff identify where information campaigns are successful and where more work may need to be done.

Currently, students may indicate if they are single mothers, former foster youth, or veterans. It may help to clarify who qualifies as a single mother or former foster youth. This question could also add more categories, even if they are not directly tied to additional funding at this time, such as LGBTQIA+, Pell Grant recipients, and first-generation students. This would allow future assessments to be completed to identify the utilization rates for these groups (or at least the rate at which students self-identify).

A number of the needs indicated by students in this data set are on the list of “expenses not covered” on the Cowboy Strong Emergency Funding website[2]. SSC staff may want to review the website to ensure that the information provided there is consistent with awarding practices and make adjustments where needed.

Limitations and Suggestions for Future Assessment

A few of the limitations for this data include the restricted number of applications reviewed, with less than one year of data. Additional questions could not be answered in this assessment because the data was not collected on application, such as referral source. Another helpful data point would be the number of applications that were started but never submitted and if there is a common point where students abandon the application.

A focus group of students to review the application could help optimize it for understanding and ease of completion. The assessment could also collect student perceptions of the application and the degree of difficulty to find it. These responses could be triangulated with the previous suggestion of incomplete applications to increase validity.

Conclusion

The Cowboy Strong emergency funding program is an important part of OSU’s commitment to supporting members of the Cowboy Family who are in need. Previous assessments at OSU have identified the need for emergency funding and the tangible benefit to students who receive it. The Office of Student Support and Conduct has successfully managed the program in the last 18 months, ensuring broad representation among student applicants. With the information from this assessment, they are well-positioned to expand the impact of this program by creating more targeted awareness campaigns and preparing for future assessments.

References

Institutional Research and Analytics. (2021). Cowboy data round-up: Spring student profile (Spring 2021) [Data set]. Oklahoma State University. https://ira.okstate.edu/cdr/index.html

Mariott, A., Day, J. M., Gage, K., Lester, C., Pratt, K., & Robinson, J. (n.d.). What our students are telling us about their health and wellbeing: Key findings from four OSU-specific assessments.

Office of Student Support and Conduct. (2021). [Applicant data for the Cowboy Strong emergency funding program] [Unpublished raw data]. Oklahoma State University.

Appendix A Data Use Agreement

Appendix B Cowboy Strong Application Questions