Chapter 8 - NSSE 2020 at UH Manoa: From Record High Response Rate to Innovative NSSE Data Tools
The Mānoa Institutional Research Office (MIRO) shifted from its past supporting role in producing NSSE reports to a proactive role in leading a successful NSSE administration. The office implemented a campus-wide marketing plan that increase survey response rate to 52%, which is more than triple the size of the response rate compared to 2011 when there was no marketing plan at all,and created innovative data & report dissemination tools to efficiently disseminate data among offices and programs across the campus in a timely manner.
Abstract
Introduction
The National Survey of Student Engagement (NSSE) was launched in the year 2000 at Indiana University Bloomington and is now considered one of the most important undergraduate surveys in the nation. The NSSE questionnaire collects important student information across five categories, using over 100 questions that survey student experiences that range anywhere from coursework, campus activities and services, to overall student satisfaction. Universities across North America administer the NSSE survey to gather valuable data that is then used to facilitate different campus initiatives and programs, such as assessment, accreditation, and program improvement.
Achieving the highest survey response rate possible is a key step of a successful survey administration. However, many universities, including UH Mānoa, are struggling with survey fatigue and are facing many challenges when it comes to improving response rates, especially because students are frequently bombarded with surveys and questionnaires.
For the longest time, the Mānoa Institutional Research Office (MIRO) did not have a comprehensive NSSE survey promotional plan in place. In 2011, the response rate was only 16%. It wasn’t until 2015 when the office first implemented a campus-wide marketing plan that the NSSE response rate rose, doubling to 32%. As marketing strategies and efforts developed further, the office saw an increased survey response rate in 2020 from 32% to 52%, which is more than triple the size of the response rate compared to 2011 when there was no marketing plan at all.
Figure 1: Impacts of Effective Survey Marketing Campaign
(Corresponding Video Here)
The numbers speak for themselves and are proof that MIRO’s survey promotional strategies work. To put this in a national context, in 2020, the average response rate for institutions with more than 10,000 undergraduate students (also the category UH Mānoa falls under) was 23% – that means over a hundred universities only had an average of 23%, which is significantly smaller when compared to Mānoa’s 52% (see Figure 2).
Figure 2: NSSE Response Rate in Comparison
(Corresponding Video Here)
A Comprehensive NSSE Marketing Plan
To disseminate the 2015 NSSE survey results, MIRO developed a number of in-house NSSE Web Apps that allows offices to generate their own customized NSSE reports and focus on specific student populations they are interested in. Some departments or student populations have a smaller number of students, so to make NSSE data more useful to them, it is important to achieve the highest response rate possible to have a better representative sample size. As a large public research university that has many commuting students, it is quite challenging to get students to complete a long student survey. Built upon everything learned from the 2015 NSSE survey administration, MIRO continued to create avenues to promote the NSSE survey across different audiences in 2020. Some of the main efforts include:
-
working with the Office of Communication to produce a series of news stories, email blasts, and the UH event calendar.
-
creating and distributing large-scale campus promotional materials (including ground stakes, banners, posters, and flyers).
-
working with the university IT Services to embed a pop-up survey link in the university’s learning management system.
-
connecting with colleges, departments, and different offices to promote surveys within their student populations.
-
securing funding for attractive survey incentives.
-
partnering with faculty of high enrollment courses and speaking in their classrooms.
-
increasing awareness about NSSE among students by hosting campus events, such as NSSE fairs, and offering free yoga.
-
building data tools that allow different levels of academic units (such as program, major, department, and college) and different offices to track survey completion rates on a daily basis while helping them adjust their promotional strategies.
Figure 3: Avenues to Promote the NSSE survey
(Corresponding Video Here)
This marketing campaign was truly multifaceted. In order to effectively carry out these efforts in a timely manner, MIRO drafted a 9-month marketing timeline (see Figure 4) that was broken down into four critical time periods: (1) Early Preparation, (2) the Gear-up Month, (3) the Launch Day, and (4) the Survey Open window.
Figure 4: NSSE Survey Marketing Timeline
(Corresponding Video Here)
Early Preparation (Timeline Phase 1)
The first six months, from July to December, are classified as the Early Preparation period, which ends one month before the launch day. During this time, MIRO not only fulfills the NSSE survey registration and data preparation requirements, but also focuses on multiple tasks. The reason the “Early Preparation” period is the longest is because it takes time to curate and develop impactful content.
First, MIRO secures generous funding from the Student Affairs Office for survey promotion needs. Once the funding is received, much thought is put into deciding what types of incentives to offer the students. Research shows that offering incentives can increase the survey response rate by 6%. After consulting with students about attractive incentives to offer, we chose a combination of gift cards and Airpods Pro as drawing prizes for 2020’s NSSE efforts.
Figure 5: NSSE Survey Incentives
(Corresponding Video Here)
Visually appealing designs help capture students’ attention, but they take time to design. For the 2020 survey preparation, MIRO staff worked with their student assistant, who was also a professional graphic designer, to create four primary designs, each highlighting a unique aspect of the survey to attract different student groups. The designs were then modified to suit the different promotional materials (posters, flyers, and other swag) and tailored for various social media platforms (Instagram, Facebook, etc.). MIRO also reached out to the Office of Communication to post NSSE ads on the university's official social media accounts since MIRO does not own the account.
Figure 6: Visual Campaign
(Corresponding Video Here)
In order to provide easily accessible information about NSSE, MIRO prepared a NSSE survey landing webpage that addresses most questions students have about the survey. This landing page was very helpful and MIRO used it frequently when there was not enough space to provide additional details about the promotional materials.
Figure 7: NSSE Landing Page
(Corresponding Video Here)
During the 2020 Early Preparation period, MIRO also offered a series of training sessions for Mānoa faculty and staff to show them how useful NSSE data is. Many participants expressed their interest in helping promote the 2020 survey administration after attending those training sessions.
Figure 8: NSSE Data Workshops
(Corresponding Video Here)
To increase community awareness of the survey, MIRO collaborated with various offices to plan a few NSSE fairs and free yoga sessions. This expanded the opportunities for MIRO to advertise NSSE across Mānoa’s campus community.
The work involved in the Early Preparation period is not a simple task; it would not be possible to complete preparations immediately before the NSSE survey launch date, which is why MIRO takes six months to prepare. Collectively, the time spent investing in these efforts helps boost student involvement and increases survey results.
Gear-up Month (Timeline Phase 2)
One month before the NSSE survey opens is Gear-up Month. This is where MIRO picks up speed and makes final preparatory efforts for the NSSE launch day. At the beginning of 2020’s Gear-up Month, MIRO worked with Mānoa’s Copy and Print Services to print out a large quantity of ground stakes, banners, posters, and flyers.
MIRO’s Director, Yang Zhang, Ph.D., presented at the Dean’s council meeting about the plan to administer the NSSE 2020 survey, encouraging colleges to appoint a NSSE liaison to connect with MIRO. Yang also demonstrated the 2015 NSSE survey results using MIRO’s web apps, hoping to spark interest in promoting NSSE after seeing how helpful the survey data could be. After the meeting, MIRO emailed a link to the university’s Deans where they could find handouts, email templates, class announcement scripts, and printable flyers in a centralized location of a shared Google folder.
Figure 9: Promote NSSE Among the Deans
(Corresponding Video Here)
A week prior to the 2020 survey launch, MIRO staff visited each Dean’s office with a physical packet of survey promotional materials that they could hang around their corresponding buildings, classrooms, and offices. In the Dean’s packet was a detailed checklist and instructions with a NSSE timeline (see Figure 10), including suggested promotional strategies during different time frames, college-level events, and links to multiple email templates and graphics. The packet also had a short PowerPoint presentation and an announcement script they could use to talk about the NSSE survey in class. The goal was to make it as easy as possible for administrators and faculty to talk to students about NSSE. MIRO tried to do as much of the leg work for them as possible.
Figure 10: NSSE Check-list for Deans
(Corresponding Video Here)
Sending the packets to the Dean’s offices turned out to be quite effective. With colleges’ help, NSSE flyers were hung everywhere in different buildings and in locations that MIRO staff would not have otherwise thought of like in front of drinking fountains, on bathroom stalls, and in elevators. By giving promotional materials to administrators and faculty, NSSE news was able to expand and reach many places on campus more than if MIRO had acted alone. These efforts increased efficiency and saved MIRO’s time, enabling the team to work on other important tasks.
It was also helpful for students to see NSSE promotional materials a few days before the survey launch. That way, when they receive the email invitation on the survey’s launching day, it helps spark the curiosity to open the invitation. Seeing promotional materials beforehand also validates the survey, making it familiar and more approachable, in comparison to the other random surveys they receive.
Figure 11: Increase Campus Visibility
(Corresponding Video Here)
While NSSE contacted students directly by email, it was important to connect with students on a more personal level and develop in-person relationships. Most students rarely know who sends out the surveys, but they might be more inclined to fill them out if they have met the survey administrators in person. So, MIRO generated a list of classes that had the highest freshmen and senior enrollment, partnered with deans and faculty, and scheduled brief class visits to spread awareness about the survey and rewards program.
MIRO partnered with the Office of Communications, utilized the designs created during the early preparation phase, and collaborated on a series of promotional efforts to release news stories, emails, NSSE-related events on the university's weekly calendar, and social media contests.
The day before NSSE launched their survey, MIRO staff walked around campus to set up promotional materials including 30 ground stakes where the heaviest traffic occurs, such as campus mall, library, campus center, food courts, student housing, bulletin boards, and student housing, among others. MIRO also had two large banners hung at the library and the campus center since those are the most populated areas. Having a physical representation of the NSSE survey made a tremendous impact. Oftentimes, when MIRO spoke with students about NSSE, they mentioned that they had seen information about it everywhere and it sparked their curiosity about the survey. NSSE is only given to the freshmen and seniors, but even students who had not been invited were emailing us to ask if they could take the survey. That shows how successful MIRO’s campus-wide virtual campaign was.
Figure 12: Ground Stakes Around the Campus
(Corresponding Video Here)
Last but not least, to make the survey even easier for students to access, MIRO collaborated with the IT office and embedded student’s customized NSSE survey links in the student learning portal. This also helped NSSE stand out from other surveys.
The Launch Day (Timeline Phase 3)
The “Launch Day” refers to the day the survey is released. On Tuesday, 11 February 2020, thousands of freshmen and seniors received the first NSSE survey invitation by email. When logging into their learning portals, students saw the pop-up reminder MIRO and the ITS Office created, and students could click on the link from the portal to answer the survey. It also emphasized the survey's importance because the university seldom embeds survey links on their learning portal. Students also received an email blast from the Communication’s Office that encouraged them to participate in the survey. Lastly, on the day NSSE launched, MIRO staff visited 7 different classrooms and talked to over 400 students in person. These classes were important because of their high freshman or senior enrollment numbers. Thanks to the partnership with the Deans, MIRO was invited by different instructors to visit their classes. While there, the MIRO team gave a brief presentation about how NSSE could help future student experiences, how students could locate the survey, and various prizes students could win upon completing the survey.
Figure 13: First Day Response Rate
(Corresponding Video Here)
All these efforts and months of early preparation really paid off– MIRO saw a response rate of 17.9% on the first day. In one day alone, the response rate was already higher than the 2011 final survey response rate of 16%. The differences between not having a plan versus implementing major marketing strategies are incredible.
The Survey Open Window (Timeline Phase 4)
The first three weeks after the survey launches is what MIRO calls the “Survey Open Window.” This is where MIRO continues efforts to increase the NSSE response rate. Although NSSE’s survey window remains open until the month of May, MIRO chose to shorten the timeline by setting a prize-drawing date. Based on previous experience, most students submit their responses after the first three weeks of the survey’s opening. So, instead of keeping the survey window open for a few months, MIRO set the deadline to 3 March 2020, which was about 3 weeks after the survey was launched. Thankfully, due to that decision, MIRO completed the survey administration right before Mānoa’s campus shut down and transitioned to remote learning due to the COVID-19 pandemic.
To generate more community awareness for NSSE, MIRO hosted a few NSSE fairs and activities during the survey’s open window. This was also a great way to connect with students in-person and increase overall NSSE exposure through event-specific ground-stakes and social media with attractive selling points: free yoga and free giveaways. Instead of simply saying “take NSSE, take NSSE, take NSSE,” MIRO said “Free Yoga” and “Small Giveaways,” thus refreshing the message to “take NSSE” in a less repetitive way.
Figure 14: NSSE Fair
(Corresponding Video Here)
The NSSE fairs were hosted at the campus center courtyard, a prime marketing location because many students pass by it during lunch hours and between classes. The fairs had photo booths, NSSE-oriented guessing games, and small giveaways. The two free yoga sessions were arranged at the beautiful Andrews Amphitheater, by collaborating with the Blue Zones project, a wellness program that had collaboration with the university. These events served as a reminder on the campus-wide calendar, not only framing it as an institutional effort to promote the NSSE survey but also welcoming the campus community to participate in.
Figure 15: Free Yoga Events
(Corresponding Video Here)
During this remainder survey open window, MIRO continued to collaborate with the Office of Communication to provide more institutional-level awareness to the survey. Some of the institutional efforts included promoting the survey through various university channels and social media accounts on a weekly basis, publicizing NSSE events like the NSSE Fair and free yoga sessions on the UH Calendar, and publishing news stories to introduce the NSSE project to the university community at large. MIRO also worked with the Office of the Provost to include NSSE events in the weekly “Opportunities at UHM” emails.
Response Rates Tracking Web Apps
Throughout the 2020 survey’s open window, MIRO assisted colleges, departments, and offices with their own promotional efforts for their respective students. Many wanted to track student response rates to see if they needed to improve or adjust their promotional strategies, so MIRO created two Response Rate web apps during the NSSE administration. These apps tracked student completion rates by college, department, student demographic data, and MIRO updated them daily. This made it easier for users to check their college’s response rate and compare it with the university’s average. Users could also use the second web app to compare the completion rates among different colleges and departments.
Figure 16: NSSE Response Rates Tracking Web App
(Corresponding Video Here)
Those Completion Rates web apps also helped MIRO identify effective survey promotion strategies. After examining the daily survey response rate, MIRO staff found that sending an email titled “Last Chance to Win Airpods Pro and Great Gift Cards” was an effective boost to the survey response rate; there was a 4% increase in responses after the email was sent.
Looking back at the NSSE administration process, three strategies stood out to be effective for a high survey response rate: (1) ensure broad visibility of the survey, (2) demonstrate the usefulness of survey data, and (3) advertise quality survey prizes.
If other offices or institutions can implement some of these strategies, we believe they can also achieve more satisfying survey response rates.
Data Dissemination
Pushing a high survey response rate is just the first step to having a successful NSSE administration. Collecting data is necessary and helpful, but we also need to efficiently disseminate data across the various offices and departments on campus. But how can a small office customize reports for so many offices with a quick turnaround time? MIRO’s solution was to build a series of homegrown NSSE data reporting tools to enable Mānoa’s faculty and staff to pull all kinds of NSSE data efficiently with the autonomy to complete various analyses on their own.
The data tools were designed and developed in-house by the Mānoa Institutional Research Office. Unlike other institutional research offices, MIRO did not use any paid service or software like Tableau because building the data tools on our allowed for more flexibility to accommodate data users’ needs and to customize the reports.
MIRO created two sets of NSSE web apps. NSSE Administration web apps are where users can track the daily completion rates for specific academic units and student groups. This data helps users to adjust their survey promotional strategies and improve the response rates. The second set of NSSE web apps are the Survey Results tools. After NSSE released the survey data late summer of 2020, MIRO was able to quickly import data to the NSSE data tools. MIRO’s web apps were already programmed, which made it easier to turn the raw data into customized reports so that the information could be disseminated to the whole campus within a few days. The quick turnaround time was an extreme benefit that allowed the university to use timely NSSE data for improvements.
MIRO’s NSSE Survey Results tools are organized to help data users search through over a hundred questions to easily find the data they need. MIRO grouped all NSSE questions into 7 web apps: Engagement Indicators, High-impact Practices, Time Usage, Overall Satisfaction, Skill Development, Miscellaneous Questions, and Additional Comments. Extra web apps were also created for special modules: Diversity, Information Literacy, Civic Engagement, and Sustainability.
Figure 17: NSSE Data Web Apps
(Corresponding Video Here)
MIRO’s NSSE web apps provide a variety of filters to customize survey data. The “NSSE Survey Year” filter allows users to isolate a single year’s worth of data–or even multiple years of data–for comparison. Other filters can be used to customize NSSE reports in numerous ways with different combinations. Filters include demographic filters, like Race, Gender, and the geographic region; as well as the academic unit filters, such as College, Department, Major and Program. MIRO also created new filters using data collected from the NSSE survey, such as Educational Aspiration and whether students will choose our institution again.
In addition to the quantitative data, MIRO’s NSSE web apps can also display qualitative data. The “Additional Comments” app was designed to analyze and display open-ended survey data. Users can isolate results from specific questions as well as view comments from a variety of demographic and academic units filters. This qualitative report also includes a table of themes and indicators that contain key areas of interests or topics, followed by a WordCloud image graphic, and a list of related student comments.
The table of themes and indicators was designed based on MIRO’s own organizational chart. This tool can help administrators easily find the information relevant to their offices, such as advising, financial aid, library, etc. Assuming that people want to know students’ feedback on teaching and learning, users can click the indicator of “teaching and learning” and read comments related to that topic. Students are very generous in giving their honest feedback and suggestions. After reading students’ comments, MIRO has been able to better understand their challenges and use their feedback to find new ways to help them succeed.
To help users further understand the NSSE results, MIRO created two resource pages: the “Results Overview” page has reports and presentations prepared by the MIRO office, and the “Reports from NSSE” stores the reports generated by the NSSE institute.
Closing Remarks
Successfully improving NSSE’s response rate and efficiently distributing NSSE results to the various data users is an incredible achievement, but the purpose of collecting and analyzing data is to use data for decision making. For institutional researchers, there’s nothing more rewarding than hearing that data provided was used to help students and to improve programs. After learning from experts and leaders at the University of Hawai‘i at Mānoa, MIRO is thrilled to see that the NSSE data and MIRO’s data tools are widely used in different offices such as the Office of Assessment, Undergraduate Research, Civic Engagement, Accreditation, and more. To this day, MIRO continues to conduct training and information sessions on campus and has created multiple in-person trainings and virtual symposium sessions to share valuable NSSE data with decision makers on campus.