Still Searching for the NSSE? Reflections on the National Survey of Student Engagement

Speaker A: Sa.

Speaker B: Hello and welcome to episode 269 of Silver Lining for Learning. This one is titled Still Searching for the Nessie. Reflections on National Survey of Student Engagement. The National Survey of Student Engagement nsse. So Indiana doesn't just have a great football team and occasionally a good basketball team, but we have the Nessie. You've been looking in the wrong place. We have three co hosts or uh, three guests today who will tell us all about the NSSE and other things. In the post Secondary Research center here at Indiana University. We have Dr. George Koo, who is a Chancellor's Professor Emeritus. I've never met a Chancellor's Professor Emeritus before, but I know George and I know one. Uh, he's in higher education at Indiana University. He founded IU center for Post Secondary Research and the National Survey of Student Engagement and all the related instruments. He's also the founding director of the National Institute for Learning Outcomes Assessment as well as the national UH Strategic National Arts Alumni Project snap. The first ever in depth look at the factors that help or hinder the careers of graduates of art, intensive training, high schools and post secondary institutions. He's done many, many other things including having 400 publications and a lot of books and uh, a lot of talks all around the world. We have also with us today Gillian Kinsey, who's the associate director now for the National Survey of Student Engagement and the center for Research or UH Post Secondary Research at Indiana University School of Ed. She conducts research and leads project activities on the effective use of student engagement data to improve educational quality and serves as a senior scholar within the National Institute for Learning Outcomes Assessment NILOA Project. She's been co author of an author of many books. They're listed in the posting for this week. I won't read through every one of them, but she might want to highlight one or two. And we also have the director of the National Survey of student engagement, Dr. Leonard Taylor. His, his research focuses on investigating and improving how student success commitments are enacted at higher education institutions. He uses various organizational theories and methodologies and works to understand and interrogate how administrators, faculty and staff members and other post secondary stakeholders use research data and promising practices to enhance post secondary outcomes. His work's been funded by the National Science foundation, the Bill and Melinda Gates Foundation, Lumina, and the list goes on and on. Um, um, so I'll stop there and ask George, then Lillian, then Leonard to reintroduce themselves and tell us a little bit more about themselves and what led us to this episode, what they've contributed in Regards to the nsse and then we'll take it from there with some questions. And I think we should start with George, who, uh, just celebrated his 80th birthday here at Indiana. We had a special event for George

Speaker C: and uh, Sarah, did you want to introduce the co hosts?

Speaker B: Oh, I will, I will. We have Dr. Punya Mishra, who, when he's not watching cricket, is creating graphics for this show. And he's did that one for this, this particular show late last night. So thank you, Punya, for that. He's at Arizona State University where everything is now happening. If there's one place that's the hub of, of uh, innovative online teaching and learning, especially with technologies. Arizona State. Check it out. Go visit punya. We have Dr. Young Zhao who hasn't been here for a few weeks. He's rejoined us today for this episode. I know he's excited about this episode. He's got lots of questions. He's at the University of Kansas. We have Lydia Kao from the University of Toronto with us as well. And Christine from Harvard is with us. The five of us will ask. And I'm Kurt Bonk from Indiana and I've been interested in this project for a long time. I'm really happy to get George, uh, Julian and Leonard on this episode. 269. So, George, you want to add a bit to that brief introduction of yourself? I know your, your bio could last us the whole episode. So, um, what would you like?

Speaker D: Why don't we, why don't we start and read that bio aloud and then we can skip to lunch? No, but Kurt and ky colleagues, uh, thanks for the invitation. Be with you. I don't have anything, uh, really at this point to add. I mean, if there are more questions than, uh, we can answer in our allotted time, I think we should just go right to them.

Speaker B: Okay, Gillian.

Speaker E: All right. Well, after George's generous, uh, move to the matter, um, at hand and questions, I feel that some obligation to do the same. I'm really just happy to be talking about this project. I'll just say it is remarkable to me that we are still talking about this topic since it's started in 2000, really 1999 and 2000. So I am happy that this very important idea about student experience has persisted for this long and that we are still at trying to help colleges and universities improve. So that's exciting for me and Leonard.

Speaker F: I would say there's lots of excitement. Um, and I agree with, with George Yalls. Questions are probably more, uh, generative than my ramblings about student uh, success or student engagement. So um, I'll defer.

Speaker D: Well, now that I've tipped off my uh, other two invitees, I do want to add something uh, to what Gillian just said. There are very few, uh, in fact off the top of my head, I can only think of one, um, ongoing national project that was funded by philanthropy, uh, uh, and so to have this thing become self sustaining after the third year and continues so now it's what year 26. Leonard Gillian or something like that, uh, uh, it says, I mean it's a testament I think to the quality of the data that this project generates on an annual basis, plus what Gillian, Leonard and others at the center have been able to do to help institutions. This sounds odd, but to help institutions figure out how to use the data, uh, we used to call it institutional research in a box because essentially NSSE did virtually everything from getting the information, the survey out to students, collecting the data, cleaning the data, turning it back over to institutions, and helping institutions through figure out where they had soft spots in terms of student engagement. So uh, like others we could talk, uh, we could talk for a long time about what that looks like on the ground. But I just want to, I just wanted to emphasize um, how, how unusual uh, this project has become because of its longevity.

Speaker B: So I think given we have three of you that maybe represent the past, present and future, we should maybe talk about the past, present and future. Starting with George. Could you give us some of the history of the NSSE and the project as a whole and the Post Secondary Research center and your founding of it and maybe some of the, a, uh, bit of the scale of this in terms of institutions initially that were involved and how it grew a bit. And then Gillian could add to that scale issue and how many it presently serves and maybe in the present what kinds of items are included in here, what are you looking at and that kind of thing. And maybe Leonard could talk about the future. What's not in there, you know, what's not been looked at, what you might be looking to do in the future. And then we'll turn it over to my co host after we get through the past, present and future. I might have a follow up, quick question about the past, about the data. But let's start with George and go to Gillian and then Leonard.

Speaker D: This project um, began uh, in 1998. Um, we were invited, I was invited specifically to participate in a group that was going to design a tool, some sort of tool that was going to be the counterpoint, if you will. To U.S. news and World Report rankings. Uh, now if you were in the US back in the 1990s, uh, the fall rankings, September was a big month because you found out where your institution fell on the list of high performing institutions and colleges. UH and universities, four year colleges and universities. Um, the problem with the rankings is they tell you nothing about the quality of undergraduate education. The rankings were in those days and still to a large extent all about what institutions had in terms of resources. You know, they all saw a number of books in the library, number of faculty with PhDs and so forth and so on. But we knew from decades of research that that was not a window really into how much students were learning or what institutions could do to improve student learning. So there were some people around the country. I'll mention Russ Edgerton, who's now uh, departed, deceased. But he was at the Pew Charitable Trusts which in the 90s was doing a lot of funding of higher ed projects. Uh, Russ was previously the director of the Fund for the Improvement of Post Secondary Education in the US and if you're familiar with that these days it's really a political animals. But back in those days they were actually doing some very innovative things in terms of improvement of undergraduate education. Well, fast forward 98. A group of people were assigned, came together to work on a design team for some sort of tool. That design team was led by Peter Ewell. There were maybe seven or eight of us on there. And uh, then uh, Indiana was selected to do field tests. So Pew was going to fund us to do two, two cycles of an administration of a survey tool that we had just developed. I should say that survey tool was um, Questionnaire drew about two thirds of its items from an instrument that we had already had at Indiana called the College Student Experiences Questionnaire. A lot of background there. I won't go there. Um, one of the things we were able to do with uh, and so we did two field tests demonstrated the viability of the tool and institutional responses to the uh, the data and what the data showed them. Uh, in 2000 we launched the first national survey. We were hoping that we would get 250 schools to participate. We heavily incentivized schools because they really didn't have to pay much money at all. So we did a lot of stumping around the country to sell the nesse, uh, which got its name. Um, not long before we launched the national survey. We had some bland title to it, prior college student experiences or something. We had 276. So we beat the target and the Next year I'm gonna, I'm not going to remember the exact numbers. We probably had 375. And then the next year maybe close to 400. Over 400. The good news is Pew had given us um, three and a half million dollars to subsidize this tool. And we were more than breaking even. Um, the good folks at Pew said keep all the money, just go and do good work. After 2000, uh, two. I think the peak year might have been 2006 or 2007. Jillian can correct me. I think we had something like 750 or so, uh, different institutions participating in that year. Uh, it's leveled off somewhat. I mean Nessie was a bright new shiny toy in the early 2000s. It got a lot of national press. Um, it didn't, it wasn't as though uh, the Wall Street Journal and the New York Times sought us out. We went to them hat in hand and we had someone who worked with the national media, a guy by the name of Bill Tyson. Uh, we asked him to consult with us and he was so good we put him on our advisory board. Bottom line is, uh, we went hard on a hard sell to institutions. Made it really impossible for them to turn down because there was nothing like this in the country. I'll make one other point and then turn it to Gillian and Leonard. This also uh, was a period of time when the national accreditors were increasingly interested in getting institutions to show the quality of what they were doing in terms of the undergraduate education program. So there was a lot of external pressure on institutions who did not have the data. So NSSE became kind of a go to cool to uh, um, uh, for institutions to show how well they were performing. Um, I should say when someone said this to me in 2003, I took umbrage but, and Jillian and Leonard could take this further back in those days NSSE was a blunt instrument. Uh, it was uh, it was good at showing you where there were probably shortfalls in terms of what was going on with student performance, student engagement particularly. But it was not as not be to be used at a national level as fine tuned as some of the tools that came along later. Uh, like the um, uh, faculty Survey of student engagement, for example. You mentioned the, the family of tools. So there's a beginning college student survey and surprisingly uh, a law school survey of uh, student engagement, which, which probably is now as successful as nesse has become.

Speaker B: Wow, there's a lot there. And before we turn it with Jillian, I would like you to introduce your friend the Nessie. So we get it on camera. Could you.

Speaker D: So, so yes, we found, uh, with the title of this. Now I don't know how this camera's picking this up, but yeah, this is Nessie, the Loch Ness Monster in my hand here. And so if we're looking for Nessie, we have found her right here.

Speaker B: And Indiana keeps searching and keeps finding Jillian. You want to talk about it?

Speaker E: Sure, yeah, I'm happy to provide, uh, be, uh, the Ghost of Christmas present here. The, you know, what George really described as the blunt instrument too, I think is really the point of departure for the present period, or what started as now the present period is, uh, the idea that we did need to really improve some of the measures and what we were inviting students to tell us about. So you know, I would describe kind of the middle period of the NSSE work as one where we improve the tool. We spend a little bit more time thinking about what are reasonable measures that institutions could understand. So it was a little more concerned with how do we shape these, uh, indicators. We ended up calling them engagement indicators to be a little bit better in terms of measurement qualities without going over the top. It's not a test, it's a assessment tool. So we wanted to really keep it somewhere pretty sweet in the measurement where people could still replicate findings, but also that it wasn't going too far down the road of the testing kind of side. So we spent time really improving measures, um, adding to the tools. We added a whole batch of topical modules which were additional item sets that could allow institutions to elect things that they wanted to dig deeper on. So if they wanted to learn more about, uh, the civic engagement or if they wanted to learn more about their students, career and workforce preparation or mental health and well being, we created these modules that they could elect to add. So that gave it a little bit more customization and flavor to the tool, um, and was more responsive to new issues in higher education. So that was important during this middle period. And you know, with that I think it came, we tried to document more about the, how institutions were using their results, what they were doing with it, how they were leveraging it. So spent a little bit more time chronicling some of that and then from that we produced a little bit more, um, nuanced reporting and findings about the quality of high impact practices. And as George knows, the high impact practices as something that took off and is now a trajectory all on its own, but really originated with NSSE in about 2007. Uh, and wow, that whole topic has really just taken off uh, in terms of the importance of involving students in experiential learning. So those are important outcomes of this work. I think the other thing I would add to that is a real emphasis on the value of thinking about these practices. So the things that students do are very important for faculty educators in the classroom and outside in the co curriculum to really think about how do we help design experiences so students are more likely to be interacting with their faculty or interacting with their peers in learning in and outside the classroom and applying their learning in different contexts. So that was sharpened I think in the middle period here. And then the only other thing I'll add to that is I think that we spent a little bit more time thinking about learning and how to identify students gains in what they are learning. So we spent a little bit more time on deep learning and uh, qualifying applied learning. So that's been uh, a consistent measure in our work. So the combination of trying to get a little bit better with our measurement tools, providing uh, better uh, examples of what you can do with these results. George mentioned accreditation. That's number one still what drives people to do this work is how do we represent the quality elements of our educational programs and then also where we need to work a little bit harder and quality improvement. Uh, so those are probably the big things I would point for this middle period.

Speaker D: Thank you.

Speaker B: Interesting. I was going to talk about high impact practices. I'll wait till later on in the show and episode here. Um, Leonard?

Speaker F: Yes. I'll do my best to kind of add some pieces that haven't, you know, haven't been discussed and kind of where we're going. Um, so you know, just kind of build on um, George's early comments around just the significance of the project. Um, you know, so I study higher education from kind um, of an organizational lens. Uh, and I think about it both as a social institution as well as like you know, individual organizations. And one of the things that NSSE has kind uh of achieved in his time was really a global impact. Uh because the lexicon that we were using to describe what was happening in colleges and universities, you know, was, was shifted, you know, based on kind of the nsse, you know, way of thinking and doing and kind of conceptualizing. Um, as you know, I think Jillian mentioned accreditors, you know, policies and priorities shifted at various levels of legislation, um, that were influenced and shaped or informed by what we identified. You know, in the early years of NSSE as you know, kind of effective educational practices. Um, also the data landscape uh, changed. Right. With the advent of nsse. So what, you know, student level data, uh, is still, you know, kind of the collection of student level data is still restricted in a particular way at the federal level. But NSSE became, you know, one of the premier and largest, you know, uh, bodies of student level data that reflects their experiences in college environments. Um, and now we have, you know, 26 years of that, those data, you know, which is a particular kind of archive in and of itself of the college student experience over a quarter century. Um, and then in, you know, I would say probably from you know, the early 2000s, but really 2005 to 2015, the use of NSSE data for higher education research was immense. Uh, and I say that because I was a doctoral student, uh, somewhere in that period. And I remember, you know, how um, uh, many, you know, studies were coming out, you know, whether they were uh, affirming or critiquing, you know, what was, what, you know, kind of was happening in the Nessie, um, the Nessie space. So I just wanted to offer that context, um, building onto Jillian's point as well in the contemporary, you know, so we still serve on average about 400 institutions a year. You know, in a high year it might be 450 institutions, in a low year might be 350 or so, um, institutions. But the post secondary landscape has also shrunk. Um, you know, on average 20 schools close a year and that's been happening for a long time. Um, so the way that we talk about it now is that we roughly, um, provide assessment data in a three year window for about 40% of four year colleges in the U.S. um, which is a significant kind of portion of the post secondary landscape. Um, a couple other things. The offshoots are also fun to talk about. So I think about sase, the South African Survey of Student Engagement, Aussie to Australia. So again even in different uh, international contexts, the ideas that started in this space have reshaped the ways that other folks are thinking about assessment and quality. Um, even in international context. When it comes to kind of like our forward look, um, it really goes back to what uh, you know, what George said and what Jillian said around using the data. Uh, you know, a part of my research and what attracted me to the Nessie role is that like I describe it as studying the social side of data use. You know, as a former administrator, you know, I know all of the bureaucracy that is involved when it comes to assessment data within these environments and the complications that come from that. And so understanding how data ah, are used in an environment. What Data mean, uh, in a particular context or for a particular set of purposes is helping helps us to better support campuses in um, three things which we call kind of like our CUB model if you will. Um, but uh, we work with campuses to curate their data landscapes in alignment with their priorities. Um, and needs to use those data, uh, effectively and innovatively. Um, and particularly the decenter uh, ir, uh, office. Not to remove IR offices but not make IR offices be the only place where those data live and are held. So thinking about how we create smaller reports that can be proliferated throughout the institution so that people who are most close, working most closely with students, can um, access the data that's relevant to their work and um, use that to inform their practice. And then the B is build. Um, so identifying and helping campuses build the requisite capacity to do data work well. And so part of our shift, um, since 2023 when I took over, maybe it's not a shift, but just a reinvestment is in our NSSE Institute work. Um, and all of the work that we do to help campuses have uh, the environments that they need to be able to use our data well. Uh, because the critique uh, that I had of uh, Nessie as an early career kind of scholar was not other people's critique. You know, people wanted to talk about psychometry and all these other things and it was, you know, all of that stuff is separate. But, you know, my challenge was while the data are promising, if campuses aren't using them, do we know that they are useful? And that wasn't a nesse issue. That was a campus issue. Right. There are campuses that we know had really ripe, you know, landscapes for data use, lots of resources in that area. They were able to do really good work. But there are other campuses who perhaps didn't. And so that became a part of our um, priorities in the nesting space is to ensure not just that we provide high quality assessment, but that we also support campuses in being able to use those data effectively.

Speaker B: Right. All interesting. And I think there's a hesi too High School Survey of student Engagement. Because when my son was a student here, he worked in the center for Jonathan Plucker doing some of the data input.

Speaker A: I think.

Speaker D: Um, that's right. That's right. In fact, I was had a command performance to talk with a small group of superintendents from Central Indiana. Uh, they wanted to know about Nancy. And within 15 minutes they said why don't we have a high school survey of student engagement? It uh, has not grown as fast. Part of that was due to no Child Left behind because the emphasis on the quality of education went to something, uh, outcomes based. So we had to teach to that a particular test in an institution. I just want to make one other comment about early days and the national advisory board that Nesse put together. Actually Russ Edgerton and Peter Ewell and some others. We had some high profile luminaries on that board. Derek Bach, some other people, but they kind of gave us a framework from which, from which to work. And one of the things they insisted on, and this goes to Leonard and Jillian's point about data use, actually I should say data awareness. We had to send, we, we sent uh, every, uh, every institution's own report to three different offices. Uh, the I.R. office of course, or whatever that was called at the time, the media Relations office, and directly to the president. And what happened initially was president, somebody in the president got this thing and said what the hell is this? What's going on? And they called the IR office. Now the office was never called by the president's office for anything substantive with regard to undergraduate education. So Messi began to uh, churn inside institution in, in ways that data about the student experience never did. And the other thing I think was alluded to by both Julian and Nessie. Nessie introduced the phrase student engagement. And Kurt, I think in your own overview of what Nessie did was you mentioned this. I mean if you go back into the 1990s, you won't that phrase anywhere. Now there are surrogate phrases for it of course, involvement and whatnot. But that student engagement piece has stuck for a long, long time and it probably won't. Well, I don't know. Uh, and we're glad it did because it focuses on something that's important to the quality of learning, but also what institutions can do about improving the quality of learning.

Speaker B: And you're right. In the past 25 years there's been more emphasis on student engagement. I know my own students 20 years ago gave up studying the topic because they didn't think it could be measured. Now they are, um, seeing ways to measure engagement. I know Chris has a question. Chris, you want to jump in here?

Speaker A: Sure. So I've long felt that motivation was underappreciated in terms of learning. Uh, I taught for many years a course on motivation and learning at the Harvard Graduate School of Education. And we had maybe 48 courses on learning and my one course on motivation. And then when I stepped away from teaching there were zero courses on motivation. So I'm, I'm glad that the Survey has uh, achieved the stature that it has because so much of the energy goes into building learning experiences that fail because they're boring. Um, that said, um, there's, there's sort of self reported engagement and then there's behavioral engagement, which I think is a true measure. And I think what's scaring a lot of people now is the behavioral engagement where students are simply not going to class, they're having AI do their homework, they're, you know, um, uh, reporting that they see no relevance of much of what they're learning in higher education to their future life and career and so on and so on. It's, it's um, and, and, and you know, then there's the debate about well, is this a problem with the students or is this a problem with higher education? To which I would say there's plenty of blame to go around here. But I'm curious as to um, whether you've done any studies that attempt to gauge behavioral um, engagement from self, um, reported engagement or is that just a non. Issue?

Speaker F: Well, I might, I'm, I might start off just, just to say so, just some, um. For us, you know, all of our data is self reported, um, and so even so within the behavioral elements there are students, you know, reports of their behaviors.

Speaker A: Right.

Speaker F: So we don't have any like direct observations of student behaviors per se. Um, however, you know, one of some of the things that we're thinking about, um, and I'm going to try to think a little bit more about specific studies, but some of the things that we're thinking about right now is the utility of NSSE to help institutions understand the dissonance between what they think students are doing, um, or what they might be directly observing and then what students are saying about their experiences. Because what we find as part of, you know, sometimes like you said, there are plenty of blame to go around, but there's also plenty of confusion, uh, and assumptions to go around. And you know, so like a specific example is like we have campuses who, you know, they'll administer a fesi, the Faculty Survey of Student Engagement and uh, nsse, the Student Focused Survey. And they'll see, you know, around similar items, very different, um, you know, kind of measures or different, different, you know, outcomes or not outcomes, but different measures. And then it raises questions around like what are faculty's perceptions of what's happening versus what students perceptions are of what's happening and how do we kind of reconcile that? So anyway, I'll stop here but just to say that um, I think actually I won't stop here. There's one more thing I'll add. Um, one, I think NESSE is a prime tool to help campuses do some of that sense making UM around like what students are experiencing versus what they say that they're um, experiencing. And then two, we've been thinking about this a lot with regard to AI, so we just designed uh, and launched the AI module. Um, and I'm going to say it here because then we are kind of committed to it. But you know, I think that that AI module is going to be the start of a broader um, attention to how NESSE can better capture students relationship to technology. Because one of the kind of uh, um challenges I think in the current kind of landscape, um, globally is that technology has reshaped students lives faster than people can conceptualize. And so we don't know um, how students are making sense of the environments, the technology, the virtual environments, the you know, kind of material environments. Um, and I think that that will not answer a question but I think that it'll help us to ask better questions uh, about how we improve or align um, what students expect from college, what colleges intend for students and what society needs from both.

Speaker E: I just want to add one more thing to what Leonard just said. So. Well, the idea that the concept of student engagement implicates both students and institutions in a very similar way. So it doesn't let anybody off the hook here for what creates an engaging experience. And, and for me that's a very critical element for institutions to have to wrestle with. And that's where the data can really be illuminating or perhaps shed some light on challenges in the learning environment. So you know, Leonard's example of where the Fesse data, the data that come directly from faculty about the same concepts can really be you know, kind of mind blowing, disrupting for people on a campus. It can also be inherent even in student responses. So when students talk about the fact that they're not having many interactions with their faculty at a small residential college that claims that this is the place where you're guaranteed that kind of experience, it can be really disruptive at the institution. And you know, uh, I mean I've heard institutions that get data that doesn't square with what they think about themselves and they say well we should stop doing that, we should stop doing NESSE because it doesn't suit us. And I think what in the world, this doesn't make any sense. But the ones that will really sit down and say wow, what's going on here? We've got two Years of data that say pretty much the same thing, and it doesn't square with our impressions of ourselves. What are we going to do about it? To me, that's where the real richness comes in.

Speaker D: Uh, the motivation question, Chris, is a hard one for me to respond to. And as we've all agreed, blame or responsibility goes in all directions. But you mentioned HESI earlier and if you look at the hess, I, uh, haven't looked at HESI data in a long time. But HESI data, as I recall it showed a sharp decline from the first year of high school to the last year of high school in terms of student effort, time on task, number of hours spent studying. So what we have seen is we're inheriting uh, cohorts of high school graduates starting college who have not developed the habits of the mind and heart to do college level work. There are other indicators here too, but I just want to say I think you can backward change as Gillian and Leonard have been explaining, backward chain from student self report to behavior. When you see what faculty do in terms of ramping up, uh, the uh, academic challenge of math courses. We have examples of institutions where students are, uh, where institutions have not changed what's offered in math classes. And with more and more students in those particular institutions coming better prepared in math, they say, uh, if you break this down by major, the math majors are not being challenged. Well, when faculty see those data, and we have instances where faculty have in fact changed the nature of the curriculum, you can see the nature of students self reports over time changing, saying there's much more. Now the academic challenge. I'm just using math ah as as an example. Last thing here is we make talk about hipps now or later. Um, but I uh, think the introduction of certain kinds of student experiences that are more likely to motivate and are uh, aligned with what policymakers and employers are expecting out of college graduates, hips have been, have been a kind of a window, an avenue into what institutions can better prepare students for post college life. So Jillian used the phrase experiential learning in her introduction. Um, I think we've given not, not nessie alone, but nessie has been one of the tools that has raised the level of experiential learning. So it's not something that you can no longer say in a college or university, which wasn't the case back in 2000. If you said experiential learning, you know, my eyes glaze over. That means count off by fours and go off and talk among yourselves and come back and tell Us what you talked about. So I think uh, I think the uh, the uh, not emergence but the documentation of, in the quality of learning associated with internships and study abroad and doing research with faculty and being involved in a learning community. NISSI was a forerunner in terms of pointing out the high value of those kinds of experiences.

Speaker B: Thank you for that. Um, I think we're going to jump to Young, who I know probably has given all the books he's written in these various topics related to this show. He's probably got dozens of questions. Let's start with the first one. Young.

Speaker C: Well, thank you. Uh, thanks George, Jillian and uh, Leonard. Uh, this is a great opportunity to chat about big things. You guys have done amazing work here. Um, I had really actually two questions. The first one relates to what Punia wants to ask. First is over the last 26 years, what kind of patterns have you seen? And related to that I just want to add, you know, because I think Leonard, you were talking and genie a lot about technology especially Leonard was talking about. So over the 26 years, um, I came to the US 1992. Just imagine, just shortly before, you know, you guys started the whole thing. If you look at the large political, environmental, economical and technological changes, it has been amazing and related to that pattern, you know, I'm sure you've seen, I mean you draw a quarter of the institutions. So in Nessie, do you see major uh, changes over the time? You know, and that's one question. The second question, uh, you guys don't have to answer all of them because I'm just very curious about this. You. With any measure, any measure in education is temporary. It's intermediary. You measure this moment, whatever you measure, right? And then people um, over a certain time they want to verify its validity. Does it really predict anything? Remember, that's what we've been seeing, we've been doing with sat, SAT creativity measures and some uh, uh, never do that. Um, you also saw problem that Gallup is making a lot of money, uh, producing this kind of uh, uh, uh, uh, service uh, in high school as well. Uh, it's meaningful. But I was just wondering uh, if any of you guys have picked any sample, you know, over 6, 26 years, you can probably go check to say, did it predict anything? Did you know, did the assessment predict anything? You know, like yes, I was engaged. I should be better than those who were not engaged. Uh, well, I don't know. But uh, you can have a different theories of change. So I was just wondering if you guys have done any of those work and what the outcome was in terms of verifying the validity and how it was. So pick any one of those questions. This is great. Thank you very much.

Speaker E: Wow, that's a rich, um, multifaceted question. So thank you for asking it and I appreciate your curiosity and thank you for your interest in this work. I'm going to talk on one topic that I feel okay touching on and I know Leonard and George will have something more to say. But the piece that the last question you kind of centered on is what do we have that predicts something else that is valued in higher education? And there is a long history before NSSE about the value of involvement and all the things that we're asking students about that prediction, students completion and their success in college and their learning. So you know, we're, we're really just lifting up measures that have been proven to be associated with important post secondary outcomes. So we are building on the existing research. So that's one important point. The second thing is that, you know, we have done these studies to the degree that we can with the same kinds of outcome measures. So when we can get actual retention data and when we can get actual graduation, college graduation data, we have done those studies and demonstrated, they've demonstrated that, yeah, students who are more engaged in these things are more likely to persist, graduate, and then a whole host of other things, um, have a satisfying learning experience, be more satisfied when they graduate. So all of those things we've done to the degree that we can because we don't have some of those outcomes data unless we really go out and ask for it and get it. So there's those dimensions. The other thing is other people have done that. So Leonard mentioned the large body of research from other scholars. Other scholars have done this. So other scholars, other projects. This huge study that uh, happened with uh, at the time was a group from Iowa and a group from actually Wabash College that did a lot more very independent of us, which helped demonstrate and further illustrate the value of engagement to a whole host of outcomes. Lifelong learning using other tools and measures, outcome measures. So there's been a, uh, large body of work on this topic. But for me the most important element is when institutions are doing this kind of analysis with their own data. So when they're applying, they're adding their engagement measures to their own institutional research. So if they want to better understand, well, we designed some interesting learning communities this year. Let's see if that made a difference in our engagement. So they're doing their own very bespoke you know, crafted thoughtful uh, analysis with their own data. To me that's the other level that I think we're trying to facilitate and help institutions do is answer those questions about their own students.

Speaker F: Yeah, I'll jump into some of the other, other questions but I might circle back around to the prediction pieces as well. Um, the piece around technology I think is really interesting because as we think about who there's who NSSE serves which is institutions, there's who NSSE surveys which is students and then there's who NSSE touches which is like the people at institutions. And so often we're having to think in uh, several different registers with regard to how our technology or our socio technical kind of um, uh interactions uh, need to make sense to folks. Right. We need to have something that appeals to you know, students across a range of identities and ages. Right. Because we know the post secondary landscape is shifting, you know, toward adult learners in a particular way. We also have to make sure that what we provide is useful for a range of staff and administrators with varying levels of uh, experience, knowledge, sensibilities related to data, um, and technology. And I say varying in the true sense of varying. It's not like good or bad necessarily, it's just different. Um, and then we also have to think about like what institutions as a whole need to be able to represent uh, as they use these data to do case making for performance based funding, uh, to demonstrate to their accreditors, whether disciplinary based or you know, kind of national accreditors that they are meeting a particular quality or to advertise to students to say like here's what the student experience is and here are the data that show that. Um, and so I think one of the things that's changed um, you know, in your observations, observation young um, over the last few decades is really how um, while the kind of socio technical landscape is just like really kind of complicated in all of these different ways, um, we've not lost um, any sensibilities, we've just added some. Right. There are still um, our very analog folks who want to do things in very analog ways and then we also have folks who want to do everything on AI. So there's a lot of noise in the landscape. Um, and I think one of the things that we've been sitting with recently is how do we position ourselves unapologetically um, within that landscape. Um, I'll use one specific example and I don't want to. This is not a critique but it's just to say that like you know, hired uh, Birnbaum talked about this several times. I think he has a book like, like it might be up here somewhere, 1990, about higher education management fads. You know, higher uh, education institutions are particularly susceptible to you know, that type of memetic isomorphism, right? Like the ways that you know, they see one campus doing something that they aspire to be and now they're going to do it whether or not it works for your context. Um, and we've seen a lot of this happen with the application of predictive analytics, um, and third party companies who have been, um, you know, kind of partnering with colleges to do that type of predictive analytic work that was actually part of some of my early research. One of the things that I found in that uh, in those early studies is that campuses will adopt these technologies from the top, right? Like the institution will say yes, we're going to do this because it serves our purpose, but the people in those institutions have no appetite or interest in participating. Right? So like if you have predictive analytics software and then, and advisors need to use it to inform their practices and they say no, it creates some consternation and frustration within the institution. But you also can't force them to use the platform. They can even say that they're using it and not actually use it. And so I say that to say we're seeing campuses who uh, have leaned toward or leaned into these more kind of highfalutin, uh, uh, uh, and technical solutions to social problems that are starting to come back and say actually we just need to ask students directly what they're experiencing. Experiencing. You know, uh, maybe prediction isn't our goal here is relationship. Right. Uh, you know, and, and, and so that, that's part of what we're seeing that I think is um, you know, important at least from my, my vantage. And the other thing that I would say about that, there are a couple things that we're thinking about. Um, there's so many things to say because you asked really good questions. Uh, one, um, and I'm trying not to get on a soapbox either because I'm, George and Jillian probably both noticed about me where I'm like, oh, like I am activated now. I want to talk about all of the, you know, all of my thoughts and opinions around, you know, technology, blah, blah, blah, blah. But an important piece around things being temporary as far as outcomes, um, is that they're also contextual, right? Like the things that could, the experiences that contribute to outcomes happen in context. Um, and we get at this in our, in our survey by asking questions around students reflections on quality. Right. Like the quality of advising, the quality of high impact practices. Right. Because students on the same campus could both study abroad, but you know, if they have varying quality experiences, the impact of that experience is not necessarily going to, you know, translate for both of those, those students. So we've been thinking a lot about context and how to capture or identify what context, uh, practices are happening in. Uh, and also context as time. So one of the things that Jillian is leading us on right now is a panel, ah, study where we start to have follow up conversations, uh, or follow up surveys with students, um, you know, six months, 12 months, 12 months, 18 months after they graduate, um, related to their experiences with career and workforce preparation to see how they're talking about those things. Uh, because this is a hill that I will die on. Hopefully not. But you know, we'll see. Um, colleges and universities, best data to demonstrate their value, uh, is, should be captured after students leave. Um, you know, and not to say that what we capture is not useful. It is very useful to demonstrate kind of the immediate experiences. But you know, many of us know that like the story that we tell about our undergraduate experience, for example, is going to be different six months after we graduate and then, you know, or six years after we graduate. And I think colleges and universities stand to also have some of that rich data, um, to tell the ways that their experiences have contributed to the lifespan of students is that we're starting to tinker a little bit with how we can do more longitudinal work, um, and hopefully then be able to do maybe some more predictive work around um, how students, experiences in college are affecting them, um, after college. I'll stop there.

Speaker C: Well, thank you. Let me just add one little. This is amazing. Okay, thank you guys. I should come to uh, Bloomington sometime. But anyway, uh, I used your data, uh, many, many years ago. But uh, in 2019 I was writing a book. But actually one thing when Leonard, you were saying, you know, like, uh, a lot of businesses published schools, graduates who make how much money? You know, I don't think that really matters. I think you guys might want to add a new index called Happiness, you know, how happy the kids are after a year of graduation, finding a job. You know, I'm serious. It's just something interesting. But I know Lydia has a, uh, more important question than my add in the happiness index. Look, Punya is happy listening to his daughter. But anyway, that's good.

Speaker F: And I'll just say really quickly in response to that. The one thing that we did pilot this year is value. Um, so students perceptions of the value of their undergraduate experience, which kind of gets at what you're saying, like, yes, you can make a lot of money, but did you value your college experience?

Speaker B: Before we turn over to Lydia, I know Punya asked the question, and he had to run, and he's back. Um, we have seven, eight minutes left. When you have a minute or two, you want to talk about your. The comment on motivation.

Speaker C: No, it wasn't about. Thank you. Uh, and apologies for hopping in and out. Kids, you know, they grow up, but then they're still your kids. It's beautiful.

Speaker D: They never. They never leave. They never.

Speaker A: No, no.

Speaker C: I love the fact she's traveling and

Speaker F: she had some questions that needed to

Speaker C: be answered now, you know, um, but that's good.

Speaker B: That's good.

Speaker C: Um, I just wanted to understand, like, the broader trends that you're seeing, because we hear a lot about sort of, uh, I mean, particularly post pandemic, that we hear, uh, about this drop in student engagement and, you know, sense of loneliness and so on. Is the data reflecting anything of that? Briefly, Because I don't want to eat into Lydia's time. If any of you can, uh, jump in.

Speaker B: Yep, just a minute or two, and then we'll go to Lydia.

Speaker F: I mean, I'll say really quickly, and then maybe hand off to Jillian or Georgia if y' all want to add to it. Relationships matter. I think that's one of the things that has been enduring, and it's to say that while we might see trends in the decrease of kind of like the interpersonal interactions that folks are having or the diversity of interactions that folks are having, that the impact of those things is consistent.

Speaker E: Yeah, Leonard's absolutely right about the relationship element, and it's been, I think, tested and challenged post pandemic, in particular, is reestablishing that. But the importance of relationships is really still an important matter. The, uh, other issue I would add to that is that our data showed that while some collaborative learning and experiential learning of all sorts were diminished or depressed during the pandemic, most of that has rebounded since. Um, maybe not to the same level that we saw before, but I think that's been the case mostly. But I think we're going to have years of pandemic challenges to deal with with college students today.

Speaker B: Sure.

Speaker C: Thank you. Thank you. Lydia, Go ahead.

Speaker A: Thank you.

Speaker E: I think there are definitely overlapping our

Speaker D: questions in some ways.

Speaker F: Uh, it's really remarkable that Nasty's history

Speaker E: spans, um, I feel the unique stretch

Speaker F: of change from like early Internet to social media now into the age of AI.

Speaker E: It's wonderful to hear there's something that's

Speaker F: timeless, like the value of relationships. And it's also really great to hear

Speaker D: that you're also adapting your instruments.

Speaker E: As the context evolve, you're in integrating

Speaker F: like value into the survey.

Speaker E: So I'm just wondering, you know, as the kind of broader context reshape itself, um, how does it reshape what student

Speaker F: engagement look like in higher education? Absolutely. This is a great, a great question, Lydia. Um, um, so we're actually, that's one of the things that we're thinking about right like right now in this moment. So, um, in the history, um, Nessie has always been evolving. I don't know that we've done a good job of telling our story, but we did a major kind of overhaul that was, you know, kind of rolled out in 2013. But between 2009 and 2013 there was some tinkering with the content of the survey. The psychometry, you know, that kind of nuts and, and bolts. We're current, um, we call it Nesse 2.0. Um, we're working right now on Nesse 3.0. And that is a kind of revision not just of the content, but our strategy for measurement, our strategy for administration, and our um, strategy for supporting the use of those data. And so one uh, of the things that I think we also had to really sit with is what does student engagement mean in 2025 and beyond, given that the post secondary landscape and students interface with it is so different. Um, and so we have a kind of working framework, um, that will be kind of a new iteration of what student engagement, um, is. And I'll say without getting into all of the weeds, um, the three kind of areas that are most important are relationships and interactions. So how students are having relationships and interactions with other people, whether those people are staff, advisors, uh, faculty or students, as well as what their relationship is to the institution, uh, there's aspirations, uh, and achievement. So like what it is that students are seeking to get out of their college experience and how do they conceptualize those goals and then how do they move, uh, uh, you know, toward them and then context and environment. You know, how do, how do we understand at different uh, levels, you know, what their relate, what their, what their interactions are with their environments, uh, what their perceptions are of their environments, positive or negative. So that's kind of where we're thinking right now, around kind of some evergreen concepts that might be useful, uh, regardless of certain Changes, Um, and then as you highlighted and as I mentioned before, trying to be more specific around, uh, being adaptable and agile, when we realize that there are shifts that are happening in the post secondary landscape, like with AI, you know, value was really important to us because we recognize that, like, there's a lot of conversations around earnings tests and you know, the new Carnegie classifications, you know, uh, have some, you know, representation of value, you know, in a particular way. And we find that students, perspectives on value are rarely a part of those conversations. And so that's another area that we're trying to push up. So not a direct answer, uh, to your question, but a way that we're trying to be kind of. Of have an agile stance, um, as we think about those changes. But again, Jillian and George, you know, will have some. Have some thoughts here too, I'm sure.

Speaker D: Well, I got invited into that conversation you're having now, but I worry a, um, little bit about. Well, I worry a lot about asking students, for example, uh, how valuable their education has been when they lack a context in terms of what education will mean to them two, five, ten years from now. I also worry a little bit about whether you mentioned Evergreen. It's a very. I mean, I'm, I'm enamored with that concept. Are there not some things that should not. Here's a double negative. There may be some things about student engagement that should not change, uh, irrespective of where technology goes, irrespective of what AI means. I don't know what those things are. You mentioned achievement, but you didn't mention academic challenge in that. In fact, what I heard you say was it's kind of what student, students want out of their education. And of course, higher ed has bent over backwards in the last 25 years in terms of giving students what they want, whether it's physical attributes of the campus, whether it's, uh, grade inflation, whether it's the nature of what requirements are to graduate. So I worry, uh, I worry a lot about the dampening of the quality of education, um, in the context of. We want to be sure students are satisfied. We used, you know, earlier we were talking about a happiness indicator. I don't think happiness is our goal. Uh, it might be to some extent satisfaction, but there ought to be some, some layering of what colleges are supposed to do in terms of increasing the quality of student learning and personal development. So that's the, that's the Luddite comment.

Speaker F: Um, yeah, no, I, But I think that, I mean, you bring up. I think you Bring up a really important tension. Right. Because I think part of this is also. And, Jillian, I love your thoughts here too. Part of this is, you know, what is our responsibility to the field of higher education writ large? What is our responsibility to institutions specifically? And one of the things that, like I'm on sometimes on skew on the opposite end of the spec. Maybe the opposite end of the spectrum. I think colleges have bent over backward for some students. Some, um, and not all. Um, and I think that the students at colleges have not bent over backwards. Four are the ones whose perceptions and experiences are perhaps least attended to, um, or solicited.

Speaker A: Right.

Speaker F: Uh, and so that accommodation piece, I think, is a challenge and a problem. But I think there's also a group of folks that stand to have their opinions heard in a particular way. The other thing, that's just a hard reality of a lot of institutions right now in this place, especially those that are enrollment driven. Um, if students don't want to come, then the school will go away. And so there's this tightrope of providing an experience that's satisfactory, but also providing an experience, to your point, George, that has enduring value beyond perhaps their developmental scope. Um, and I think that's. We're trying to find how. Figure out how to curate data that gives campuses the ability to decide for themselves, you know, where they fall in that spectrum of priority. Invite you to those conversations, Leonard.

Speaker C: Sounds like we did continue for another hour. Uh, because now I began to see the challenges. Now is, has the institutions bent over or has students bent over? That's actually. Well, Kurt, that's around for another hour. This would be fun, you know, but. But anyway, thank you. This is very interesting discussion.

Speaker B: Young, you and I can sit down and think of what could be part two or episode two. And I'm sure George would love to come back and Leonard and Jillian, uh, we'll talk. Um, Jillian, you get the last word on this episode. Uh, before I introduce the next episode.

Speaker E: I get the last word.

Speaker B: Yep.

Speaker E: Oh, my gosh. Okay. I love this idea of the enduring concept of engagement. So, you know, I would leave everybody with thinking about that. What are the things that are so powerful, we must attend to them and ensure they endure. This idea of agency and ethic, of care and challenge and quality relationships, to me, are the most important thing for us to keep top of mind as we imagine the future of higher education.

Speaker B: Thank you for that. And thank you, Leonard. Um, thank you, George. Thank you, Gillian, for everything. This hour has progressed into many different, um, interesting channels, and we could have gone to each one of those for an hour. Um, there's so much data that you have, so many interesting ideas and next steps. Um, so thank you for sharing what's happened in the past, what's going on in the present, and what's possibly going to happen in the future, uh, in episode 269. But I need to introduce. Yeah, thank you. Thank you, George. Uh, thank you, Leonard. Um, I need to introduce the next show briefly, and we're going to take it up to Michigan and, uh, speak to folks who were at Georgia Tech previously. Mark Guzdiel, a friend of mine since I was in grad school days, and Barbara Erickson, both from the University of Michigan, stopped down in Bloomington and did a presentation about computer programming and the pedagogical innovations that are happening at the University of Michigan and how they're teaching programming courses to folks in the humanities and social science areas. And I thought, what a wonderful topic for silver linings. So they'll be here next week at 10 o' clock Saturday morning, an early show. Um, see you then,

Speaker D: Sam.