The Real Data Revolution

By Brandon Busteed    //    Volume 24,  Number 4   //    July/August 2016
AGB Trusteeship Magazine: July/August 2016, with cover article "The Real Data Revolution"

There’s a coming data revolution in higher education, but it’s not the “big data” revolution that many have been hyping. This revolution will be more about the right data than bigger data. And it’s not data on traditional education metrics, but rather data that have been largely—indeed, embarrassingly—missing from higher education. This revolution will be about the voices of consumers and constituents in higher education. It will be defined by the addition of behavioral economic measures, not just classic economic measures. And it will usher in a new era of rigorously tracking the expectations, experiences, emotions, and outcomes of students, alumni, staff, and faculty in the spirit of understanding how institutions of higher education are performing, and how they can improve. Boards, take heed.

As with any industry or sector—from business to government—understanding the human side of the ledger is critical to organizational performance. The same is true for higher education. It’s almost impossible to imagine the world today without the voice of the consumer. We decide on restaurant selections by using tools like Yelp to see ratings from customers. Every time we take an Uber, we provide ratings on the drivers. Nearly every experience we have today measures our interaction with a product and our feelings about it. Imagine if we actually had this kind of information to inform how we measure and improve quality across higher education. With rare exceptions—such as business school rankings, where alumni surveys are a component of the criteria—higher education is not systematically measuring the experience of students or alumni. Current college rankings are not reported in the vein of Consumer Reports, which includes reviews from consumers who own and have tested products and services; the current college rankings include no such thing.

College rankings certainly drive plenty of consumer attention to higher education, as parents and prospective students flock to websites and magazines in order to judge which colleges and universities are “best.” In reality, we have no idea which are best. If we judge “best” by factors like admissions selectivity and endowment size, we know which institutions score well. But if we were to attempt to judge colleges and universities by the learning growth and development of students from matriculation to graduation, we’d have no idea. No one has ever measured this. Or what if we were to judge institutions by the career and life outcomes of alumni? Remarkably, that hasn’t been done, either. In 2014, Gallup and Purdue University partnered to conduct a large-scale, nationally representative study of college graduates and their long-term outcomes. The Gallup-Purdue Index was the first effort of its kind in the several-hundred- year history of higher education. The reality is that the voice of the consumer in higher education doesn’t factor into how we come to judge or understand or improve it in any way. And that’s a huge missed opportunity.

The End of College Rankings as We Know Them

Although this coming data revolution has implications for rankings, it will be defined much more by a spirit of continuous improvement, of identifying actionable data aimed at bettering the outcomes that matter most. A February 4, 2016, article in The Washington Post by Jeffrey Selingo titled “The End of College Rankings as We Know Them” articulates the seemingly subtle yet titanic shift from measuring inputs versus outcomes in higher education. In the article, Selingo likened the current ranking system to a Consumer Reports that “judged products based on the quality of their raw ingredients rather than the final product.” Although it’s fair to say that current college and university rankings have led leaders to change, the real question is whether the data and the changes based on them have actually moved the needle on things like quality, affordability, or outcomes. Decisions based on current rankings have led, in many cases, to the growing arms race in higher education (fancy buildings and student services, ballooning endowments, concern for marketing and brand reputation, increasing selectivity), and even to unethical practices like falsifying admissions data.

The uncomfortable truth is that higher education desperately needs to end its obsession with rankings and focus more on simply improving—in all the right ways. The coming data revolution will shift colleges’ and universities’ focus from how they fare in rankings to a serious look at whether their own institution is improving year-over- year on the metrics that matter most to their own constituents and consumers.

Behavioral Economic Measures

Most of us like to believe that human beings are very rational creatures, that we make decisions based on data and information, and that forces in the world behave in predictable, rational ways, too. Indeed, classic economics is based on this notion of rational decision making. And although these classic economic assumptions hold true to a certain degree, they certainly don’t explain all that takes place in financial markets or a particular individual’s decision-making process and behavior. The generally accepted 70/30 ratio tells us that as much as 70 percent of the decisions we make as human beings are based on emotions, while only about 30 percent are based on rational information. This is the fundamental distinction between behavioral and classic economic measures. Make no mistake—both matter. This is not an either/or conversation; it’s a both/and. For that reason, and in order to have a fuller picture of how higher education is doing and how it can improve, we need to regularly and rigorously track both classic and behavioral economic measures.

In the current higher education environment, institutions track plenty of classic economic measures, such as grades, test scores, and graduation rates. But institutions writ large are barely paying any attention to behavioral economic measures like student or graduate well-being, hope for the future, or engagement. This is the fundamental data revolution ahead. Big data and analytics utilizing classic economic measures are helping many institutions improve admissions yields, retention, and graduation rates. This is indisputably valuable and will continue to be so, but concerning behavioral economic measures, higher education is decades behind other organizations and industries.

Classic Economic Measures

Nations have typically measured their health through classic economic indicators such as gross domestic product (GDP), unemployment rates, or even crime rates. And as important as those measures are, they are also limiting. Consider Egypt and Tunisia in the five years leading up to the 2011 Arab Spring, when GDP in both countries was rising steadily. But data from Gallup’s World Poll, which samples 98 percent of the world’s population over the age of 15 each year to report on well-being—that is, how people rate and experience their lives—told a very different story. Those measures, and thus well-being itself, plummeted in both countries prior to the Arab Spring. In other words, well-being was going in the opposite direction of GDP, and it turned out to be a much stronger predictor of unrest.

 

Similarly, consider unemployment. While it is critical to quantify the number of people working or who are looking for work, this measure alone is quite limited. It’s one thing to have a job, but there’s a substantial difference between one that’s just a job (the only one a person can find, or they aren’t suited for, or that they don’t excel at) and one where a person is fully engaged (emotionally connected to their work and workplace, where they do what they are best at each day and are supported by a great manager in their development). There’s no comparison between these two scenarios. Decades of Gallup research have proven that employees who are engaged in their work are more productive, less likely to be sick or absent, have lower turnover rates and fewer safety incidents, and drive more revenue and profit for their organizations.

Today, nearly every K-12 student, teacher, and school is measured by three classic economic measures: grades, test scores, and graduation rates—the same core indicators used to measure higher education, too. And, once again, these measures are critical for understanding certain aspects of performance. But in and of themselves, they are grossly insufficient. It’s great to know a student’s grades, for example. But is that student truly engaged in what he is learning and excited about his future? Does she feel like her teachers care about her as a person, or that she has a mentor who encourages her goals? Does he feel he has a chance to do what he is best at in school, or does he even know what he is best at?

Not only are we missing the important voices and emotions of people involved in the process of learning, but classic economic measures in education have come increasingly under fire. The United States achieved its highest high school graduation rate in history this past year, yet the public dialogue about schools in our country is as negative as it’s ever been. The average college GPA has gone up a full letter grade in the past 30 years, thanks to rampant grade inflation across the industry. It’s hard for employers to understand what a good grade is anymore because everyone has a good grade. Google, for instance, no longer even asks candidates for their grades and test scores because they have found no correlation between those measures and success on the job. Dr. Shane Lopez, the world’s foremost expert on hope—which he describes as measuring one’s ideas and energy for the future— has published numerous studies showing that hope is a stronger predictor of college completion than SAT scores, ACT scores, or high school GPA. We have held these measures dear for centuries and only now are beginning to understand that there is so much more to success in life than these things alone. This is not to suggest that we abandon metrics like grades, tests, and graduation rates; they can and will serve an important purpose. But we have an immediate imperative to expand our scope beyond these limited metrics.

The Ultimate Outcomes of Higher Education

A diploma or degree is an important outcome of a college education. And there is powerful classic economic evidence that those with college degrees earn considerably more over their lifetimes than those without degrees. That’s a very important outcome of an education, to be sure. But many in higher education will rush to point out that it’s not the only outcome. According to our mission statements, the reach goes way beyond that. It’s about getting a job, yes. But more importantly, it’s about having a fulfilling career. And not just a fulfilling career, but a great life, too, in which a person thrives in their overall well-being and participates in society as a valued and engaged citizen. So how do we go about measuring this?

The Gallup-Purdue Index is a massive, ongoing, representative study of U.S. college graduates measuring their long-term outcomes in work and life. It measures classic economic elements like household and personal income and employment rates. But it also measures behavioral economic elements, like workplace engagement and overall well-being, including asking about key educational experiences had during college. The findings have provided a breakthrough set of insights. With more than 60,000 graduates surveyed to date, the Gallup-Purdue Index reports not only on outcomes at an aggregate national level but also provides benchmarks and comparisons across nearly any type of institutional cohort—across Carnegie Classifications, athletic divisions, by state, etc. A number of pioneering institutions have now done a census of their own alumni to measure their outcomes and understand how they compare with others.

Five Essential Elements of Well-Being: What Does a “Great Life” Look Like?

Purpose
How you occupy your time; liking what you do each day

Social
Relationships and love in your life

Financial
Managing your economic life to reduce stress and increase security

Community
Engagement and involvement in the area where you live

Physical Well-Being
Good health and enough energy to get things done daily

Five Elements of Well-Being

Gallup has been studying and measuring well-being for nearly 80 years. It began with early studies led by George Gallup that tried to “quantify a life well-lived,” as he described it. And it is now expanded, both to a daily poll of 500 U.S. adults and the annual Gallup World Poll. What Gallup has learned is that there are five essential elements of well-being that are consistent across all ages, races, religions, etc. They are described as purpose, social, financial, community, and physical well-being. These are important measures because they are powerful predictors of a number of key outcomes. Employees who are thriving in all five elements of well-being, for example, represent one-third of the healthcare cost burden to their organizations compared with employees who are not thriving in any one element. In short, a metric for well-being is not a nice-to-have, it’s a need-to-have.

In addition to well-being, Gallup has been studying workplace engagement for decades, too. Being engaged at work, to remind, is a measure of your emotional connection with your work and workplace, and it is linked to all sorts of key performance indicators. Engaged employees bring all the new ideas and energy to their workplace. Employees who are not engaged are not necessarily bad employees, but they are not invested in their work; they punch in and punch out each day. And then there are actively disengaged employees who are so miserable they try to spread their misery to others. Like well-being, being engaged at work makes a big difference in an employee’s life and in the outcomes of the organizations they work for.

Highlights from the Gallup-Purdue Index revealed that, on the likelihood of graduates being engaged in their work later in life or thriving in their well-being, there was no difference by institutional typology—no difference between public versus independent, highly selective versus not, and not even when comparing the top 100 ranked schools in U.S. News & World Report to all others. To be sure, there are individual colleges and universities that are excelling on these measures, but it’s not because of the type of institution; rather, it is because of what the institution is doing or not doing, what it values or doesn’t value. In other words, it’s more about intentionality than typology.

What matters is the kind of emotional support or deep, experiential learning the graduates experienced as students. And the whole story is told by behavioral economic measures. Graduates who were emotionally supported during college or who had deep, experiential learning are two times as likely to be engaged in their work and thriving in their well-being later in life. The Gallup-Purdue Index identified six of these key experiences that serve as potential career- and life-trajectory game-changers. The key emotional support elements can be described as follows: having a professor who made students excited about learning, professors who cared about them as a person, and a mentor who encouraged their goals and dreams. The experiential learning elements can be described thusly: working on a project that took a semester or more to complete, having a job or internship where they applied what they were learning, and being extremely involved in extracurricular activities. The good news is that these critical experiences for students dramatically improve their odds of success later in life. The bad news is two-fold: most colleges and universities have never measured these things in earnest, and only 3 percent of U.S. college graduates strongly agree that they have experienced these six measures (see chart).

Mission: We Are What We Measure

Many of us have heard the expression, “We are what we measure.” But few stop to ponder whether our organizations are actually measuring what we value. Consider college and university mission statements. Presumably, these statements articulate the most important outcomes the institution wants to achieve. Trustees of colleges and universities have a responsibility to guide the institution according to its mission or, at times, to modify it. They are entrusted to hold the institution accountable to its stated mission. Otherwise, why even have such a statement? But if there were a grade assigned to how higher education is doing according to its collective mission statements, it would be an “F” or a dropped class. Colleges and universities simply aren’t measuring whether they are accomplishing their stated missions.

These mission statements articulate lofty ideals and goals: engaged citizenry, research in the service of society, global leadership, fulfilling lives, etc. The challenge is less about whether those are the right aims and more about how institutions will know if they are meeting them. Gallup read and analyzed hundreds of college mission statements as part of a recent study; most are very similar, with a handful of words and phrases used by the majority of institutions, the most common being “life-long learning.” Presumably, then, there is extensive data measuring how well universities are doing in producing life-long learners among their graduates. Ask yourself whether you’ve ever seen such data, nationally or for your own institution. Likely, you haven’t, because few if any institutions have ever sought to measure it.

What would it look like if we measured our mission statements in higher education? Recent Gallup studies shed some light on this, but here’s a hint: doing so requires using behavioral economic measures.

Less than one-third of all U.S. college graduates, for example, strongly agree that their professors cared about them as a person, that they had a job or internship where they applied what they were learning, or that they worked on a long-term project. Collectively, a mere 3 percent hit the mark on all six experiences. And stunningly, a full quarter missed the mark on all of them. And here’s how important the nuance of the wording in the questions is: We also asked graduates whether they had a paid job in college as a “yes” or “no” question. Among those who said “yes,” their response had no relationship with their workplace engagement later in life. But if they strongly agreed with the separate statement indicating they had a job or internship in which they applied what they were learning in the classroom, it doubled the odds that they ended up engaged at work later in life. The difference is demonstrated by their feelings as to whether or not their work experience had relevance to their learning. There is no other way to know this than to ask the student or graduate directly.

For graduates who strongly agreed with all six of these statements, 82 percent of them also strongly agreed that their college prepared them well for life after college; among those who missed the mark on all six, only 5 percent strongly agreed. This is a 16x differential on feeling well-prepared depending on whether graduates agree with these six key statements.

The Future of Behavioral Economics and Higher Ed

Now that Gallup has conducted these kinds of studies for dozens of individual colleges and universities, it’s become clear that institutions are gaining dramatic new insights that point them toward specific ideas and actions for improving. And it’s providing the country with a different perspective of what matters and how to measure quality in higher education. There is, indeed, a wide range of variance in how institutions are doing on these measures. There’s even a wide variance within programs or departments on the same campus. Based on the national data, few are satisfied with how higher education is scoring on these metrics. They all beg for improvement. Even institutions that fare well relative to national benchmarks are eager to improve. And the best news is that we have nowhere to go but up. Higher education has never really measured these kinds of things, and once it starts doing so, it will make incredible progress.

These measures are also changing how we think about performance. Take historically black colleges and universities (HBCUs), for example. On measures such as graduation rates and student loan default rates, they don’t fare well. But in looking at the percentage of their graduates who strongly agree they had the six key experiences, as documented in a Gallup- USA Funds Minority Graduate study, the story changes dramatically. Black graduates from HBCU institutions are two to three times more likely to hit the mark on these experiences than black graduates from all other types of institutions. It’s a remarkable difference. The behavioral economic measures that look so good for HBCUs don’t change the classic measures like graduation rates that don’t look so good, but this most definitely changes our view of their overall performance. Their relevance, based on these new measures, has never been greater.

As our measures of performance and success shift, so will our understanding of which colleges and universities are “best.” Today, the Ivy League dominates the top rankings on nearly every dimension of classic economics in higher education: selectivity, pre-matriculation test scores and GPAs, graduation rates, and endowment size. But how do they fare on the six key college experiences identified in the Gallup-Purdue Index? Their graduates rank higher than the national average on three of the six, and they are at the national average on one experience. They are among the lowest of all institutional cohorts Gallup has studied on two particular items: having professors who cared about them about as a person and having a mentor who encouraged their goals and dreams. Among graduates from the Ivy League, 17 percent strongly agreed their professors cared about them as a person; among graduates from HBCUs, 58 percent did. On this measure—based entirely on the view of graduates themselves— HBCUs perform nearly three times better than the Ivy League.

The institutional-level variance Gallup has observed on these measures is also astounding. Some colleges and universities are doing six times better on these metrics than others, as demonstrated by the chart above, in which the range is 44 percent to 7 percent of graduates who strongly agree their professors cared about them as a person.

Board members need to know where their institutions stand on these measures, which need to be an ongoing part of a broader set of metrics used to guide institutional performance. Just as the healthcare industry was transformed by patient surveys a decade ago, higher education will be transformed by the coming data revolution. It will be an exciting new frontier that will yield a dramatic improvement in performance on factors that matter most for the relevant constituents and consumers of higher education—and to the aims of its collective, powerful mission, as well.