Breaking Through the (Scholarship) Noise

Scholarship awarding is fraught with noise and bias. Much has already been written about bias but not noise. In terms of awarding scholarships, noise is tantamount to the effects bias can have on who gets which scholarships and for how much. Focusing on bias alone when making evaluative decisions (like scholarship selections) is equivalent to flipping a one-sided coin. We need to be aware of both bias (heads) and noise (tails) to achieve fairer outcomes in scholarship selections. 

First, we can talk about bias. Bias has become synonymous with “error.” But bias is simply a consistent preference. If we look at judgments or the errors in those judgments, and they all follow the same direction on average, that is bias. But to be clear, bias is baked into scholarships when a donor decides the eligibility criteria for their gift. While there are often issues with donor criteria, that is not the focus of this article. The bias we are addressing is about what happens after a student has met the donor’s minimum requirements. The two types of bias we may see are implicit and cognitive. They are different, and that distinction is critical for students. 

When selecting students for scholarships, we certainly want to prevent stereotypes from influencing evaluations. We try to mitigate implicit bias by hiding applicant names, race, ethnicities, gender or any information that could potentially negatively impact a student’s chances of winning a scholarship.  

We also want to mitigate the kind of bias that could unfairly boost one student’s chances to earn a scholarship over another, better-suited candidate, which is where cognitive bias comes into play. This is affinity bias, meaning, we favor others like us. For example, say the student is from the same hometown as the reviewer, and therefore that student receives a more favorable rating over other, stronger applicants.  

Now, it’s time to talk about noise. Noise, in this context, is not dogs barking on Zoom. It is not a measurement error or that shaming voice in our heads for not working out more. The noise we are referring to stems from Daniel Kahneman’s book: Noise – A Flaw in Human Judgement. In this context, noise is a judgment error where we don’t show the same behaviors for similar decisions. More specifically, noise is an unwanted variability in professional judgments. Broadly speaking, professional judgment means a process for making repetitive decisions, like deciding who and how much to award in scholarships to. 

Many scholarship awarding decisions are made by a single individual. Some of those individuals will be more generous in the award amounts they give out, while others may always divide the balance of a fund up between several students regardless of the total available to award. These individuals would be considered as having a bias towards either giving out larger awards or spreading smaller awards among more students. Both types of individuals have biased tendencies when allocating funds. 

But when comparing these two awarders, we see “level noise” because the amount a student receives depends on which awarder is assigned to the fund that the student is under consideration for.  Suppose a student is under consideration for a fund that has a starting balance of $2,000. The student may receive all $2,000 if the fund is assigned to a generous awarder. Or the student may only receive $500 because the fund is assigned to someone who prefers to spread the $2,000 across four different applicants. This poses an important question to awarders: Should a scholarship recipient and award amounts depend on who is assigned responsibility for awarding a particular fund, or should awarders be interchangeable? 

And there’s also occasion noise, which suggests that the mood of the awarder may influence awarding decisions. Our moods can be drivers of our judgments, and we are affected by a multitude of factors like how rested we feel, when the last time we ate was, our stress level, and even the weather or the outcome of a game. 

If we are in a bad mood, we may be disengaged from the process and only give a cursory look over applicants. Or we may become hypercritical and discount a student’s entire application for things we normally wouldn’t, like a typo in an essay for example. (Seriously, please don’t do that.)  

On the other hand, if we’re in a good mood, we may become less critical. That could result in students who we may normally not consider for a scholarship ending up in the running. Or we may switch our own bias of spreading awards out to larger, more generous gifts because we really liked a student at that moment.  

Here are some recommendations to reduce bias and noise in the scholarship selection process: 

  • Emphasize diversity – If you have a review committee, make sure they are as diverse as possible for the sake of having multiple viewpoints. This helps mitigate bias. 
  • Seek independence – Do not have review committees discuss their reviews with each other. Undue influence can be introduced simply by the first person who speaks or if one of the reviewers is an authority figure. Reviews should be conducted independently. 
  • Find and hide discriminatory data – As noted earlier, hide any information from reviewers that could result in undue bias like student names, race, ethnicity, gender, etc.  
  • Deploy mood regulation – it is important to be aware of what type of mood you’re in when reviewing and awarding scholarships. Don’t let ups and downs or fatigue affect your work on the task at hand. If your moods are too much in one direction, step away until you’re more even keel.  
  • Review scales – many are familiar with scoring guidelines like “demonstrates leadership.” however, “demonstrates more leadership than the previous student” will tend to bring less noise, as it is a direct and relative comparison. Therefore, the first student you look at in a session should not be scored until having also reviewed another student for comparison.  
  • Regularly audit – conduct bias and noise audits regularly. Start by surveying the process and results for a set of funds. Who was selected? Are there patterns? What were the balances versus award amounts given, and to how many students were they awarded to? Remember, a bias or noise audit is not meant to punish anyone for the decisions they made, but rather, to see where bias may exist and how much noise exists, too. 
  • Introduce awarding scales to your processes – Consider implementing awarding scales to reduce level and occasion noise. Any deviation from the scale would require a note as to why, such as the fund’s remaining balance being below the required minimum award amount. 
  • Replace human judgment with algorithms – Do this anywhere possible, as they are often more exact than those made by experts, even when the experts have access to more information than the formulas used. Humans always introduce noise. Algorithms do not get “hangry” or moody. They also never get tired; that means no decision fatigue.  

But you might say, algorithms have shown themselves to be biased. In fact, they are. They are designed to create repeatable results without fluctuations of mood. The concern with biased outcomes from algorithms is more about the data or inputs of the algorithms that result in unwanted outcomes.  

Remember: The goal isn’t eliminating algorithms from the process entirely. It should be to use them more, make them less biased towards undesirable outcomes and ultimately create more balance in awarding. ScholarshipUniverse’s SmartRank capability is one such algorithm that can help improve awarding consistency.  SmartRank combines the quality of a student’s specific application to rank applicants for awarding. Factors like demographics, academic and financial data are all considered as part of this process.

Every student deserves access to funding for a higher ed degree. If we know that scholarships are a critical piece to ensuring that opportunity, we must do all we can to make the process equitable. One of the most vital steps to make the process of awarding equitable is to understand our own biases and work to reduce noise. 

 

About the Author

Ken Downs, Product Evangelist

Evangelizing how technology and best practices can remove friction for students, schools, and organizations in the scholarship and financial aid space, Downs is CampusLogic’s Product Evangelist. He was instrumental in leading the development of the University of Arizona’s national award-winning ScholarshipUniverse product, now part of CampusLogic’s platform. A longstanding member of the National Scholarship Provider’s Association, Downs has given workshops titled: Big Data and the Coming Scholarship Sea Change, Scholarship Hacking, and the Scholarship Universe Story.

Follow on Twitter Follow on Linkedin Visit Website More Content by Ken Downs, Product Evangelist
Previous Article
CampusLogic at NASFAA 2022: Session Preview
CampusLogic at NASFAA 2022: Session Preview

Amy Glynn shares her can't-miss session list (and a few tips) to make the most of NASFAA 2022 in Austin.

Next Article
It’s Decision Season For Students and Colleges
It’s Decision Season For Students and Colleges

Figuring out how to make buying college simpler – be it by reducing steps or speeding up communications – i...