Google published a detailed explainer about how changes to search rankings are evaluated internally before being rolled out to users.
Danny Sullivan, Google’s Public Liaison for Search, goes over the role of search quality raters and how their evaluations are incorporated into algorithm updates.
Thousands of changes to Google’s search algorithms are made every year, and they all go through a process to determine whether they will be useful for searchers around the world.
Here’s are some ways that feedback from quality raters and regular people are used in algorithm updates.
Research
Google’s research team talks to people around the world to understand how Search can be more useful:
“We invite people to give us feedback on different iterations of our projects and we do field research to understand how people in different communities access information online.”
Ultimately, Google’s mission is to make information universally accessible and helpful, with a commitment to serving all users in pursuit of that goal.
Search Quality Raters
Google has publicly available rater guidelines that describe how its algorithms intend to surface content.
The guidelines are over 160 pages long, but if it were to be narrowed down to single phrase then Sullivan puts it this way:
“… we like to say that Search is designed to return relevant results from the most reliable sources available.”
Many signals are picked up by Google’s algorithms automatically. But when it comes to signals like relevance and trustworthiness, those require human judgement.
Google has a group of more than 10,000 people all over the world who are referred to as “search quality raters.”
Raters help Google evaluate how people are likely to experience search results.
Ratings are provided based on Google’s guidelines, and are intended to represent real users and their likely information needs.
Search quality raters study and get tested on Google’s rater guidelines before they can begin to provide feedback.
Related: Google’s Search Quality Raters Guidelines: A Guide for SEO Beginners
How Search Quality Raters Work
Google assigns a group of raters a set of queries, and they’re shown two versions of results pages for those searches.
One set of results is from the current version of Google, and the other set is from an improvement we’re considering.
Every page that appears in the results is evaluated against the query, based on the rater guidelines.
To evaluate things like expertise, authoritativeness, and trustworthiness—sometimes referred to as “E-A-T”—raters are asked to do reputational research on the sources.
Once raters have done this research, they then provide a quality rating for each page
Ratings Not Used Directly For Rankings
Sullivan states:
“It’s important to note that this rating does not directly impact how this page or site ranks in Search. Nobody is deciding that any given source is “authoritative” or “trustworthy.” In particular, pages are not assigned ratings as a way to determine how well to rank them.”
Search quality ratings are a data point that help Google measure how well its systems are working to deliver great content that’s aligned with the its guidelines.
For more information, see Google’s article here.
Related: Quality Raters Guidelines and Google Ranking Signals
