Online Search Engines - Overview
Table of Contents
After reading this post, you’ll have a general overview of the components of online search engine (OSE) and how an OSE interacts with large language models (LLM).
You may have heard that LLMs will replace traditional online search like Google’s since LLMs can produce informative answers to our search queries relatively quickly, but that’s not quite true. While the interface will mostly contain LLM’s outputs, the AI needs to have updated information about the world for many queries. Just like a human needs to read the news (using an OSE 😄). Thus, search engines are here to stay.
The Web and OSEs form a suitable infrastructure to find results related to a query and fed them into an LLM. However, in a more futuristic (and dystopian) setting, our personal smart devices could be connected to a hub, constantly pouring all our information into a database consumed by an AI, which would know it all. Kind of like in my sci-fi novel, where such devices are our wired brains: GÖD’S GATE Check it out! 📚🚀
Online Search Engine - Main Components
An online search engine (OSE) is composed of several building blocks that together enable users to find relevant information on the Internet by typing queries. It may involve several main interconnected components that work together to (1) crawl, (2) index, (3) process queries, (4) rank, and (5) serve web content. This explainer is centered around Google’s OSE, since it is the one with most information available online.
1. The Web Crawler (Spider/Bot)
With a set of initial URLs, the crawler visits and revisits these URLs and downloads and renders page content to extract more hyperlinks (new URLs) and content. Website owners can submit a sitemap for Google to crawl, which helps Googlebot’s “crawl budget,” so that pages are crawled efficiently.
Crawl budget refers to the number of pages a search engine will crawl on a website within a given timeframe. It’s determined by two main factors: the crawl limit—how frequently the search engine can crawl without overloading the server—and crawl demand—how frequently the search engine wants to crawl the site based on its popularity and freshness of content.
2. The Web Index (Indexer)
This component will process, analyse, and organise crawled data for quick retrieval. It will parse web content (in raw HTML form) to extract key terms (tokenization) from data: titles, metadata, headings, links, text, images, videos, etc. and build an inverted index.
This inverted index is a massive database designed to quickly map search queries to documents (typically represented by URLs or document IDs). In this index, each term is associated with a list of documents that contain it, often alongside metadata such as term frequency (TF), document length, language, content type, freshness, and quality signals. This index may also store additional fields like keyword importance or semantic annotations. To ensure scalability and low-latency retrieval, it is typically stored in a distributed file system with caching and replication layers for fault tolerance and fast access.
3. The Query Processor
It interprets and prepares the query to retrieve URLs. Processing involves tokenization, spelling correction, synonym expansion, language detection, query rewriting, etc. With the processed information, the OSE will match query terms to the inverted index.
4. The Ranking Engine
It ranks the matched documents from the index based on their relevance to the user, with the goal of presenting the most relevant and authoritative content at the top of the search results. This is the secret sauce. The ranking engine will consider content relevance (term frequency, keyword matches, content freshness), authority signals (backlinks, domain authority, page speed), user behaviour (click-through rate, bounce rate, dwell time), and semantic understanding of the query’s contextual meaning and user intent. The ranking engine uses machine learning (ML), like BERT and RankBrain, to improve such understanding and increase the overall OSE’s accuracy. Here are some of the algorithms used by Google to rank results:
PageRank is an algorithm that measures the importance of a web page by counting the number and quality of links pointing to it (called backlinks or inbound links). The underlying assumption is that more important websites are more likely to receive links from other high-quality sites, with each hyperlink acting as a “vote of support.” A page linked to by many high-ranking pages will itself receive a high rank on the results page.
RankBrain is an algorithm that uses artificial intelligence to learn how users interact with search results and adjusts its understanding of queries beyond keywords (semantic search) to improve future search experiences. It sorts search queries into “word vectors” based on linguistic similarity, which allows it to make educated guesses about the intent/meaning of words or phrases it hasn’t seen before. That’s a critical capability, since unknown searches account for a large portion of all searches. RankBrain learns from data, including how users click on search results and how long they spend on those pages. RankBrain works alongside other algorithms, such as Google Hummingbird (launched in 2013), designed to better understand the intent behind search queries rather than just matching individual keywords.
In essence, RankBrain helps Google understand the “meaning” behind search terms, even if they’re not exact matches to content.
BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing (NLP) model that learns bidirectional representations of text improving contextual understanding, i.e. BERT learns from both the left and right context of words, providing a more comprehensive understanding of their meaning. It uses the Transformer architecture to analyze text sequences, which allows for parallel processing, making it more efficient than traditional models. It is trained using two primary methods: masked token prediction (predicting missing words in a sentence) and next sentence prediction (determining whether one sentence logically follows another). BERT helps Google understand the context of user queries to deliver more relevant search results.
Google uses these three algorithms depending on the query’s needs.
| Feature | PageRank | RankBrain | BERT |
|---|---|---|---|
| Introduced | 1998 | 2015 | 2018 |
| Based on | Link analysis (graph theory) | Word embeddings + ML | Transformers (deep NLP model) |
| Focused on | Ranking web pages by importance | Query reformulation, ranking adjustments | Semantic matching of queries and documents |
| Stage | Ranking | Query understanding + ranking | Query and document understanding |
Query Understanding: What the user intents to say (spelling correction, sysnonym expansion, named entity recognition (e.g. “apple” = fruit vs. company), language detection, intent detection, parsing grammar).
Ranking: Order the search results by relevance and quality to the query.
PageRank says: “This page is important.”
RankBrain says: “This confusing query is related to another one I’ve seen.”
BERT says: “I understand which documents make sense in the query context.”
Beyond PageRank, RankBrain, and BERT, Google leverages proprietary datasets to maintain its ranking edge:
Glue Data: Essentially a “super query log” that records detailed information about a user’s query and subsequent interactions with the SERP (search engine results page). This includes:
Query details (text, language, user location, device type).
Ranking information (the 10 blue links, Knowledge Panel, People also ask, etc.).
SERP interaction data (clicks, hovers, dwell time, bounce rate).
Query interpretations and suggestions (spelling corrections, salient terms).
A key component of Glue is Navboost Data — a memorization system that aggregates click-and-query interactions to strengthen ranking signals. Google trains Navboost on 13 months of user data, which is equivalent to over 17 years of data processed by Bing, giving Google a significant scale advantage.
RankEmbed Data: Ranking models (and later iterations such as RankEmbedBERT) that blend click-and-query logs with human rater scoring, sed by Google to measure the quality of organic search results. While this represents only a small fraction of Google’s traffic, it has substantially improved Google’s quality, particularly for long-tail queries. These AI-driven models use deep learning for natural language understanding, enabling better retrieval of documents even when queries lack exact keyword matches.
FYI Importantly, remedies from recent US legal actions seek disclosure of underlying user-side data (like Glue or RankEmbed training data), but not the actual models or signals derived from it (https://www.justice.gov/opa/pr/department-justice-wins-significant-remedies-against-google).
5. The Search Interface
The frontend that shows the SERP provides an interactive interface for users to type queries. It’s the page you see when you type a search query into a search engine like Google or Bing. The SERP displays the list of results that the search engine returns in response to your query, including both organic and paid search results. Every SERP is unique because search engines customize the results for users based on factors beyond just the search terms, such as the user’s physical location, language, device, browser history, and other settings. While the main function of the SERP is to provide the user with a search bar and show search results, the user may query, filter, and interact with content in other ways depending on the UI developer.
Examples of Search Engines
While Google is the dominant player in online search, there are several alternatives, including Microsoft Bing, Brave Search, DuckDuckGo, and Qwant. It’s important to distinguish between a web browser — which renders and displays web content — and an online search engine (OSE) — which retrieves relevant content based on a user query. Many browsers, such as Mozilla Firefox and Safari, use third-party OSEs like Google Search by default. Alternatively, some services like Ecosia (a “search engine”) syndicate search results from larger OSEs such as Bing (as of this writing) — meaning that when you enter a query into Ecosia’s search bar, the query is forwarded to Bing’s search engine, and Bing returns the results on ECOSIA’s SERP.
Some other OSEs are metasearch engines like Dogpile, which aggregate results from multiple sources. Social media platforms like YouTube and Amazon also function as specialized search engines.
Monetization
OSEs primarily make money through advertising, with the most popular model being Pay-Per-Click (PPC). Advertisers bid on keywords, and their ads are displayed alongside search results. The search engine gets paid each time a user clicks on an ad.
Integration with LLMs
The relationship between OSEs and LLMs is mutually beneficial and is creating a new paradigm for search.
LLMs are integrated into search engines to:
Improve Query Understanding: They analyze the semantic content and context of a query, going beyond keywords. This allows them to handle complex or conversational questions. Further, LLMs can evaluate the semantic relevance of documents to a query, prioritizing content that aligns with the user’s intent (see point 4 above on BERT).
Generate Content and Summaries: LLMs can synthesize information from multiple sources to generate direct answers, summaries, or featured snippets, which appear directly in the search results. This is often achieved using a technique called Retrieval-Augmented Generation (RAG), where the LLM uses real-time search results to provide current and accurate information, helping to prevent “hallucinations,” (i.e. when the LLM outputs false statements).
Personalize Results: By analyzing a user’s history and behavior, LLMs can tailor search results to match a user’s expertise or interests.
GÖD’S GATE
Liked this post? Then there’s a statistically significant chance you’ll enjoy my sci-fi novel GÖD’S GATE, check it out!
Donations
Feeling generous? Toss a coin to your blogger via Paypal 😀
References
In-Depth guide to how Google Search works
https://developers.google.com/search/docs/fundamentals/how-search-works
Domain authority
https://ahrefs.com/website-authority-checker
RankBrain
https://en.wikipedia.org/wiki/RankBrain
BERT
https://en.wikipedia.org/wiki/BERT_(language_model)
Crawl budget
https://www.conductor.com
Semantic Search
https://blog.ml6.eu/semantic-search-a-practical-overview-bf2515e7be76
Website “Authority” Checker
https://ahrefs.com/website-authority-checker
Google Search Primer\ https://developers.google.com/search/docs/fundamentals/how-search-works
Google Search Overview
https://surferseo.com/blog/how-do-search-engines-work/#:~:text=Search%20engines%20are%20systems%20designed,content%20to%20deliver%20relevant%20results
Search Engine - Wikipedia
https://en.wikipedia.org/wiki/Search_engine
Department of Justice Wins Significant Remedies Against Google
https://www.justice.gov/opa/pr/department-justice-wins-significant-remedies-against-google
Disclaimer
Any views expressed in this post are solely those of the author and do not represent the opinions or policies of any affiliated organizations.
