How Search Engines work – Understanding Google & Co

by Maximilian Claessens

Search engines are crucial for finding information and websites. But do you know how search engines work? In order to understand how search engines work, we have to take a look at their functions.

Search engines have two major functions:

  1. crawling and building an index (indexing), and
  2. Providing search users with a ranked list of the websites they’ve determined are the most relevant.

In the following we will explain each of these two basic functions of search engines, to give you an idea of how search engines work.

  1. Crawling and Indexing – How Search Engines work

Imagine the World Wide Web was a network of stops in a big city subway system.

Each stop stands for one unique document (in most cases a web page, but sometimes a PDF, JPG, or other file). The search engines need a way to “crawl” the entire city and find all the stops along the way. Of course they use the best path available – we call them links.

How Search Engines work - 1. Crawling and Indexing

How Search Engines work – 1. Crawling and Indexing

The link structure of the web binds all the pages together.

Links allow the search engines’ automated robots, called “crawlers” or “spiders,” to reach the many billions of interconnected documents on the web.

Once the search engine finds these pages, they decipher the code from each and store selected pieces in giant databases, to be recalled later when needed for a search query. To carry out the seemingly impossible task of holding billions of pages that can be accessed in a fraction of a second, major search engine companies have constructed data centers all over the world.

  1. Providing the Answers – How Search Engines work

In principle, search engines are answer machines. When a person performs an online search, the search engine searches in its billions of documents and does two things: first, it returns only those results that are relevant for the searcher’s query; second, it ranks those results according to the popularity of the websites serving the information. Both relevance and popularity are factors that SEO attempts to influence.

How Search Engines work - 2. Providing the Answers

How Search Engines work – 2. Providing the Answers

How do search engines determine relevance and popularity?

To a search engine such as Google, relevance means else than finding a page with the right key words. In the early days of the web, search engines didn’t go much further than this. Over the years, search engine companies have devised better ways to match results to searchers’ queries. Today, hundreds of factors influence relevance.

Search engines typically assume that the more popular a site, page, or document, the more valuable the information it contains must be. This assumption has proven fairly successful in terms of user satisfaction with search results.

Popularity and relevance aren’t determined manually. Instead, the engines use mathematical equations (algorithms) to sort the results, and then to rank them in order of quality (popularity).

These algorithms often comprise hundreds of variables.

You may also like