TechnologyWebsite BuildersWordPress Hosting

What Is Googlebot & How Does It Work?

Googlebot is the online crawler utilized by Google to collect the data wanted and construct a searchable index of the online. Googlebot has cell and desktop crawlers, in addition to specialised crawlers for information, pictures, and movies.

There are extra crawlers Google uses for specific tasks, and every crawler will establish itself with a unique string of textual content referred to as a “person agent.” Googlebot is evergreen, which means it sees web sites as customers would within the newest Chrome browser.

Googlebot runs on 1000’s of machines. They decide how briskly and what to crawl on web sites. However they are going to decelerate their crawling in order to not overwhelm web sites.

Let’s take a look at their course of for constructing an index of the internet.


How Googlebot crawls and indexes the internet

Google has shared just a few variations of its pipeline up to now. The beneath is the newest.

Google begins with an inventory of URLs it collects from varied sources, akin to pages, sitemaps, RSS feeds, and URLs submitted in Google Search Console or the Indexing API. It prioritizes what it needs to crawl, fetches the pages, and shops copies of the pages.

These pages are processed to seek out extra hyperlinks, together with hyperlinks to issues like API requests, JavaScript, and CSS that Google must render a web page. All of those further requests get crawled and cached (saved). Google makes use of a rendering service that makes use of these cached assets to view pages much like how a person would.


It processes this once more and appears for any adjustments to the web page or new hyperlinks. The content material of the rendered pages is what’s saved and searchable in Google’s index. Any new hyperlinks discovered return to the bucket of URLs for it to crawl.

We’ve got extra particulars on this course of in our article on how search engines like google work.

How to regulate Googlebot

Google offers you just a few methods to regulate what will get crawled and listed.

Methods to regulate crawling

Methods to regulate indexing

  • Delete your content material – In the event you delete a web page, then there’s nothing to index. The draw back to that is nobody else can entry it both.
  • Limit entry to the content material – Google doesn’t log in to web sites, so any form of password safety or authentication will stop it from seeing the content material.
  • Noindex – A noindex within the meta robots tag tells search engines like google to not index your web page.
  • URL removal tool – The identify for this instrument from Google is barely deceptive, as the best way it really works is it would quickly conceal the content material. Google will nonetheless see and crawl this content material, however the pages received’t seem in search outcomes.
  • Robots.txt (Photographs solely) – Blocking Googlebot Picture from crawling signifies that your pictures won’t be listed.

In the event you’re undecided which indexing management you must use, try our flowchart in our put up on eradicating URLs from Google search.


Is it actually Googlebot?

Many web optimization instruments and a few malicious bots will fake to be Googlebot. This will permit them to entry web sites that attempt to block them.

Prior to now, you wanted to run a DNS lookup to verify Googlebot. However not too long ago, Google made it even simpler and offered a list of public IPs you should use to confirm the requests are from Google. You may examine this to the info in your server logs.

You even have entry to a “Crawl stats” report in Google Search Console. In the event you go to Settings > Crawl Stats, the report incorporates a variety of details about how Google is crawling your web site. You may see which Googlebot is crawling what recordsdata and when it accessed them.

Line graph showing crawl stats. Summary of key data is above

Closing ideas

The net is a giant and messy place. Googlebot has to navigate all of the totally different setups, together with downtimes and restrictions, to collect the info Google wants for its search engine to work.

A enjoyable reality to wrap issues up is that Googlebot is often depicted as a robotic and is aptly known as “Googlebot.” There’s additionally a spider mascot that’s named “Crawley.”

Nonetheless have questions? Let me know on Twitter.

Show More

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *