Incapsula Security on The Behaviors of Googlebot & Fake Googlebot

Incapsula Security’s latest report shares insight into the behavior and motives of Googlebot – and while the findings proved to be positive for SEO managers, they were slightly overshadowed by the habits formed by malicious Googlebot imposters, also known as Fake Googlebots.

two-faces-of-google-dr-crawlit-mr-hack

 

Here are some of the report’s most noteworthy finds.

On Googlebot:

  • Google owns, naturally, the most efficient of all bots on the web. However, Incapsula found that Google crawls more pages than all other spiders combined, shouldering 60.5% of all total page crawls.
  • They saw no correlation between volume of human visits and Google bot visits, meaning Google does not spend any more time on websites deemed to be “popular”.
  • From 210 million+ Googlebot sessions, Incapsula found that Googlebot’s average visit rate per website is 187 visits/day, and average crawl rate is 4 pages/visit. However, content-heavy and websites that were updated routinely were, as a whole, more thoroughly crawled. This is especially true with news sites, large forums and high-scale retailers with frequently updated products.
  • Over 98% of all Googlebot visits originate in the US, with small shares coming from the UK, France, Belgium, Denmark and China.

On Fake Googlebot:

  • 4% of bots operating with Googlebot’s HTTP(S) user agent are not who they claim to be. Fake Googlebots now account for one in every 24 visits to sites across the web, as malicious activity is up 61%.
  • 34% of all imposters ID’ed were explicitly malicious; 23% of these were used for Layer 7 DDoS attacks.

fake-googlebots-by-type-of-activity

  • While a quarter of Fake Googlebot visits come from the US, other origins of suspect include China, Turkey, Brazil, India and Thailand.