Get the Top 10 Inbound Marketing News Every Month

Sign Up to GrowthI'm in!

Tag: GoogleBot

15MB Only for Googlebot Today

When Google decided to update its official documents in June 2022, SEO practitioners and specialists started worrying about a particular piece of content that’s been added to the Googlebot crawl limit. Stated within Googlebot’s official Google documentation is the following statement: “Googlebot can crawl the first 15MB of an HTML or support text-based file. Any […]

Read More

GoogleBot User Agent’s Update Rolls Out in December

GoogleBot optimization can be tricky especially since you cannot control how the spider perceives your site. As Google announced that they are going to update the user agent of GoogleBot come December, this is also a call for us SEOs to rethink how we can optimize for Google crawlers.

Read More

The Complete Guide to Robots.txt and Noindex Meta Tag

How do I use Robots.txt and the noindex meta tag? Quick Answer: To create a Robots.txt file, you can use any text editor (such as Notepad). Make sure to save the file with UTF-8 encoding during the save file dialog. This file must also be named “robots.txt,” and your site can only have one such […]

Read More

Using Log File Analysis to Optimize Your SEO Strategy

  The challenge for webmasters everywhere is to accurately know what GoogleBot is doing on your website. GoogleBot’s primary job whenever they enter a website is to crawl a specific number of pages set by the website’s crawl budget. After crawling, they save the pages they crawled to Google’s database.

Read More