Get the Top 10 Inbound Marketing News Every Month

Sign Up to GrowthI'm in!

Tag: robots.txt

Google: Robots.txt to Become an Official Standard After 25 Years

In a series of tweets by Google Webmasters, they have announced their proposal of a draft stating that Robots.txt is well on its way from becoming a de facto standard to an official one.

Read More

The Complete Guide to Robots.txt and Noindex Meta Tag

How do I use Robots.txt and the noindex meta tag? Quick Answer: To create a Robots.txt file, you can use any text editor (such as Notepad). Make sure to save the file with UTF-8 encoding during the save file dialog. This file must also be named “robots.txt,” and your site can only have one such […]

Read More

How Faceted Navigation Affects SEO

Faceted navigation is problematic for almost all e-commerce websites. The number of pages that e-commerce websites have on different versions of a single product poses a threat to the crawler’s efficiency – which could negatively affect your SEO. But what does faceted navigation really mean?

Read More

How Robots.txt and Meta Tags Affect Search Engine Crawling

If you are concerned about the privacy of your website and you do not want the search engine crawlers or bots to crawl certain pages of your website, then “Robots.txt” is the one-stop solution that will keep the crawlers away from the ‘No Entry’ zone.

Read More