Google’s John Mueller answered a query on Reddit a few generally used robots meta tag and what would occur if it was lacking. Mueller’s reply, whereas it is sensible and is documented, should come as a shock to many publishers and SEOs.
Robots Meta Tag
The HTML meta ingredient communicates metadata. Metadata is machine readable data {that a} crawler like Googlebot can learn.
There are lots of sorts of meta components just like the meta description ingredient however the Robots Meta Ingredient is totally different in that it could actually management the search engine crawlers.
The knowledge communicated by the robots meta tag known as a directive, which signifies that robotic crawlers are obligated to obey the directions within the robots meta tag.
There are lots of instructions to move alongside by the robots meta however the next meta tag is one that’s related to the query John Mueller answered.
The noindex, no comply with meta tag:
<meta title="robots" content material="noindex, nofollow">
The above meta tag tells the search engine crawlers to not index the content material on the webpage and to not comply with any hyperlinks.
Some of the widespread meta tags is that this one, which instructions search engines like google to index the content material and comply with all of the hyperlinks:
<meta title="robots" content material="index, comply with">
Whereas the above meta tag is widespread, there’s a vital quantity of confusion about it. There’s a line of reasoning that as a result of Google helps nofollow then it should indicate that Google helps the comply with directive.
I discovered quite a few authoritative web sites that say that Google makes use of the meta robots index, comply with meta tag.
However that’s not truly how Google makes use of these directives, as John Mueller makes clear in his reply.
What’s The Impact Of Leaving Out The Meta Robots Index Tag?
The particular person on Reddit requested the next query:
“I’m a bit confused with a web site I’m engaged on.
So, that is what the meta snippets on a lot of the web sites I work on appear like:
<meta title=’robots’ content material=’index, comply with ….
However, on the web site at hand, it’s lacking the ‘index’ tag.
My query is: What’s the impact of the location lacking the ‘index’ tag.”
John Mueller answered:
“The “index” robots meta tag has no perform (no less than in Google) – it’s utterly ignored. Additionally “comply with”.
Google has https://builders.google.com/search/docs/crawling-indexing/special-tags & https://builders.google.com/search/docs/crawling-indexing/robots-meta-tag to doc the meta tags which have capabilities. You need to use anything, it’ll be ignored. <meta title=”robots” content material=”topranking bestcheese”> is an possibility, if you wish to throw individuals off.”
Why Google Ignores Robots Index & Comply with
The easy purpose why Google ignores the robots index and comply with meta tag is as a result of indexing and comply with are the defaults.
Indexing and following hyperlinks is what search engine robots do, they don’t should be advised to index content material and comply with hyperlinks as a result of that’s their objective.
Google’s documentation on robots tags advises:
“The default values are index, comply with and don’t have to be specified.”
The total record of legitimate directives for Google is right here.
If the robots meta you need to use isn’t’ listed there then Googlebot goes to disregard it.
Is Index, Comply with Fully Ineffective?
It’s true, documented and official that in the case of Googlebot, <meta title=”robots” content material=”index, comply with”> is a waste of HTML area and is ignored by Googlebot.
Bing treats index, comply with in the same approach however with a slight distinction, as described within the official Bing documentation for meta tags.
That is what Bing’s documentation says in regards to the index directive:
“By default we assume “index”, but when wanted you need to use <meta title=”robots” content material=”index”> to explicitly state that we could index the web page.”
And that is what it says in regards to the comply with directive:
“By default we assume “comply with”, however you possibly can explicitly state “comply with” if that’s the case desired.”
In my 20+ years of working in website positioning creating web sites, optimizing them and rating them, I’ve all the time thought-about it a superb coverage to provide the bots what they count on and take a look at to not give them something that’s sudden. So if a meta description just isn’t crucial then my impulse can be to depart it out as a result of the entire level of optimizing is to make it as straightforward as doable for the various search engines to index and perceive the content material, which suggests to eliminate something that may work in opposition to that aim.
On this case, it’s extremely probably it’s not going to have an impact in some way.
However… There’s one other approach that comply with and index journey individuals up.
Some publishers use this robots meta tag:
<meta title="robots" content material="noindex, comply with">
Some websites advise that if the web page isn’t listed, the usage of the “comply with” directive compels the search engine to comply with the hyperlinks.
However that’s not true if there’s a “noindex” directive for the straightforward causes that Google can not comply with a hyperlink on a web page that it isn’t listed. If it’s not within the index then the hyperlinks on these pages are usually not within the index.
Featured Picture by Shutterstock/Bangun Inventory Productions