Earlier this week, Google eliminated its Robots.txt FAQ assist doc from its search developer documentation. When requested, John Mueller from Google replied to Alexis Rylko saying, “We replace the documentation sometimes. Be at liberty to submit suggestions when you really feel one thing’s lacking. Robots.txt is certainly nonetheless a factor.”
The Robots FAQ doc lived over right here: builders.google.com/search/docs/crawling-indexing/robots/robots-faq
That now redirects to the primary Google robots.txt assist web page.
What did the Robots FAQ web page say, nicely the Wayback Machine has a duplicate, so I’ll archive it right here:
(Q) Does my web site want a robots.txt file?
(A) No. When Googlebot visits a web site, we first ask for permission to crawl by trying to retrieve the robots.txt file. An internet site with out a robots.txt file, robots meta tag, or X-Robots-Tag HTTP headers will usually be crawled and listed usually.
(Q) Which technique ought to I exploit to dam crawlers?
(A) It relies upon. In brief, there are good causes to make use of every of those strategies:
- robots.txt: Use it if crawling of your content material is inflicting points in your server. For instance, you could wish to disallow crawling of infinite calendar scripts. Do not use the robots.txt to dam non-public content material (use server-side authentication as an alternative), or deal with canonicalization. To guarantee that a URL just isn’t listed, use the robots meta tag or X-Robots-Tag HTTP header as an alternative.
- robots meta tag: Use it if you’ll want to management how a person HTML web page is proven in search outcomes or to guarantee that it isn’t proven.
- X-Robots-Tag HTTP header: Use it if you’ll want to management how content material is proven in search outcomes or to guarantee that it isn’t proven.
(Q) Can I exploit robots.txt, robots meta tag, or the X-Robots-Tag HTTP header to take away another person’s website from search outcomes?
(A) No. These strategies are solely relevant to websites the place you may modify the code or add recordsdata. Be taught extra about easy methods to take away info from Google.
(Q) How can I decelerate Google’s crawling of my web site?
(A) You possibly can usually alter the crawl price setting in your Google Search Console account.
(Q) I exploit the identical robots.txt for a number of web sites. Can I exploit a full URL as an alternative of a relative path?
(A) No. The foundations within the robots.txt file (with exception of sitemap:) are solely legitimate for relative paths.
(Q) Can I place the robots.txt file in a subdirectory?
(A) No. The file have to be positioned within the topmost listing of the web site.
(Q) I wish to block a non-public folder. Can I forestall different individuals from studying my robots.txt file?
(A) No. The robots.txt file could also be learn by numerous customers. If folders or filenames of content material aren’t meant for the general public, do not checklist them within the robots.txt file. It isn’t really helpful to serve totally different robots.txt recordsdata based mostly on the person agent or different attributes.
(Q) Do I’ve to incorporate an enable rule to permit crawling?
(A) No, you do not want to incorporate an enable rule. All URLs are implicitly allowed and the enable rule is used to override disallow guidelines in the identical robots.txt file.
(Q) What occurs if I’ve a mistake in my robots.txt file or use an unsupported rule?
(A) Net crawlers are usually very versatile and sometimes won’t be swayed by minor errors within the robots.txt file. Typically, the worst that may occur is that incorrect or unsupported guidelines will likely be ignored. Keep in mind although that Google cannot learn minds when deciphering a robots.txt file; we’ve to interpret the robots.txt file we fetched. That stated, in case you are conscious of issues in your robots.txt file, they’re normally straightforward to repair.
(Q) What program ought to I exploit to create a robots.txt file?
(A) You should use something that creates a sound textual content file. Widespread applications used to create robots.txt recordsdata are Notepad, TextEdit, vi, or emacs. Learn extra about creating robots.txt recordsdata. After creating your file, validate it utilizing the robots.txt Tester.
(Q) If I block Google from crawling a web page utilizing a robots.txt disallow rule, will it disappear from search outcomes?
(A) Blocking Google from crawling a web page is prone to take away the web page from Google’s index.
Nonetheless, robots.txt disallow doesn’t assure {that a} web page won’t seem in outcomes: Google should determine, based mostly on exterior info akin to incoming hyperlinks, that it’s related and present the URL within the outcomes. In case you want to explicitly block a web page from being listed, use the noindex robots meta tag or X-Robots-Tag HTTP header. On this case, do not disallow the web page in robots.txt, as a result of the web page have to be crawled to ensure that the tag to be seen and obeyed. Discover ways to management what you share with Google
(Q) How lengthy will it take for modifications in my robots.txt file to have an effect on my search outcomes?
(A) First, the cache of the robots.txt file have to be refreshed (we usually cache the contents for as much as in the future). You possibly can pace up this course of by submitting your up to date robots.txt to Google. Even after discovering the change, crawling and indexing is a sophisticated course of that may typically take fairly a while for particular person URLs, so it is unattainable to provide an actual timeline. Additionally, remember that even when your robots.txt file is disallowing entry to a URL, that URL might stay seen in search outcomes regardless of that indisputable fact that we won’t crawl it. In case you want to expedite removing of the pages you’ve got blocked from Google, submit a removing request.
(Q) How can I briefly droop all crawling of my web site?
(A) You possibly can briefly droop all crawling by returning a 503 (service unavailable) HTTP standing code for all URLs, together with the robots.txt file. The robots.txt file will likely be retried periodically till it may be accessed once more. We don’t advocate altering your robots.txt file to disallow crawling.
(Q) My server just isn’t case-sensitive. How can I disallow crawling of some folders fully?
(A) Guidelines within the robots.txt file are case-sensitive. On this case, it is suggested to guarantee that just one model of the URL is listed utilizing canonicalization strategies. Doing this lets you have fewer traces in your robots.txt file, so it is simpler so that you can handle it. If this is not doable, we really helpful that you just checklist the widespread mixtures of the folder identify, or to shorten it as a lot as doable, utilizing solely the primary few characters as an alternative of the complete identify. As an example, as an alternative of itemizing all higher and lower-case permutations of /MyPrivateFolder, you might checklist the permutations of /MyP (in case you are sure that no different, crawlable URLs exist with these first characters). Alternately, it might make sense to make use of a robots meta tag or X-Robots-Tag HTTP header as an alternative, if crawling just isn’t a difficulty.
(Q) I return 403 Forbidden for all URLs, together with the robots.txt file. Why is the positioning nonetheless being crawled?
(A) The 403 Forbidden HTTP standing code, in addition to different 4xx HTTP standing codes, is interpreted because the robots.txt file would not exist. Which means that crawlers will usually assume that they’ll crawl all URLs of the web site. So as to block crawling of the web site, the robots.txt have to be returned with a 200 OK HTTP standing code, and should comprise an acceptable disallow rule.
(Q) Is the robots meta tag a substitute for the robots.txt file?
(A) No. The robots.txt file controls which pages are accessed. The robots meta tag controls whether or not a web page is listed, however to see this tag the web page must be crawled. If crawling a web page is problematic (for instance, if the web page causes a excessive load on the server), use the robots.txt file. If it’s only a matter of whether or not or not a web page is proven in search outcomes, you should utilize the robots meta tag.
(Q) Can the robots meta tag be used to dam part of a web page from being listed?
(A) No, the robots meta tag is a page-level setting.
(Q) Can I exploit the robots meta tag exterior of a
part?
(A) No, the robots meta tag must be within the
part of a web page.
(Q) Does the robots meta tag disallow crawling?
(A) No. Even when the robots meta tag at the moment says noindex, we’ll must recrawl that URL often to examine if the meta tag has modified.
(Q) How does the nofollow robots meta tag evaluate to the rel=”nofollow” hyperlink attribute?
(A) The nofollow robots meta tag applies to all hyperlinks on a web page. The rel=”nofollow” hyperlink attribute solely applies to particular hyperlinks on a web page. For extra info on the rel=”nofollow” hyperlink attribute, see our documentation on user-generated spam and the rel=”nofollow”.
(Q) How can I examine the X-Robots-Tag for a URL?
(A) A easy solution to view the server headers is to make use of the URL Inspection Instrument characteristic in Google Search Console. To examine the response headers of any URL, attempt looking for “server header checker”.
Google has discontinued the English model of the Robots FAQs (https://t.co/AJoGFD7if8), now redirecting to “Intro”. Variations in different languages stay unaffected at the moment. Personally, I discovered the unique web page to be helpful.
Are we on the trail to leaving robots.txt? #search engine optimization pic.twitter.com/ThgBl9K3Zi
— Alexis Rylko 🇺🇦 (@alekseo) November 22, 2023
We replace the documentation sometimes. Be at liberty to submit suggestions when you really feel one thing’s lacking. Robots.txt is certainly nonetheless a factor.
— I’m John – ⭐ ⭐ SRS BZNS ONLY ⭐ ⭐ (@JohnMu) November 22, 2023
I assume possibly Google thinks it’s redundant from what’s already printed on the opposite pages?
Discussion board dialogue at X.