{"id":17838,"date":"2022-04-22T00:54:00","date_gmt":"2022-04-21T21:54:00","guid":{"rendered":"https:\/\/adsterra.com\/blog\/?p=17838"},"modified":"2023-07-26T10:45:16","modified_gmt":"2023-07-26T07:45:16","slug":"what-is-robots-txt","status":"publish","type":"post","link":"https:\/\/blogdev.adsterratech.com\/blog\/what-is-robots-txt\/","title":{"rendered":"What Is Robots.txt: Why Do You Need It and How to Create One?"},"content":{"rendered":"\n

Today’s topic is not directly related to traffic monetization. But robots.txt can impact your website’s SEO and, eventually, the amount of traffic it receives. Many web admins have ruined their websites’ rankings due to botched robots.txt entries. This guide will help you avoid all of those pitfalls. Make sure to read to the end!<\/strong><\/p>\n\n\n\n

<\/div>\n\n\n\n\n\n
<\/div>\n\n\n\r\n
\r\n\tToo long? Ask AI to summarize<\/strong>\r\n\t
\r\n\t\t\r\n ChatGPT\r\n\t\t<\/a>\r\n\r\n\t\t\r\n Perplexity\r\n\t\t<\/a>\r\n\r\n\t\t\r\n\t\t\tClaude\r\n\t\t<\/a>\r\n\t<\/div>\r\n<\/div>\n

What is a robots.txt file?<\/strong><\/h2>\n\n\n\n

The robots.txt, or robot exclusion protocol, is a set of web standards that controls how search engine robots crawl every web page, down to the schema markups on that page. It\u2019s a standard text file that can even prevent web crawlers from gaining access to your whole website or parts of it.<\/p>\n\n\n\n

<\/div>\n\n\n\n

While adjusting SEO and solving technical issues, you can start getting passive income from ads. A single line of code on your website returns regular payouts!<\/em><\/p>\n\n\n\n

\n
EARN FROM ADS<\/a><\/div>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n

How does a robots.txt file look?<\/strong><\/h2>\n\n\n\n

The syntax is simple: you give bots rules by specifying their user-agent and directives. The file has the following basic format:<\/p>\n\n\n\n

Sitemap: [URL location of sitemap]<\/p>\n\n\n\n

User-agent: [bot identifier]<\/p>\n\n\n\n

[directive 1]<\/p>\n\n\n\n

[directive 2]<\/p>\n\n\n\n

[directive …]<\/p>\n\n\n\n

<\/div>\n\n\n\n

User-agent: [another bot identifier]<\/p>\n\n\n\n

[directive 1]<\/p>\n\n\n\n

[directive 2]<\/p>\n\n\n\n

[directive …]<\/p>\n\n\n\n

<\/div>\n\n\n\n

How to find your robots.txt file<\/strong><\/h2>\n\n\n\n

If your website already has a robot.txt file, you can find it by going to this URL: https:\/\/yourdomainname.com\/robots.txt <\/strong>in your browser. For example, here\u2019s our file<\/p>\n\n\n

\n
\"robot-txt-file\"<\/figure><\/div>\n\n\n
<\/div>\n\n\n\n

How does a Robots.txt file work?<\/strong><\/h2>\n\n\n\n

A robots.txt file is a plain text file that does not contain any HTML markup code (hence the .txt extension). This file, like all other files on the website, is stored on the web server. Users are unlikely to visit this page because it’s not linked to any of your pages, but most web crawler bots search for it before crawling the entire website.<\/p>\n\n\n\n

A robots.txt file can give bots instructions but can’t enforce those instructions. A good bot, such as a web crawler or a news feed bot, will check the file and follow the instructions before visiting any domain page. But malicious bots will either ignore or process the file to find forbidden web pages.<\/p>\n\n\n\n

In a situation when a robots.txt file contains conflicting commands, the bot will use the most specific set of instructions.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Robots.txt syntax<\/strong><\/h2>\n\n\n\n

A robots.txt file consists of several sections of ‘directives,’ each starting with a user-agent. The user-agent specifies the crawl bot with which the code communicates. You can either address all search engines at once or manage individual search engines.<\/p>\n\n\n\n

Whenever a bot crawls a website, it acts on the parts of the site that are calling it.<\/p>\n\n\n\n

User-agent: *<\/p>\n\n\n\n

Disallow: \/<\/p>\n\n\n\n

User-agent: Googlebot<\/p>\n\n\n\n

Disallow:<\/p>\n\n\n\n

User-agent: Bingbot<\/p>\n\n\n\n

Disallow: \/not-for-bing\/<\/p>\n\n\n\n

<\/div>\n\n\n\n

Supported directives<\/strong><\/h3>\n\n\n\n

Directives are guidelines that you want the user-agents you declare to follow. Google currently supports the following directives<\/a>.<\/p>\n\n\n\n

<\/div>\n\n\n\n

User-agent*<\/strong><\/h4>\n\n\n\n

When a program connects to a web server (a robot or a regular web browser), it sends an HTTP header called “user-agent” containing basic information about its identity. Every search engine has a user-agent. Google’s robots are known as Googlebot, Yahoo’s \u2014 as Slurp, and Bing’s \u2014 as BingBot. The user-agent initiates a sequence of directives, which can apply to specific user-agents or all user-agents.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Allow<\/strong><\/h4>\n\n\n\n

The allow directive tells search engines to crawl a page or subdirectory, even a restricted directory. For example, if you want search engines to be unable to access all of your blog’s posts except one, your robots.txt file might look like this:<\/p>\n\n\n\n

User-agent: *<\/p>\n\n\n\n

Disallow: \/blog<\/p>\n\n\n\n

Allow: \/blog\/allowed-post<\/p>\n\n\n\n

However, search engines can access \/blog\/allowed-post but they are unable to gain access to:<\/p>\n\n\n\n

\/blog\/another-post<\/p>\n\n\n\n

\/blog\/yet-another-post<\/p>\n\n\n\n

\/blog\/download-me.pd<\/p>\n\n\n\n

<\/div>\n\n\n\n

Disallow<\/strong><\/h4>\n\n\n\n

The disallow directive (which is added to a website’s robots.txt file) tells search engines not to crawl a specific page. In most cases, this will also prevent a page from appearing in search results.<\/p>\n\n\n\n

You can use this directive to instruct search engines not to crawl files and pages in a specific folder you\u2019re hiding from the general public. For example, content that you are still working on but mistakenly published.  Your robots.txt file might look like this if you want to prevent all search engines from accessing your blog:<\/p>\n\n\n\n

User-agent: *<\/p>\n\n\n\n

Disallow: \/blog<\/p>\n\n\n\n

This means all the subdirectories of the \/blog directory would also not be crawled. This would also block Google from accessing URLs containing \/blog.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Sitemap<\/strong><\/h4>\n\n\n\n

Sitemaps are a list of pages you want search engines to crawl and index. If you use the sitemap directive, search engines will know the location of your XML sitemap. The best option is to submit them to the search engines’ webmaster tools because each can provide valuable information about your website for visitors.<\/p>\n\n\n\n

It’s important to note that repeating the sitemap directive for each user-agent is unnecessary, and it doesn’t apply to one search-agent. Add your sitemap directives at the beginning or end of your robots.txt file.<\/p>\n\n\n\n

An example of a sitemap directive in the file:<\/p>\n\n\n\n

Sitemap: https:\/\/www.domain.com\/sitemap.xml<\/p>\n\n\n\n

User-agent: Googlebot<\/p>\n\n\n\n

Disallow: \/blog\/<\/p>\n\n\n\n

Allow: \/blog\/post-title\/<\/p>\n\n\n\n

User-agent: Bingbot<\/p>\n\n\n\n

Disallow: \/services\/<\/p>\n\n\n\n

<\/div>\n\n\n\n

Unsupported directives<\/strong><\/h3>\n\n\n\n

The following are directives that Google no longer supports<\/strong> \u2014 some of which were technically never endorsed.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Crawl-delay<\/strong><\/h4>\n\n\n\n

Yahoo, Bing, and Yandex rapidly respond to websites’ indexing and react to the crawl-delay directive, which keeps them in check for a while.<\/p>\n\n\n\n

Apply this line to your block:<\/p>\n\n\n\n

User-agent: Bingbot<\/p>\n\n\n\n

Crawl-delay: 10<\/p>\n\n\n\n

It means that the search engines can wait for ten seconds before crawling the website or ten seconds before re-accessing the website after crawling, which is the same thing but slightly different depending on the user-agent in use.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Noindex<\/strong><\/h4>\n\n\n\n

The noindex meta tag is a great way to prevent search engines from indexing one of your pages. The tag allows bots to access the web pages, but it also informs robots not to index them.<\/p>\n\n\n\n