What is Custom Robot.txt? How to enabled Robot.txt in Blogger Blog? - FindersAdda : Tutorials of Make Money, Computer or Internet

Monday, 14 August 2017

What is Custom Robot.txt? How to enabled Robot.txt in Blogger Blog?


Today, I am going to share what is Robot.txt and what is used of Robot.txt and How to enabled Robot.txt in blogger blog.


how to add custom robot text in blogger blog

What Is Robot.txt?

Robot.txt is a file which is used to hide some information from Search engine like Google, Bing, Yandex etc. It is available in all post. If you want to hide any information then you can hide by Robot.txt from search engine.



The main advantage of Robot.txt file is that to restrict web crawler to index your pages or some URLs. You can restrict to index your some pages or links from web crawlers like labels, demo pages and some other pages or links. Search engine always scan blog robot.txt file before searching in your blog.

User-agent: Mediapartners-GoogleDisallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.findersadda.com/sitemap.xml.


Copy the above code and replace your domain name instead of findersadda.

The above code is divided into three sections or parts. I will define every part one by one in this post.

User-agent: Mediapartners-Google

This code is for Google AdSense robots which help them to serve better ads on your blog. Either you are using Google AdSense on your blog or not simply leave it as it is.

User-agent:*


This is for all robots marked with asterisk (*). In default settings our blog’s labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

Disallow: /search


This code ignores labels or search to index in search engine. You can see the below links. In below links, only domain name will be indexed.


http://www.findersadda.com/search/label/bloggers


When we remove Disallow:/search from the code then web crawler will crawl or index our labels or entire blog.

Allow:/ refers to index our homepage that means allow code give permission to web crawler to index our homepage in search engine.


Sitemap: http://www.findersadda.com/sitemap.xml


This code helps search engine to index our blog post easily that means this code keeps all links of our blog post and pages at one place.


How to add Custom Robot.txt in Blogger Blog?



Step 1: First of all, go to blogger plate form

Step 2: Now, click on the Setting then click on the Search Preferences option


click on the setting and then search preferences



Step 3: Then in the crawl and indexing part,  tick the option yes in front of Custom robot.txt

Step 4: Then paste the first code of this website which was you copied.

paste the code in the box


Step 5: Now, click on the Save option


How to check your robot.txt File?


You will be checked that your robot.txt file has added to your blog or not.


check robot.txt file is added or not


Write your blog URL in address bar and after blog URL,  write /robots.txt you can see this link below.

http://www.findersadda.com/robots.txt

Final Words


In this article, I have written that how to add custom robot.txt in blogger blog. I have tried to make it simple, so that all people could understand. If you have any doubt related to it then you can ask from me.And I hope you liked this post so much. If you loved this post, so, share with your friends and also give some suggestion that what should i improve in this post. Thank You So Much...... 

About Author

SUMIT KUMAR GUPTA
SUMIT KUMAR GUPTA

Author & Editor

Myself Sumit Kumar Gupta & 21 year's old person. I am a programmer and blogger. I spend much more time on programming and blogging and helps other programmers and bloggers.

0 comments:

Post a Comment

We are eager to see your comments.
Comment Rule :

1. Don't use any link in the comment box. If you have any problem related to link then contact me with the contact form.

2. Don't use the wrong word if you write the wrong word in comment box then it will get spam.