- 1 Setup instructions for all robots
- 2 Allow all but block specific crawler.
- 3 Block Images indexing:
- 4 Add sitemap:
- 5 Default Sitemap for Blogspot Blog if you don’t know how to setup :
- 6 Share this:
- 7 Like this:
- 8 Related
Google now lets you edit your robot.txt file to blogger. What are Robots.txt files, robots files are plain text files which instruct the search engines what to not index.
Follow the steps :
1) Log-in to your blogger account.
2) Go to your blog.
3) Now go to “Settings>Search Preferences”
4) Under the “Crawlers and indexing” you can find “Custom robots.txt”
5) Click “Edit” link.
Now you can add different Crawler to instructions. This post I added different useful crawl robots.txt codes. First we identify how this code works.
User-agent: – Mention crawler name. For instance, Google bot.
Disallow: – Specify which pages should not crawl.
Allow: – Which pages crawl.
/ (slash):- indicate your home page.
Setup instructions for all robots
Block label and search pages crawling.
If you use above code, it will cause duplicate content issues and your site rank will be reduced. So that, you can use the following code. It will allow entire blog but, not allow crawl label and search pages.
Block certain page (s).
Some reasons, you many need to hide your selected page or pages from the search engines. At that time you can use the following code.
If you need to block more-than one page add their URL one by one on Disallow section like below.
Allow all but block specific crawler.
If you want to block single crawler, you can add the following code.
Instruction on how to setup crawler of adsense :
Block Images indexing:
If you need to block any-other bot crawl your site, use following code. However you need to add selected bot name in “User-agent:” section.
User-agent: <required crawler name>
You can find more crawlers and their user agent information in here and in here.
If you need to improve your blog’s search engine visibility and crawl entire blog other than labels and search pages using the following code as your robots.txt.
Sitemap: <paste your blog sitemap here>
Default Sitemap for Blogspot Blog if you don’t know how to setup :