Configure Robots.txt and Robots Header Tag in Blogger Blog

If you are using Blogger as your bloggnig platform.You might have noticed about two options like Robots.txt and Robots and Custom robots header tag under the Search Preference OPitons on Settings.This part is very important for search engine to crawl and and what no to crawl information are given in such types Robot txt.

Basically there is no difference between Robots.txt and Custom robots header tag both are used for how can search engine Crawl the website and what to not.In robots.txt we are giving command lines but in case of  Robots custom header tag just we want to tick mark the basic details.

Custom Robots.txt

1.Go to Dashboard > Settings > Search Preference > Under Crawling and Indexing

2. Click on edit button just near to custom Robots.txt

3.Type the below code into the space provided 

#robots.txt of this blog/website, thanks to ipeeworld.comUser-agent: Mediapartners-GoogleDisallow:User-agent: *Disallow: /searchDisallow: /p/*Disallow: /view/*Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0Allow: /Sitemap:
 Above given is an example of a SEO friendly Custom Robots.txt, Just i'm going to share what each and every line does for us
User-agent: Mediapartners-GoogleDisallow:
This line  is only for Adsense Crawlers, If you want to show ads on your blog this line make the way for adsense to crawl your blog.If you are not a Adsense publisher it is not required to keep this line you may delete this line
User-agent: *
This is used by all crawlers expect Adsense

Disallow: /search

This line indicates Which line are not to be crawled by Search engine.This line also plays a major role in making your site SEO friendly by not crawling Labels, Because we don't want any labels or achieve to be crawled by Search Engine

Disallow: /p/*
This line is used to stop crawling your blog pages

Disallow: /view/*

Stop's crawling blog's dynamic view links, If you are using Dynamic views just remove this line

Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0

This line prevents from crawling mobile redirected pages. If you don't want them you will see that your blogs link with mobile friendly links

Allow: /

If you use above said line which will restrict certain links on blog, You may add links that are free to crawl 

Sitemap: http://ipeeworldcom/sitemap.xml

It is the Sitemap of your blog, Now replace with your preferred  domain URL

Custom robots header tags

You can edit Custom Robots header tag like given below (Also refer the image)
Homepage  - all and noodp
 Archive and Search pages - noindex and noodp
Default for Post and pages - all and noodp

Hi Guys, I'm Astro Joseph a young Blogger from Kerala,India.I Started IPEE World as passion and it is now empowering Millions of Readers around the Globe.Read more at About Page

Sharing is Caring :)

Related Articles

Previous Article
Next Article »


25 January 2016 at 01:39 delete

Also read how to configure robot.txt at


We’re very excited to see your comments. However, Please Keep in mind that, All Comments are strictly moderated by us before publishing.Comments with Links and other unnecessary contents will be removed and will not be published. :)

Contact Form


Email *

Message *