Configure Robots.txt and Robots Header Tag in Blogger Blog

If you are using Blogger as your bloggnig platform.You might have noticed about two options like Robots.txt and Robots and Custom robots header tag under the Search Preference OPitons on Settings.This part is very important for search engine to crawl and and what no to crawl information are given in such types Robot txt.

Basically there is no difference between Robots.txt and Custom robots header tag both are used for how can search engine Crawl the website and what to not.In robots.txt we are giving command lines but in case of  Robots custom header tag just we want to tick mark the basic details.

Custom Robots.txt

1.Go to Dashboard > Settings > Search Preference > Under Crawling and Indexing
2. Click on edit button just near to custom Robots.txt
3.Type the below code into the space provided 

#robots.txt of this blog/website, thanks to ipeeworld.comUser-agent: Mediapartners-GoogleDisallow:User-agent: *Disallow: /searchDisallow: /p/*Disallow: /view/*Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0Allow: /Sitemap: http://ipeeworld.com/sitemap.xml




 Above given is an example of a SEO friendly Custom Robots.txt, Just i'm going to share what each and every line does for us






User-agent: Mediapartners-GoogleDisallow:


This line  is only for Adsense Crawlers, If you want to show ads on your blog this line make the way for adsense to crawl your blog.If you are not a Adsense publisher it is not required to keep this line you may delete this line




User-agent: *

This is used by all crawlers expect Adsense


Disallow: /search


This line indicates Which line are not to be crawled by Search engine.This line also plays a major role in making your site SEO friendly by not crawling Labels, Because we don't want any labels or achieve to be crawled by Search Engine


Disallow: /p/*

This line is used to stop crawling your blog pages


Disallow: /view/*


Stop's crawling blog's dynamic view links, If you are using Dynamic views just remove this line


Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0


This line prevents from crawling mobile redirected pages. If you don't want them you will see that your blogs link with mobile friendly links


Allow: /


If you use above said line which will restrict certain links on blog, You may add links that are free to crawl 


Sitemap: http://ipeeworldcom/sitemap.xml


It is the Sitemap of your blog, Now replace ipeeworld.com with your preferred  domain URL



Custom robots header tags

You can edit Custom Robots header tag like given below (Also refer the image)
 
Homepage  - all and noodp
 Archive and Search pages - noindex and noodp
Default for Post and pages - all and noodp




About Astro Joseph

Hi, I'm Astro Joseph the founder and CEO of IPEE World and IPEE Network.I started this blog for a passion and it is now empowering millions of readers around the globe.Read more about me at About and Contact page.

One thought on “Configure Robots.txt and Robots Header Tag in Blogger Blog

Leave a Reply

Your email address will not be published. Required fields are marked *