#Hi Gaurav, Its a bad practice to hide information using robot.txt file, actually its used for how to instruct the search engine (web crawlers/spiders) to crawl and index web pages.
The Robots.txt file tells the search engine spiders what not to index. Also make sure the file doesn’t accidentally exclude important files, directories or the entire site (this has been known to happen!).
The robots.txt file is one of the basic parts of a web hosting settings and has a great importance in the basic settings at the level of SEO.
As we have already mentioned the robots.txt is a file that says its extension is a txt file or text file, this is known as the robots exclusion protocol or robots.txt protocol.
Comments (13)
Rishi Kumar
6
SEO Webmaster
#Hi Gaurav, Its a bad practice to hide information using robot.txt file, actually its used for how to instruct the search engine (web crawlers/spiders) to crawl and index web pages.
Gaurav Kumar
12
Sr. SEO Executive
robots.txt file used to hide web pages from search engine crawlers.
Emily Marie
6
marketer
The Robots.txt file tells the search engine spiders what not to index. Also make sure the file doesn’t accidentally exclude important files, directories or the entire site (this has been known to happen!).
Askir.bobker@gmail.c...
2
couture 100% en main
Help me for citybank trensfering my foud to my account bank in moroco
Miguel G.
7
Marketing,sales,advertising consult
The robots.txt file is one of the basic parts of a web hosting settings and has a great importance in the basic settings at the level of SEO.
As we have already mentioned the robots.txt is a file that says its extension is a txt file or text file, this is known as the robots exclusion protocol or robots.txt protocol.
Natilia James
6
Part Time Writer
when you do not want to show any type web page or content from search engine boats. you apply robat.txt for that. according to me.
Rajeev Sharma
3
Travel Services
ya I Agree with preet sharma
Preet Sharma
10
Content Writer
Great answer by Sheryl and Vishwajeet cinnhotels.com
Vishwajeet Kumar
14
Blogger
robot.txt file helps search Engines to determine which page of your site to crawl or not.
Rishi Kumar
6
SEO Webmaster
thanks to all of you.
Actually, It is the Robots Exclusion Protocol to tells the browsers to visiting to the all parts of the websites.
Its a bad practice to hide information using Robot.txt
Sheryl K. Malone
10
Custom Logo Design by Award Winners
Web site owners use the /robots.txt file to give instructions about their site to web crawlers!
Radha Chauhan
8
Search Engine Optimizer
Robot.txt tell the crawler which page you dont want to visit them
HansRaj Deshwal
5
Web Developer and SEO Expert
robots.txt file used to hide web pages from search engine crawlers for protection of duplicate content pages or security pages!!