Why robots.txt file is so important?

Posted by Vivek Singh
6
Feb 15, 2017
37 Views
7 people like it
avatar avatar avatar avatar avatar
Comments (23)
avatar
Aaron Habel
3

Writer

It's the first file of the website from where the crawlers enter the website.

Aug 31, 2016 1 Like Like it
avatar
Gaurav P.
6

Internet Marketing Expert

robots.txt file contains some url that you don't want to indexed its about admin folder or other.You can also make a folder not to crawl by this

Aug 31, 2016 2 Likes Like it
avatar
John Mathew
2

Financial Market Expert

It is the roadmap for crawlers to read website pages as per command.

Aug 31, 2016 1 Like Like it
avatar
Please sign in to add comment.