Writer
It's the first file of the website from where the crawlers enter the website.
Internet Marketing Expert
robots.txt file contains some url that you don't want to indexed its about admin folder or other.You can also make a folder not to crawl by this
Financial Market Expert
It is the roadmap for crawlers to read website pages as per command.
Comments (23)
Aaron Habel
3
Writer
It's the first file of the website from where the crawlers enter the website.
Gaurav P.
6
Internet Marketing Expert
robots.txt file contains some url that you don't want to indexed its about admin folder or other.You can also make a folder not to crawl by this
John Mathew
2
Financial Market Expert
It is the roadmap for crawlers to read website pages as per command.