It totally depends on the type of cms you are using! Use * i.e for all bots and hide the pages which you don't want to let the crawler to crawl Also additionally you can put the robots.txt file on the root directory of the domain.
It depends! First of all how are you working your web page with a CMS, a framework or just html.
Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
For best SEO I recommendation for robots.txt to make a perfect SEO.
If you want every crawler crawled your website then set
User-agent: *
Disallow: /admin
It depends! First of all how are you working your web page with a CMS, a framework or just html.
Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
web design, development and digital marketing comp
1.Place your robots.txt file in the top-level directory of your website code to simplify crawling and indexing.
2.Structure your robots.txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap. This way, search engine spiders access categories and web pages in the appropriate order.
3.Make sure that every URL you want to “Allow:” or “Disallow:” is placed on an individual line. If several URLs appear on one single line, crawlers will have a problem accessing them.
4
Comments (17)
Nitish Chandra Sharm...
7
Digital Marketing Expert
robot file basic code, save with robots.txt and upload it on the root directory of the domain -
User-agent: *
Allow: /
John A.
1
Owner at Organic Beds in Toronto
It totally depends on the type of cms you are using! Use * i.e for all bots and hide the pages which you don't want to let the crawler to crawl Also additionally you can put the robots.txt file on the root directory of the domain.
Modernday Music
2
Modern Day Music School
User-agent: *
Allow: /
Mobiloitte Technolog...
6
Marketing Manager
Use * i.e for all bots and hide the pages which you don't want to let the crawler to crawl
Parahombre USA
2
Business
It depends! First of all how are you working your web page with a CMS, a framework or just html.
Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
Jack Dolson
1
SEO
Disallow all the unwanted bots in robots.
Wonders Mind
3
Web Designing Company
Hi,
Robots file will help you in block certen pages from search engines.
Think Tribe
3
Think..Solve...Execute
Take a .txt file and decide which page do you want to get follow and indexed by search engine and which page do not want.
And write the below :
User-agent: *
Allow: / which page you want to get follow and index by search engine.
Disallow: / which page do not want to get follow and index.
Leave Allow: / It will follow and index all page.
Leave Disallow: / It will not follow and index any page.
And upload this robots.txt file in index page on your web server.
Thank you
Murtza Abbas
7
Sr. Digital Marketing | SEO | SMO
User-agent: *
Allow: /
Ganesh Kulariya
5
Content Writer, SEO, SMO Expert
For best SEO I recommendation for robots.txt to make a perfect SEO.
If you want every crawler crawled your website then set
User-agent: *
Disallow: /admin
Enterslice ITES Pvt....
8
Start and Manage Business
Syntex for Robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /goto/
Sitemap: https://***/sitemap_index.xml
Sitemap: https://***/post-sitemap.xml
Ahmad Quershi
3
Ahmad Quershi
it depends on what cms are you using and then you need to disallow such content you need to block like
Disallow: //.pdf
JONATHAN PAUL
10
Jonathan Paul working at PHPProgrammers, a leading
user-agent:* Allow: /
Rob Stephen
14
getaprogrammer
User-agent: *
Allow: /
Shriya World
1
IT
Hello,
If you are using wordpress then below is the reference sample sitemap
User-agent: *
Disallow: /wp-admin/
Disallow: /xmlrpc.php
Disallow: /trackback/
Disallow: */trackback
Disallow: /cgi-bin/
Disallow: /wp-login/
Disallow: /wp-register/
Allow: /goto/
Allow: /wp-content/uploads/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://***/sitemap_index.xml
Sitemap: https://***/post-sitemap.xml
1. User-agent: * Represent allowing all search engine bots to crawl your website.
if you want only
Joaquin Velazquez
3
Marketing
It depends! First of all how are you working your web page with a CMS, a framework or just html.
Depending on this you must make the configuration of your robots file, restricting access to folders that you do not want to be indexed, pointing to the sitemap.xml and caching the necessary files to save loading time.
Starboard Technology...
4
web design, development and digital marketing comp
1.Place your robots.txt file in the top-level directory of your website code to simplify crawling and indexing.
2.Structure your robots.txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap. This way, search engine spiders access categories and web pages in the appropriate order.
3.Make sure that every URL you want to “Allow:” or “Disallow:” is placed on an individual line. If several URLs appear on one single line, crawlers will have a problem accessing them.
4