Google webmaster tools?

user-agent: *
Disallow:

I think you get you solution if not then you can read webmaster blog about robots.txt.

According to Wikipedia, Google Webmaster Tools is a no-charge web service by Google for webmasters. It allows webmasters to check indexing status and optimize visibility of their websites. It has tools that let the webmasters:

- Submit and check a sitemap
- Check and set the crawl rate, and view statistics about how Googlebot accesses a particular site
- Generate and check a robots.txt file. It also helps to discover pages that are blocked in robots.txt by chance.
- List internal and external p

To exclude all robots from the entire server follow this :

User-agent: *
Disallow: /

To allow all robots complete access follow this :

User-agent: *
Disallow:

To exclude all robots from any specific folder
USer-agent: *
Disallow: /wp-admin

To exclude all robots from the entire server follow this :

User-agent: *
Disallow: /

To allow all robots complete access follow this :

User-agent: *
Disallow:

informative conversation here.. thanks to all

This tools provides you with detailed reports about your pages' visibility on Google.

what ur website add me @syed Ahmed Kabir Hashmi

they helped me index my website into google search engine!

Its very easy to use robots.txt.
Here is the syntax of that file:
user-agent: *
Disallow:

You can define search bot and url which you want to restrict for indexing.

code for robots.txt

user-agent: *
Disallow:

In robots.txt file the web master gives the instruction to the crawler to crawl your page or not.

in robots.text file, we give the instructions to the crawler about the access of a domain, file,folder,directory etc..