Creating a robots .txt file

The robots.txt file contains directives, created by you, that spiders are programmed to obey based on The Robots Exclusion Protocol. These directives instruct the visiting spider which part or parts of your website should or should not be indexed.

Search engines like Google, Yahoo or MSN crawl the internet by sending out their spidering software. These crawlers are commonly known as spiders. The spiders visit websites all around the internet to include them in their index listings. The first thing they look for when they visit your website is a file called the robots.txt file. This file is found in the root directory of your website.

Not all spiders are "good spiders"

Some spiders and robots such as malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers do not follow standard protocol (the directives of your robots.txt file). These are "Bad Bots". These bad bots prowl the Internet looking for things like email addresses and other information that they can use to make your life miserable.

The robots.txt file is used to deny the "Good" search engine spiders access to restricted areas of your website. The files that you do not want indexed such as downloads, images or a cgi-bin directory.

How to create a robots.txt file.

Creating a robots.txt file is not complicated. You can create a robots.txt file using a simple text editing program such as NotePad.

To create a robots.txt file that allows all robots to crawl and index your entire website. Open notepad and type in this directive:
User-agent: * Disallow:

The asterisk * indicates all search engines. An empty (blank) robots.txt file or no robots.txt file is the same as giving all search engines permission to crawl your site.

To block all robots from your website type in this directive:
User-agent: * Disallow: /

To block certain files:
User-agent: *
Disallow: /images
Disallow: /cgi-bin
Click file-save, type in robots and save as a .txt file. Upload the file to the root directory of your website.