Search Marketing and Internet Marketing Blog Online

Robots.txt Generator

A few months ago I wrote a blurb about robots.txt - what it is and how to use it. Google Webmaster Central just released their own Robots.txt Generator. To access the tool, log-in to your Google Webmaster Tools account, then click on the Tools menu option on the left-hand side of the screen after you select one of your verified sites. You'll see a "Generate robots.txt" link among the tool options. That's what you want.

The tool lets you craft specific rules for these particular Google crawlers:

• Googlebot
• Googlebot-Mobile
• Googlebot-Image
• Mediapartners-Google
• Adsbot-Google

After you make your file, upload it to the root directory of your web site. If you don't know what that is, find someone who does! This is important!

For more about Google's webmaster tools, be sure to check out the quick start guide they offer.