Duplicate content is the bane of search engines and, as such, they have no problem taking it out on corporate Web sites that refuse to take proper precautions when promoting their online Web copy.

It is not by penalty, but rather by specialized filtering that search engines lower your site's rank for select keywords if measures are not taken to minimize identical pages or gateways that reproduce many versions of single pages, entries and articles.

Setting up a robots.txt file in your root directory is one way to keep duplicate content from being indexed in the search engine indices. When combined with proper instantiation of .htaccess files (read about .htaccess usage and URL canonicalization in my blog archives), you can easily minimize the amount of duplicate content that your Web site or blog may unknowingly produce.

For a more detailed write-up and discussion on why the Robots Protocol is important to your corporate Web site and for ways in which you can take advantage of this bot controlling directive, read the original entry at "Tips for Promoting Only Your Relevant, Informative Content."

Jeffrey Olchovy is a Web developer, designer and marketing strategist.