Coupon code copied
Server Intellect
(855) 850-HOST

Toll Free 24/7/365        (4678)

Get help from a Microsoft Certified Engineer.

Intellect Connect

Technical problem with your Windows Hosting?

Let our Microsoft Certified experts handle the problem for you.
Chat Now

Using robots.txt File to Allow or Deny Search Engines

If you have portions of a website that you do not wish for search indexes to see, you can protect them with a “robots.txt” file dictating which search engines are allowed or disallowed from seeing specific folders/files.

There are many options which you can specify in a robots.txt file to explicitly deny or allow specific search-bots to index certain folders or files.

The simplest robots.txt file uses two rules:

  • User-agent: the robot the following rule applies to
  • Disallow: the URL you want to block

These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry.

Please see the following articles which explains how a robots.txt file works, and how to configure one.

http://www.robotstxt.org/

https://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

https://en.wikipedia.org/wiki/Robots_Exclusion_Standard

Interested in letting our experts solve your IT problems for you?
Get a free, no-obligations consultation with one of our experts today!

Call us at (855) 850-HOST

Chat Now

Start Your Order

X

This form does not accept free e-mail accounts. Please enter a business e-mail to submit it.

Submit Form Cancel

Please wait...

Submitting your info. This may take less than a minute.