If you have portions of a website that you do not wish for search indexes to see, you can protect them with a “robots.txt” file dictating which search engines are allowed or disallowed from seeing specific folders/files.
There are many options which you can specify in a robots.txt file to explicitly deny or allow specific search-bots to index certain folders or files.
The simplest robots.txt file uses two rules:
- User-agent: the robot the following rule applies to
- Disallow: the URL you want to block
These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry.
Please see the following articles which explains how a robots.txt file works, and how to configure one.