Block a robot from your site

Your instance is configured by default to disable robots from crawling your site. However, if you choose to use the Google Site Search engine, your site will be crawled by the Google robots unless you specifically configure it not to do so.

Before you begin

Role required: admin

About this task

The Robots.txt Definition module allows you to define the contents of the robots.txt file associated with the instance. The syntax used is the same as for a standard robots.txt file. For more information on how to format the robots.txt file, see Robots.txt Specifications.

Procedure

  1. Go to Custom Search Integration > Robots.txt Definitions.
    Robots files
  2. Click New.
  3. Enter the contents of the robots.txt file in the Text field.
    If you want this file to be active, select the Active field. There can only be one robots.txt file active at any time. Setting a file to active automatically sets all other files to inactive. Web crawlers will recognize the contents of the active robots.txt file and honor the robots exclusion protocol.
    Robots file
  4. Click Submit.