Bot Management

Bot Management enables you to mitigate undesired bot traffic from your site using CAPTCHA and JavaScript detection tools, while enabling known published bot providers to bypass these controls.

Non-human traffic makes up most of the traffic to sites. Bot Manager is designed to detect and block, or otherwise direct, non-human traffic that may interfere with site operations. The Bot Manager features mitigate bots that conduct content and price scraping, vulnerability scanning, comment spam, brute force attacks, and application-layer DDoS attacks. You can also allowlist good bots.

Warning

When you enable Bot Management, you incur a higher rate on requests to the WAF.

JavaScript Challenge

JavaScript Challenge validates that the client can accept JavaScript with a binary decision. JavaScript Challenge is generally the first level of bot mitigation, but not sufficient with more advanced bot tools, which require more advanced challenges. Additional functionality, like detecting Network Address Translation (NAT) traffic, can mitigate the risk of blocking legitimate user traffic from users behind a shared IP address.

The Action Threshold parameter defines the number of requests that fail the challenge before the action is taken. The requests that fail under this threshold are not logged. For example, if you set the JavaScript challenge action to Block and the Action Threshold to 10, and a client that doesn't accept JavaScript makes 11 requests within the Action Expire Time, the first 10 requests will be allowed through to origin (assuming there are no other rules) and logs will show one Block entry action taken for the JavaScript Challenge.

Human Interaction Challenge

Human Interaction Challenge is an advanced countermeasure that looks for natural human interactions such as mouse movements, time on site, and page scrolling to identify bots. When an EDGE server receives requests from a client, instead of instantly reporting with the requested content, the human interaction challenge checks various event listeners in the user's browser to determine if there is a human user making a request.

Device Fingerprint Challenge

The device fingerprint challenge generates hashed signatures of both virtual and real browsers to identify and block malicious bots.

CAPTCHA Challenge

If a specific URL should be accessed only by a human, you can control it with CAPTCHA protection. You can customize the comments for the CAPTCHA Challenge for each URL. Bots are kept from accessing protected web application functionality using CAPTCHA images designed to be out of reach of computer vision and OCR technologies.

Good Bot Whitelist

Good Bots provides the list of bots managed by known providers, such as Baidu or Google. You can allow the access from a specific good bot, or block the bot if they serve no business purpose. Allowed good bots from this section are allowlisted.

Allowlisted bots are flagged with a Bypass action in the WAF policy Logs. You can select the Bypass check box from the Action filter in Logs to search for the traffic allowed from these rules. Logged good bot events are categorized as a Threat Intelligence Leads log type, however, they are not a threat when the action taken is to Bypass.

The list of good bots on this menu are managed and continuously updated. Additional good bots can be added as a new access control rule in Access Control.

Using the Console

To configure JavaScript Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure JavaScript Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click Edit JavaScript  Challenge.
  5. In the JavaScript Challenge dialog box, select the Enable JavaScript Challenge check box.
  6. In the JavaScript Challenge Action section, choose one of the following methods:

    • Detect Only: Select this option if you want to be alerted for every matched request.
    • Block: Select this option to block requests by returning a response code, error page, or CAPTCHA.
      • Block Action: Select the action that will be taken when a matching request is blocked.
        • Show CAPTCHA
          • CAPTCHA Title: Enter the text for the CAPTCHA page title.
          • CAPTCHA Header: Enter the text that will appear before the CAPTCHA image (for example, "I am not a robot").
          • CAPTCHA Footer Text: Enter the text that will be shown after the CAPTCHA input box and before the submit button.
          • CAPTCHA submit button: Enter the text for the Submit button (for example, "Yes, I am human.").
        • Set Response Code: Select a status code to return in response to blocked requests.
        • Show Error Page
          • Block Error Page Message: Defines the error or error code.
          • Block Error Page Description: Provides more details about the error, including the cause and further instructions.
          • Block Error Page Code: The error code that is displayed with the error.
  7. Enter the following information:

    • Enable Conditions: When enabled, conditions must match for a set action to be taken. See Access Control for more information about conditions and rules.
    • Action Threshold (number of requests): Specify the number of failed requests before taking action. Due to the asynchronous request from the browser during page loading, it is recommended to set a threshold of 10 for web applications with basic ajax usage, and 100 for apps with heavy ajax usage.
    • Action Expire Time (seconds): Enter the number of seconds between challenges to the same IP address. Due to client IP address changes, it is recommended that the expiry time is set to 120 seconds for apps with mobile users and 3600 seconds for apps with desktop users only.
    • Follow Redirects: When enabled, redirect responses from the origin will also be challenged.
    • Enable NAT Support: When enabled, the user is identified not only by the IP address but also by a unique additional hash, which prevents blocking visitors with shared IP addresses. It is recommended that this NAT support is disabled for high-load apps (200+RPS).
  8. Click Save Changes.

The JavaScript Challenge is added to the list of changes to be published.

To edit JavaScript Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to edit JavaScript Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click Edit JavaScript Challenge.
  5. In the Edit JavaScript Challenge dialog box, make the needed changes.
  6. Click Save.
To configure Human Interaction Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure JavaScript Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the Human Interaction Challenge tab.
  5. Click Edit Human Interaction Challenge.
  6. In the Edit Human Interaction Challenge dialog box, select the Enable Human Interaction Challenge check box.
  7. In the Human Interaction Action section, choose one of the following methods:
    • Detect Only: Select this option if you want to be alerted for every matched request.
    • Block: Select this option to block requests by returning a response code, error page, or CAPTCHA.
      • Block Action: Select the action that will be taken when a matching request is blocked.
        • Show CAPTCHA
          • CAPTCHA Title: Enter the text for the CAPTCHA page title.
          • CAPTCHA Header: Enter the text that will appear before the CAPTCHA image (for example, "I am not a robot").
          • CAPTCHA Footer Text: Enter the text that will be shown after the CAPTCHA input box and before the submit button.
          • CAPTCHA submit button: Enter the text for the Submit button (for example, "Yes, I am human.").
        • Set Response Code: Select a status code to return in response to blocked requests.
        • Show Error Page
          • Block Error Page Message: Defines the error or error code.
          • Block Error Page Description: Provides more details about the error, including the cause and further instructions.
          • Block Error Page Code: The error code that is displayed with the error.
  8. Enter the following information:

    • Action Threshold (number of requests): Specify the number of failed requests before taking action. Due to the asynchronous request from the browser during page loading, it is recommended to set a threshold of 10 for web applications with basic ajax usage, and 100 for apps with heavy ajax usage.
    • Threshold Expiry Period (seconds):The number of seconds before the threshold expires.
    • Action Expire Time (seconds): Enter the number of seconds between challenges to the same IP address. Due to client IP address changes, it is recommended that the expiry time is set to 120 seconds for apps with mobile users and 3600 seconds for apps with desktop users only.
    • Interaction Threshold (number of interactions): Number of interactions before the threshold expires.
    • Recording Period (seconds): The period of time to record the user's events.
    • NAT Support: When enabled, the user is identified not only by the IP address but also by an unique additional hash, which prevents blocking visitors with shared IP addresses. It's recommended to disable the support for the high-load apps (200+ RPS).
  9. Click Save Changes.

The Human Interaction Challenge is added to the list of changes to be published.

To edit Human Interaction Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure Human Interaction Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the Human Interaction Challenge tab.
  5. Click Edit Human Interaction Challenge.
  6. Update the Human Interaction Challenge and then click Save Changes.
To configure Device Fingerprint Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure Device Fingerprint Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the Device Fingerprint Challenge tab.
  5. Click Edit Device Fingerprint Challenge.
  6. In the Device Fingerprint Challenge dialog box, select the Enable Device Fingerprint Challenge check box.
  7. In the Device Fingerprint Action section, choose one of the following methods:
    • Detect Only: Select this option if you want to be alerted for every matched request.
    • Block: Select this option to block requests by returning a response code, error page, or CAPTCHA.
      • Block Action: Select the action that will be taken when a matching request is blocked.
        • Show CAPTCHA
          • CAPTCHA Title: Enter the text for the CAPTCHA page title.
          • CAPTCHA Header: Enter the text that will appear before the CAPTCHA image (for example, "I am not a robot").
          • CAPTCHA Footer Text: Enter the text that will be shown after the CAPTCHA input box and before the submit button.
          • CAPTCHA submit button: Enter the text for the Submit button (for example, "Yes, I am human.").
        • Set Response Code: Select a status code to return in response to blocked requests.
        • Show Error Page
          • Block Error Page Message: Defines the error or error code.
          • Block Error Page Description: Provides more details about the error, including the cause and further instructions.
          • Block Error Page Code: The error code that is displayed with the error.
  8. Enter the following information:

    • Action Threshold (number of requests): Specify the number of failed requests before taking action. Due to the asynchronous request from the browser during page loading, it is recommended to set a threshold of 10 for web applications with basic ajax usage, and 100 for apps with heavy ajax usage.
    • Threshold Expiry Period (seconds):The number of seconds before the threshold expires.
    • Action Expire Time (seconds): Enter the number of seconds between challenges to the same IP address. Due to client IP address changes, it is recommended that the expiry time is set to 120 seconds for apps with mobile users and 3600 seconds for apps with desktop users only.
    • Max Address Count (IP addresses): The maximum number of IP addresses that are added to the list before the specified action is taken.
    • Max Address Count Expiration (seconds): The number of seconds an IP address is kept in the list before it is removed.
  9. Click Save Changes.
To edit Device Fingerprint Challenge settings
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure JavaScript Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the Device Fingerprint Challenge tab.
  5. Click Edit Device Fingerprint Challenge.
  6. Update the Device Fingerprint Challenge and then click Save Changes.
To add a CAPTCHA Challenge
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to edit CAPTCHA challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the CAPTCHA Challenge tab.
  5. Click Add CAPTCHA Challenge.
  6. In the Add CAPTCHA Challenge dialog box, enter the following information:
    • CAPTCHA Title: Enter the text for the CAPTCHA page title.
    • CAPTCHA URL Path: Enter the URL path challenged by CAPTCHA.
    • Session Duration: Enter the number of seconds after which the CAPTCHA challenge cannot be resubmitted to the same user.
    • CAPTCHA Header: Enter the text that will appear before the CAPTCHA image (for example, "I am not a robot").
    • Footer Text: Enter the text that will be shown after the CAPTCHA input box and before the submit button.
    • Incorrect CAPTCHA Text: Enter the text that will appear when incorrect text is entered (for example, "The CAPTCHA was incorrect. Please try again.").
    • Submit button: Enter the text for the Submit button (for example, "Yes, I am human.").
  7. Click Preview CAPTCHA to preview the CAPTCHA challenge in a new tab.
  8. Click Add.
To edit a CAPTCHA Challenge
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to edit CAPTCHA Challenge settings for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the CAPTCHA Challenge tab.
  5. Select the check box for the CAPTCHA you want to edit.
  6. Select Edit from the Actions drop down menu.
  7. Update the CAPTCHA Challenge and then click Save.
To delete a CAPTCHA Challenge
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to delete CAPTCHA Challenge settings for. The WAF Policy overview appears.

  3. Click Bot Management.
  4. Click the CAPTCHA Challenge tab.
  5. Select the check box for the CAPTCHA Challenge you want to delete.
  6. Click Delete.
  7. In the Confirm dialog box, click Delete.
To manage the Good Bot Whitelist
  1. Open the navigation menu. Under Governance and Administration, go to Security and click Web Application Firewall.
  2. Click the name of the WAF Policy you want to configure Bot Management for. The WAF Policy overview appears.
  3. Click Bot Management.
  4. Click the Good Bot Whitelist tab.

  5. Select each bot you want to designate as a good bot.

The designated good bots are added to the list of changes to be published.
To publish changes

Updates to your WAF policy appear in the list to be published in Unpublished Changes. Pending changes do not persist across browser sessions. Once you publish changes, it cannot be edited until changes propagate to the edge nodes.

  1. Under WAF Policy, click Unpublished Changes.
  2. In the Unpublished Changes list, click the drop-down arrow beside an unpublished change to review the change.
  3. Click Publish All.
  4. In the Publish Changes dialog box, click Publish All.
To discard changes

Updates to your WAF policy appear in the list to be published in Unpublished Changes.

  1. Under WAF Policy, click Unpublished Changes.
  2. In the Unpublished Changes list, click the drop-down arrow beside an unpublished change to review the change.
  3. Select the check box for the change you want to discard.
  4. Click Discard.
  5. In the Discard Change dialog box, click Discard.

Using the CLI

You can use the CLI to enable rate limiting, device fingerprinting, and human interaction challenges.

To enable rate limiting

Open a command prompt and run the following command to enable rate limiting:

oci waas address-rate-limiting update-waf --is-enabled true  --allowed-rate-per-address 1 --max-delayed-count-per-address 2 --waas-policy-id <policy_ocid>

This default rate limit setting will allow one request per second before starting to delay. It will delay for two requests until the traffic falls within the threshold boundaries. It will use the default error response code of 503.

To enable device fingerprinting to detect

Open a command prompt and run the following command to enable device fingerprinting to detect:

oci waas device-fingerprint-challenge update --is-enabled true --action DETECT --failure-threshold 2 --action-expiration-in-seconds 240 --failure-threshold-expiration-in-seconds 600 --max-address-count 2 --max-address-count-expiration-in-seconds 255 --waas-policy-id <policy_ocid>
To enable the human interaction challenge to detect

Open a command prompt and run the following command to enable the human interaction challenge to detect:

oci waas human-interaction-challenge update --is-enabled true  --waas-policy-id <policy_ocid>