Rule Staging

Rule Staging allows dynamic toggling of rules from a staging to production state, and vice versa. Staged rules will only send alerts for historical retention, and the alerts will not be sent to any user-defined outputs, such as Slack, PagerDuty, etc.

To ensure that new rules do not flood alerting outputs with a ton of potential false positives, rules can be staged. After the initial staging period, wherein a noisy rule is tuned to limit extra noise or false positives, staged rules can be promoted.

When Rule Staging is enabled, new rules will, by default, be staged upon a deploy of the Rule Processor Lambda function. See the Skip Staging During Deploy section for more information.

Enabling Rule Staging

By default, the rule staging feature is not enabled. It can be enabled with the following command:

python manage.py rule-staging enable --true

The change will be reflected in the conf/globals.json file:

conf/global.json
{
  "infrastructure": {
    "rule_staging": {
      "cache_refresh_minutes": 10,
      "enabled": true,
      "table": {
        "read_capacity": 20,
        "write_capacity": 5
      }
    }
  }
}

Configuration Options

A few configuration options are available to customize the feature to your needs.

Key Default Description
cache_refresh_minutes 10 Maximum amount of time (in minutes) the Rule Processor should wait to force refresh the rule staging information.
table.read_capacity 20 DynamoDB read capacity to allocate to the table that stores staging information. The default setting should be sufficient in most use cases.
table.write_capacity 5 DynamoDB write capacity to allocate to the table that stores staging information. The default setting should be sufficient in most use cases.

The initial implementation of the Rule Staging feature has a hard-coded ‘staging period’, or the time a rule should remain in staging before being auto-promoted to send to user-defined outputs. This is only relevant if the Rule Promotion feature is also enabled. The current default is 48 hours.

CLI Commands

Rule Status

The status of rules, meaning whether or not they are staged, can be determined with the following command:

python manage.py rule-staging status

Sample Output:

Rule                                     Staged?
  1: rule_001                            False
  2: rule_002                            True
  3: rule_003                            False
  4: rule_004                            False

Toggling Staged Status

Staging a rule, or list of rules, is possible with the following command:

python manage.py rule-staging stage <rule_001> <rule_002>

Unstaging rules, enabling them to send to all user-defined outputs, is equally as easy and accomplished with the following command:

python manage.py rule-staging unstage <rule_001> <rule_002>

Skip Staging During Deploy

As noted above, all new rules will be staged by default during a Rule Processor deploy when the Rule Staging feature is enabled. There may, however, be occasions when all new rules should not be staged during a deploy. To allow for this, the Rule Processor can be deployed with the following command:

python manage.py lambda deploy -p rule --skip-rule-staging

This will force all new rules to send to user-defined outputs immediately upon deploy, bypassing the default staging period. Alternatively, the --stage-rules and --unstage-rules flags can be used (instead of the --skip-rule-staging flag) to stage or unstage specific rules only.

Triaging Staged Rules

Once a rule is in staging, alerts generated by that rule can be queried in Athena:

SELECT 'rule_001' as rule_name, count(*) AS alert_count FROM alerts WHERE dt >= '2018-07-25-16' AND rule_name = 'rule_001' AND staged = True
Athena Results
rule_name alert_count
rule_001 96

To help automate triaging of staged rules, StreamAlert includes an optional Rule Promotion Lambda function. This function can both send alert digests via email and auto-promote rules out of staging. See the Rule Promotion page for more detail.