Clusters¶
Background¶
StreamAlert can deploy separate infrastructure for each cluster
(or environment) you define.
What constitutes a cluster
is up to you.
Example: You could define IT
, PCI
or Production
clusters.
Strategy¶
Common patterns:
- Define a single cluster to receive and process data from all of your environments
- Define a cluster for each of your environments
- Define a cluster for each organization, which may have one or more environments
Which one you choose is largely dependent on your processes, requirements and how your team organizes itself.
Option (2) is encouraged because it provides segmentation for ingestion and analysis, which allows for the best possible performance during processing.
Configuration¶
Each cluster file lives in its own JSON
file in the conf/clusters
directory, and contains a name, region, modules, and Terraform variable outputs.
Examples¶
Kinesis Cluster¶
Contains the following:
- Rule and Alert Processor Lambda Functions
- Kinesis Stream with 1 Shard
- Kinesis Events to the Rule Processor
- CloudWatch Monitoring Alarms for Kinesis and Lambda
- Outputs to display the Kinesis username, access key, and secret key
{
"id": "kinesis-example",
"modules": {
"cloudwatch_monitoring": {
"enabled": true
},
"kinesis": {
"streams": {
"create_user": true,
"shards": 1,
"retention": 24
}
},
"kinesis_events": {
"enabled": true
},
"stream_alert": {
"alert_processor": {
"timeout": 25,
"memory": 128,
"current_version": "$LATEST"
},
"rule_processor": {
"timeout": 10,
"memory": 256,
"current_version": "$LATEST"
}
}
},
"outputs": {
"kinesis": [
"username",
"access_key_id",
"secret_key"
]
},
"region": "us-west-2",
}
CloudTrail S3 Processing Cluster¶
Contains the following:
- Rule and Alert Processor Lambda Functions
- CloudWatch Monitoring for Lambda
- CloudTrail with Kinesis disabled
- S3 Event Notifications setup on multiple S3 buckets
{
"id": "s3-example",
"modules": {
"cloudtrail": {
"enable_kinesis": false,
"enable_logging": true
},
"cloudwatch_monitoring": {
"enabled": true,
"kinesis_alarms_enabled": false
},
"s3_events": [
{
"bucket_id": "example.s3.streamalert.cloudtrail",
"enable_events": true
}
],
"stream_alert": {
"alert_processor": {
"current_version": "$LATEST",
"enable_metrics": false,
"log_level": "info",
"memory": 128,
"timeout": 10
},
"rule_processor": {
"current_version": "$LATEST",
"enable_metrics": false,
"log_level": "info",
"memory": 128,
"timeout": 10
}
}
},
"region": "us-east-1"
}
Customizing Clusters¶
Each StreamAlert cluster is made up of multiple modules.
Each module corresponds to a Terraform module found in the terraform/modules
directory, and serves a specific purpose in a StreamAlert cluster.
After making modifications to a cluster file, make sure you apply the changes with:
$ python manage.py terraform build
This will regenerate the necessary Terraform files and then apply the changes.
Module: StreamAlert¶
The main module for StreamAlert.
It creates both AWS Lambda functions, aliases, an SNS topic, IAM permissions, and more.
See Lambda Settings for all customization options.
Module: Kinesis¶
This module contains configuration for the Kinesis Streams and Kinesis Firehose infrastructure.
See Kinesis for all customization options.
Module: CloudWatch Monitoring¶
Amazon CloudWatch is a monitoring service for AWS cloud resources.
To ensure a StreamAlert cluster is running properly, this module creates metric based alarms across all component services. This ensures that ingesting, processing, and storage of data is operating normally.
If any of the services cross a predefined threshold, an alarm is generated.
To disable CloudWatch alarms, set to false
.
Template:
{
"cloudwatch_monitoring": {
"enabled": true
}
}
To configure the SNS topic used to receive CloudWatch metric alarms, use one of the following options in the conf/global.json
configuration file.
Option 1: Create a new topic. This tells the StreamAlert CLI to create a new topic called stream_alert_monitoring
. All clusters will send alarms to this topic.
{
"account": {
"...": "..."
},
"terraform": {
"...": "..."
},
"infrastructure": {
"monitoring": {
"create_sns_topic": true
}
}
}
Option 2: Use an existing SNS topic within your AWS account (created outside of the scope of StreamAlert).
{
"account": {
"...": "..."
},
"terraform": {
"...": "..."
},
"infrastructure": {
"monitoring": {
"sns_topic_name": "my_sns_topic"
}
}
}
Module: Kinesis Events¶
The Kinesis Events module connects a Lambda function to a Kinesis Stream.
By default, this connects the stream_alert
module to the kinesis
module.
To disable this mapping, set to false
Template:
{
"kinesis_events": {
"enabled": true
}
}
Module: CloudTrail¶
AWS CloudTrail is a service that enables compliance, operational auditing, and risk auditing of your AWS account.
StreamAlert has native support for enabling and monitoring CloudTrail logs with the cloudtrail
module.
When writing rules for CloudTrail data, use the cloudwatch:event
log source.
By default, all API calls will be logged and accessible from rules.
Template:
{
"cloudtrail": {
"enable_logging": true,
"enable_kinesis": true
}
}
Options:
Key | Required | Default | Description |
enable_logging |
Yes |
Enable/disable the CloudTrail logging. | |
enable_kinesis |
No |
true |
Enable/disable the sending CloudTrail data to Kinesis. |
existing_trail |
No |
false |
Set to true if the account has an existing CloudTrail. This is to avoid duplication of data collected by multiple CloudTrails. |
is_global_trail |
No |
true |
If the CloudTrail should collect events from any region. |
event_pattern |
No |
{"account": ["<accound_id>"]} |
The CloudWatch Events pattern to send to Kinesis. More information. |
cross_account_ids |
No |
Account IDs to grant write access to the created CloudTrail S3 bucket |
Module: Flow Logs¶
VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your AWS VPC.
In the settings below, an arbitrary amount of subnets, vpcs, and enis can be enabled.
When writing rules for this data, use the cloudwatch:flow_logs
log source.
Template:
{
"flow_logs": {
"enabled": true,
"log_group_name": "<name-of-cloudwatch-log-group>",
"subnets": [
"subnet-id-1",
"..."
],
"vpcs": [
"vpc-id-1",
"..."
],
"enis": [
"eni-id-1",
"..."
]
}
}
Options:
Key | Required | Default | Description |
enabled |
Yes | To enable/disable the Flow log creation. | |
log_group_name |
No | prefix_cluster_streamalert_flow_logs | The name of the CloudWatch Log group. |
subnets |
No | None | The list of AWS VPC subnet IDs to collect flow logs from. |
vpcs |
No | None | The list of AWS VPC IDs to collect flow logs from. |
enis |
No | None | The list of AWS ENIs to collect flow logs from. |
Module: S3 Events¶
Amazon S3 is one of the default datasources for StreamAlert.
S3 Event Notifications can be configured to notify Lambda each time an object is written.
When StreamAlert receives this notification, it fetches the object from S3 and analyzes it according to configured rules.
Template
{
"s3_events": [
{
"bucket_id": "<bucket-id>"
},
{
"bucket_id": "<bucket-id-2>",
"enable_events": false
}
]
}
Options:
Key | Required | Default | Description |
bucket_id |
Yes | The S3 bucket to notify upon | |
enable_events |
No | Yes | Enable/disable the notification to Lambda |