For StreamAlert and other related platforms, log forwarding is usually left as an exercise to the reader. This work is non-trivial and often requires new infrastructure and code. We wanted to make this easier for everyone and have achieved this through StreamAlert Apps.
Apps allow you to collect logs from popular services and applications in minutes. You simply provide the application’s credentials and StreamAlert will deploy an individual serverless application that will fetch and forward logs to StreamAlert for analysis and alerting.
Apps are made possible through the use of AWS technologies:
An initial deploy of StreamAlert must be performed before Apps can be configured. If you haven’t deployed StreamAlert yet, please visit the Getting Started page to get up and running.
Deploying an App only takes 3 steps:
- Configure the App through the CLI (via
python manage.py app new).
- Enter the required authentication information.
- Deploy the new App and the Rule Processor.
To get help configuring a new App, use:
$ python manage.py app new --help
1. Configure the App¶
The StreamAlert CLI is used to add a new App configuration.
$ python manage.py app new \ --type duo_auth \ --cluster prod \ --name duo_prod_collector \ --interval 'rate(2 hours)' \ --timeout 80 \ --memory 128
||Type of app integration function being configured. Current choices are: duo_auth, duo_admin|
||Applicable cluster this function should be configured against.|
||Unique name to be assigned to the App. This is useful when configuring multiple accounts per service.|
||The interval, defined using a ‘rate’ expression, at which this app integration function should execute. See AWS Schedule Rate Expressions.|
||The AWS Lambda function timeout value, in seconds. This should be an integer between 10 and 300.|
||The AWS Lambda function max memory value, in megabytes. This should be an integer between 128 and 1536.|
Duo Security’s Admin API is limited to two (2) requests per-minute. Therefore, setting the
--timeout flag to any value between 10 and 60 will be of no additional value. A recommended timeout of 80 seconds will guarantee four (4) requests happen per-execution.
2. Enter the required authentication information¶
The above command will result in a few prompts asking for the required authentication information needed to configure this App.
After the last required authentication value is entered, the values are sent to AWS SSM’s Parameter Store as a
SecureString to be used as part of this App’s config. Due to this requirement, please ensure you have the correct and valid AWS credentials loaded before continuing.
Example Prompts for Duo Auth¶
Please supply the API URL for your duosecurity instance. This should be in a format similar to 'api-abcdef12.duosecurity.com': api-abcdef12.duosecurity.com Please supply the secret key for your duosecurity Admin API. This should a string of 40 alphanumeric characters: 123424af2ae101d47d9704b783c940dffa825678 Please supply the integration key for your duosecurity Admin API. This should be in a format similar to 'DIABCDEFGHIJKLMN1234': DIABCDEFGHIJKLMN1234
Once the above is completed, a logger statement similar to the following will confirm the configuration:
StreamAlertCLI [INFO]: App authentication info successfully saved to parameter store. StreamAlertCLI [INFO]: Successfully added 'duo_prod_collector' app integration to 'conf/clusters/prod.json' for service 'duo_auth'.
Your configuration files (
conf/sources.json) have now been updated and are ready to be deployed.
3. Deploy the new App and the Rule Processor¶
The recommended process is to deploy both the apps function and the rule processor function with:
$ python manage.py lambda deploy --processor rule apps