Datasource Configuration

For background on supported datasource types, read datasources.

Overview

Datasources defined in conf/sources.json control which datasources can send to and be analyzed by StreamAlert.

Each datasource (kinesis, s3, or sns) contains a mapping of specific resource names (kinesis stream names, s3 bucket IDs) along with a list of logs coming from that source.

Log schemas are defined in conf/logs.json

Each log in the list of logs dictates to StreamAlert how to parse incoming data from a given resource. Data will only be analyzed if its type is defined here.

Example:

{
  "kinesis": {
    "abc_corporate_stream_alert_kinesis": {
      "logs": [
        "box",
        "pan"
      ]
    },
    "abc_production_stream_stream_alert_kinesis": {
      "logs": [
        "inspec",
        "osquery"
      ]
    }
  },
  "s3": {
    "abc.webserver.logs": {
      "logs": [
        "nginx"
      ]
    },
    "abc.hids.logs": {
      "logs": [
        "carbonblack"
      ]
    }
  },
  "sns": {
    "abc_sns_topic": {
      "logs": [
        "logstash"
      ]
    }
  }
}

Once datasources are defined, associated logs must have defined schemas