Datasource Configuration

For background on supported datasource types, read datasources.

Overview

Datasources defined in conf/sources.json control which datasources can send to and be analyzed by StreamAlert.

Each datasource (kinesis, s3, or sns) contains a mapping of specific datasource names (kinesis stream names, s3 bucket IDs) along with a list of logs coming from that source.

Log schemas are defined in conf/logs.json

Each log in the list of logs dictates to StreamAlert how to parse incoming data from that entity. Data will only be analyzed if its type is defined here.

Example:

{
  "kinesis": {
    "abc_corporate_stream_alert_kinesis": {
      "logs": [
        "box",
        "pan"
      ]
    },
    "abc_production_stream_stream_alert_kinesis": {
      "logs": [
        "inspec",
        "osquery"
      ]
    }
  },
  "s3": {
    "abc.webserver.logs": {
      "logs": [
        "nginx"
      ]
    },
    "abc.hids.logs": {
      "logs": [
        "carbonblack"
      ]
    }
  },
  "sns": {
    "abc_sns_topic": {
      "logs": [
        "logstash"
      ]
    }
  }
}

Once datasources are defined, associated logs must have defined schemas