Data collection and discovery using VPC Flow Logs

Service Mapping can perform discovery based on data collected using VPC Flow Logs. Amazon VPC hosts Amazon Elastic Compute Cloud (EC2) instances that provide Amazon Web Services. VPC Flow Logs collect data on IP traffic going to and from network interfaces in the VPC.

Using VPC Flow Logs for collecting data is one of the traffic-based discovery methods. Other methods deployed by Service Mapping are using netstat and lsof commands and the Netflow protocol. For more information, refer to Traffic-based discovery in Service Mapping.

In base systems, traffic-based discovery uses only TCP-related data collected with the help of the netstat and lsof commands. Discovery based on Netflow and VPC logs requires additional configuration. You can enrich your traffic-based discovery by configuring Service Mapping to use VPC Flow Logs.

Service Mapping discovery based on VPC Flow logs has the following flow:
  1. Amazon EC2 instances collect their individual logs into log streams and forward them to the central flow log group.
  2. The MID Server collects the data from the flow log and processes it.
  3. The MID Server places the processed information onto the ECC queue.

  4. A sensor retrieves the processes data from the ECC queue and writes it into the Flow Connection [sa_flow_connection] table:
  5. Whenever Service Mapping checks the ECC queue and receives information on a discovered CI, it checks these tables for any data on outbound connections related to the CI: the cmdb_tcp and sa_flow_connection tables. If these two tables contain unique data that patterns did not discover, Service Mapping enriches the information about the CI connections and adds them to the map.