FlowTraq > Articles by: Alex Barsamian

Author: Alex Barsamian

Getting Started with AWS VPC flow logs in FlowTraq

Alex Barsamian
By | June 10, 2016


We make pretty extensive use of Amazon Web Services at FlowTraq. We host a significant percentage of both our public cloud and private cloud offerings in EC2, and prospective customers are delighted to hear that we will support them with optimized configurations, best practices, and more if they want to run their own FlowTraq instance in Amazon’s data centers.

Until recently, however, network visibility in AWS was a bit of a blind spot. But last year, AWS offered the ability to generate CloudWatch logs of network flows to and from your EC2 hosts.

Unfortunately, the logs are not in IPFIX format, the de-facto standard of modern network flow reporting, or any other common format, but a proprietary format. So feeding AWS “netflow” directly into an off-the-shelf flow analyzer is a non-starter.

Fortunately, AWS also offers a clever feature called Lambda, which lets you run a snippet of code in response to an event (such as the generation of a CloudWatch log). The code runs on the AWS infrastructure, but not on a particular EC2 host, and you have your choice of a JavaScript, Java, or Python runtime environment. You pay by the request and by the millisecond.

So we can architect a fairly simple solution: write a Lambda script that responds to the generation of a VPC Flow log and puts the log information on the wire, addressed to a destination IP and TCP port of our choosing. On that destination we run a listener that grabs the information, repackages/converts it to IPFIX, and forwards it to FlowTraq for analysis.

AWS VPC flow logs into FlowTraq

As far as we know, what follows is the best and easiest way to get AWS EC2 CloudWatch logs converted to IPFIX for NetFlow analysis, in FlowTraq or otherwise!


If you haven’t already, set up an AWS CloudWatch Flow log IAM role and a log stream for the virtual interface you want to monitor, per the AWS VPC Flow Logs User Guide.

The Lambda script

The Lambda script is pretty simple:

var zlib = require('zlib');
exports.handler = function(event, context) {
    var payload = new Buffer(, 'base64');
    zlib.gunzip(payload, function(e, result) {
        if (e) {
        } else {
            var host = '';
            var port = 20555;

            var message = new Buffer(result.toString('utf8'));

            var net = require('net');

            var client = new net.Socket();
            client.connect(port, host, function() {
                client.write(message, function() {

There are two configuration variables to consider, namely the IP and destination port to send the logs. Substitute the IP of the host you plan to run the listener on (which may be, but need not be, the same as your FlowTraq server, but in any event must have a public IP) and a port that your firewall will let external IPs connect to.

Create a Lambda function and paste that bit of JavaScript into it.

The listener

As they’re emitted by our Lambda script, there’s some JSON giftwrap around the log lines, which themselves look like this:

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

On the other hand, FlowTraq’s IPFIX exporter expects inputs as follows:

client ip, port, server ip, port, protocol, client packets, client bytes, server packets, server bytes, start time, end time, exporter ip, application name

Python’s good for many things, but it’s great for 1) creating a lightweight TCP server, 2) parsing JSON, and 3) manipulating strings.

These three tasks turn out to be the bulk of what’s required of a listener, and we can accomplish our goal in just a few dozen lines of code.

import json
import socket



def printLogEvents(jsonFormat):
    data = json.loads(jsonFormat)
    logEvents = data['logEvents']
    for logEvent in logEvents:
        message = logEvent['message']
        messageSplit = message.split()

        # rearranage the fields from the VPC's log format to ftsq's in format, setting the <- bytes/packets to 0, the application "unknown"
        # and the exporter IP to the IP configured above
        ftsqFormat = ""
        ftsqFormat = ftsqFormat + messageSplit[3] + ","
        ftsqFormat = ftsqFormat + messageSplit[5] + ","
        ftsqFormat = ftsqFormat + messageSplit[4] + ","
        ftsqFormat = ftsqFormat + messageSplit[6] + ","
        ftsqFormat = ftsqFormat + messageSplit[7] + ","
        ftsqFormat = ftsqFormat + messageSplit[8] + ","
        ftsqFormat = ftsqFormat + messageSplit[9] + ","
        ftsqFormat = ftsqFormat + "0" + ","
        ftsqFormat = ftsqFormat + "0" + ","
        ftsqFormat = ftsqFormat + messageSplit[10] + ","
        ftsqFormat = ftsqFormat + messageSplit[11] + ","
        ftsqFormat = ftsqFormat + EXPORT_AS_IP + ","
        ftsqFormat = ftsqFormat + "\"unknown\""
        print ftsqFormat

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)



while True:
    connection, client_address = sock.accept()

    jsonFormat = ""

        while True:
            chunk = connection.recv(1024)
            if chunk:
                jsonFormat = jsonFormat + chunk

This time there are three configuration variables to consider. As before, an IP and TCP port to listen on, which ought to line up with the Lambda script. In addition, you must specify the IP to use as the exporter IP, which is how the flows will appear in FlowTraq. If you want to monitor multiple EC2 hosts or VPCs, you should use different exporter IPs to distinguish them.

Putting it all together

Set up the VPC flow log stream and the lambda script, and subscribe the lambda script to the flow log event source. Then on your listener host, invoke the listener script, and pipe the output to the "ftsq" command. The parameters to put ftsq in exporter mode are:

./ftsq -read -ipfix collector-ip port


./ftsq -read -ipfix 4739

Therefore, use the long-lived command

python | ./ftsq -read -ipfix 4739

To get the VPC Flow logs into FlowTraq.

The next time CloudWatch generates a batch of flow log files, you should see a new exporter appear in FlowTraq with those flows. Et, voilà!

Screen Shot 2016-06-09 at 6.52.40 PM


Alex Barsamian
By | November 21, 2013


Using FlowTraq, NH SAU70 Achieves a Perfect Balance Between Streaming Needs and Internet Abuses

This NH school unit firmly believes in enhancing education by giving faculty and students access to Internet resources but also needed to ensure movies and other copyrighted material are not illegally downloaded and shared. This requirement was particularly challenging because of the school’s policy of allowing end users to access the wireless network with their own devices including smartphones, tablets and laptops. (more…)

Georgian College

Alex Barsamian
By |


Using FlowTraq, Georgian College Quickly Identifies and Remediates Network Performance and Security Issues

GeorgianCollege_logoEstablished in 1967 and based in Ontario, Canada with seven campuses, Georgian College has consistently generated one of the highest graduate employment rates among Ontario colleges formore than a decade. Student enrollment averages more than 10,000 fulltime students per year and includes close to 500 international students from 33 countries. The college employs approximately 750 full-time and 1,500 part-time staff. (more…)

FlowTraq Full Fidelity Recall [tutorial]

Alex Barsamian
By |



FlowTraq stores all your network flows with nanosecond precision. You can go back and filter on any traffic, and perform a full forensic investigation into network breaches and failures. This video demonstrates the use of the full fidelity flow recall.


Ready to experience FlowTraq for yourself?

Request a product demonstration or start your free trial now! Your security will never be the same.


FlowTraq Dashboard and Workspaces [tutorial]

Alex Barsamian
By |


FlowTraq offers fully customizable dashboards and workspaces, so you can configure the software to show you what’s most important to your job. In this video, Dr. Vince Berk demonstrates the concepts of the FlowTraq user Dashboard.

Ready to experience FlowTraq for yourself?


Request a product demonstration or start your free trial now! Your security will never be the same.


FlowTraq with Free Flow Exporter [tutorial]

Alex Barsamian
By |


A tutorial video on a FlowTraq deployment with our free software, Flow Exporter.


Ready to experience FlowTraq for yourself?

Request a product demonstration or start your free trial now! Your security will never be the same.


FlowTraq Deployment Scenarios [tutorial]

Alex Barsamian
By |


Flow monitoring, forensic and security deployments can be made at three different levels of visibility. This tutorial by Dr. Vince Berk describes the iPoP/border, switch layer, and full deployment scenerios monitoring individual hosts. Each level increases visibility, as well as storage requirements.

Ready to experience FlowTraq for yourself?

Request a product demonstration or start your free trial now! Your security will never be the same.


Scalable NetFlow Analysis [tutorial]

Alex Barsamian
By |


Flow analysis, monitoring, and security remains scalable into the future, while full packet captures will not. This tutorial gives a brief overview why. (more…)

FlowTraq Q4/13 Released

Alex Barsamian
By | November 13, 2013


FlowTraq Q4/13 Released & Ready for Download

The FlowTraq team is proud to announce the availability of the Q4/13 release of FlowTraq.  This release provides new features aimed at improving your ability to perform network analysis to secure and manage complex networks: (more…)

Whitepaper: “Understanding NetFlow, Monitoring, and Security”

Alex Barsamian
By | October 21, 2013


Learn how to combine NetFlow with analytical tools for unparalleled control over your network.

Download the latest FlowTraq whitepaper, Understanding Netflow, Monitoring and Network Security to learn how to use NetFlow to detect data exfiltration and Denial of Service (DDoS) attacks. You’ll also discover how NetFlow is used for remediation and full history attack investigation, and how Network Behavioral Intelligence (NBAD) can be used to protect your network data.

The latest whitepaper in the FlowTraq Desktop Reference Series, Understanding NetFlow Analysis Monitoring, Security, and Forensics will cover:

  • Introduction to NetFlow
  • Network Auditing and Accurate Billing
  • Scalable Deployment: From Micro to Mega Big Data
  • Insight into Virtual Networks, Virtual Machines, and Mobile Devices
  • Network Forecasting and Planning
  • Data Exfiltration
  • Full History Attack Investigation
  • Detection and Remediation
  • Network Behavioral Intelligence

Download this white paper

Next Page »