John Murphy is a Security Researcher at FlowTraq. His research has included attack graph analysis, system performance baselining, and a game-theoretic analysis of the Conficker worm, and he has extensive experience and multiple publications in the field of network behavior-based insider threat detection. John has a PhD in engineering from Dartmouth College, and engineering degrees from West Virginia University.
Denial of service attacks have become an all-too-common headache in modern network administration: whether providing local internet to home users, serving up necessary resources to your customers, or just trying to get your mission accomplished, eventually (and persistently) someone out there will think it’s a good idea to knock you or your users offline. There are solutions, such as FlowTraq, available on the market to help determine very quickly when you’re under attack, what asset is under attack, and the nature of the attack; and for getting quick notification… but what then?
Many network operators utilize on-site scrubbing appliances that will remove attack traffic and make sure you’re only moving the bits that you want to. They’re great for protecting your downstream customers, both the ones being attacked and the innocent bystanders trying to use the same pipe. The trouble is, eventually a large attack will overwhelm your own connection to the outside world, and you can’t afford to let that malicious traffic into your network at all, even just long enough to scrub it.
In such a case, many companies have turned to BGP null-route techniques: telling upstream providers to stop routing traffic to the victimized host. That solves the immediate bandwidth crisis, but rewards the attackers, who have succeeded in knocking their victim offline.
To combat the malicious traffic, companies have selected to deploy a hybrid DDoS mitigation solution. This involves utilizing off-premise scrubbing centers with a cloud-based DDoS provider to mitigate the DDoS traffic as an attractive alternative to black holing the attack traffic. The DDoS provider can be automatically alerted when the traffic volume is threatening to overwhelm your network. They will receive the incident details: which IPs are under attack, what kind of attack it is, when it started, and other information that helps their team quickly start mitigation on the malicious traffic and only send back the healthy traffic to your network. The nice thing about having that service in the wings is that it doesn’t have to be utilized for every attack: just for the traffic you can’t or don’t want to handle yourself. In addition, having an off-site DDoS mitigation available adds another valuable tool to your toolbox.
To learn more about how FlowTraq provides the operational awareness needed to manage and deploy a hybrid DDoS mitigation solution, download the solution brief: Escalating DDoS activity effectively.
NetFlow (and associated technologies like IPFIX, J-Flow, Netstream, etc.) acts as a kind of live-reporting phone bill for your network: up-to-the-minute information about the communications taking place. This includes information such as who’s sending how much data to whom, as well as how and when: IP addresses, port and protocol, exporting device, and timestamps — plus VLAN, TCP flags, etc.
This data is widely available from devices like routers, switches, firewalls, load-balancers, hypervisors – and even as software to install on individual hosts. With data streaming in from multiple sources, a central location can get an excellent view of the network, including cross-border and purely internal traffic.
When collected in a central location and made available to view and search, NetFlow gives a network engineer visibility into what’s going over the network, including large-scale events like DDoS attacks and saturated links, and small-scale events like SSH logins and database connections. This allows the analyst to monitor for attacks, policy compliance, even data usage for billing. Everything from brute-force attacks to on-the-sly Netflix watching: if it crosses the network, it’s in the NetFlow.
Because NetFlow is a summary, it can be stored compactly — a gigabyte of NetFlow data can describe hundreds to thousands of gigabytes of actual traffic. This efficiency makes it suitable for full-fidelity recall, meaning that every single record can be stored for later analysis, down to the smallest ICMP ping. With a full-fidelity NetFlow history, it becomes possible to perform intricate analysis of one’s historical record, down to the level of the order of small transactions that occurred months ago. This enables an analyst to perform difficult tasks like tracking an intruder through multiple hops of SSH sessions, or separating potential DNS tunnels from ordinary requests. It also means that the analyst doesn’t have to be psychic: they don’t need to know today what they’ll need to search for tomorrow.
More than that intricate analysis, a fast full-fidelity NetFlow system gives an analyst an unparalleled feel for the network. Simply clicking around and exploring for a half hour can give a person a far better idea of what to expect in the traffic than days in the lecture hall. It is that intuitive feel for the network, made possible by mere exploration, that ultimately gives an analyst the knowledge and understanding needed to truly secure a computer network.
Sony was hit by hackers on November 24, resulting in a company wide computer shutdown and corporate information leaks, including employee social security numbers and executive salary information. A group calling itself Guardians of Peace took credit for the attacks. (As reported by Brooke Barnes and Nicole Perlroth of the NY Tiimes,)
John Murphy PhD, senior security researcher at FlowTraq offers his comments about the recent Sony Pictures cyber attacks, specifically how FlowTraq may have helped in a situation like this.
Question: FlowTraq can see who is sending data, and just as importantly, who is consuming data from the network. By applying filters or using Network Behavior Intelligence features, FlowTraq can identify possible threats with fewer false alarms. Would that have helped Sony or others like them?
Murphy: Without knowing the details of the attack, we can’t say for sure whether Sony would have been protected. No network monitoring can detect SneakerNet — a bad guy with a USB stick who walks the data out the front door. But a lot of attacks like this are detectable.
There are two stages to any network-based attack, unless it was an inside job. First, the initial intrusion, which at some point along the chain is going to be a successful login from a new address. The second stage is the exfiltration. For the first stage, FlowTraq has a number of detectors capable of monitoring for and alerting on incoming traffic that looks like a login attempt like SSH or Telnet. Detecting at this stage lets you nip an attack in the bud, and gives you valuable information about your attacker’s intent and resources.
FlowTraq has a built-in workspace for detecting data exfiltration, making use of our ability to tag “Internal” addresses for fast searching. It may be the very same network session as the initial login: you need to be able to see those single sessions. What’s sent is sent – nothing on the planet can put that genie back in the bottle — but the faster you see it, the faster you can stop it, and the less you lose. Even if you completely miss it at the time, it’s still better to know that you’ve lost data, and know where you lost it from.
A company should never learn about its own data breach on the news: there’s no excuse for that. As for the inside job with the USB key, you can still see people connecting to internal resources they have no business using. It’s harder to see, but if you can catch it before that employee leaves for the day, you’ve saved yourself a lot of trouble.
Question: FlowTraq can watch threshholds of data volumes, alerting on unusual volumes. Bytes, sessions, countries. What did Sony miss?
Murphy: Again, it’s hard to comment on this particular event without knowing the exact nature of the attack. But what we do know suggests a pattern of access to multiple internal resources. Unless Sony is in the habit of sending data to these attackers, somewhere along the chain of custody these must have been an unusual connection: more data, a longer connection, a new address or set of addresses. Probably some combination of all three.
When you open up a workspace in the new version of FlowTraq, go into your top Hosts or top Service Endpoint view or any other address-based view, and click on one of the IP addresses that comes up in the ranking table. Down at the bottom of the menu will be a new item, “Customize this menu…” that could change the way you troubleshoot your network.
The page that comes up will be mostly empty at first. It has a button at the top labeled “Add Custom Menu Item” which does exactly what it says: it adds a menu item… but only to selected menus.
There are two parts to each menu item: a name (left) and a GET request URL. The URL can contain any of a handful of pieces of information that might be clicked on to bring up a context menu: %IP%, %PORT%, %ASN% for IP address, port number, or autonomous system. All items have two additional pieces of information as well: the earliest time and latest time in the selected view (%TE% and %TL% respectively).
For example, it is common to find an IP address of interest on the FlowTraq Threats page and check the traffic related to it. A custom menu item enables the reverse workflow: “This host’s behavior seems unusual to me; let’s see if any alerts were raised on it.” The URL template,
will be automatically filled in with the host of interest and the time frame you’re currently viewing.
These items combine to form a remarkable range of queries for both public services and local software. Here are some suggestions for simple but powerful URL templates, using public services, to start you off:
SANS Internet Storm Center
IP address details:
Autonomous System report:
AlienVault Reputation Monitor:
SpamHaus Blocklist Lookup:
Project Honeypot IP Address Inspector:
In addition to publicly available security tools, internal tools can frequently also be accessed by URL. As one example, a local Splunk install can be automatically searched, not only for a selected IP but also in the same time range as the current FlowTraq workspace:
These are just a few possible applications of this feature; there are many more possibilities available, using public tools, private ones, and custom web pages. We’re always interested in hearing about new uses for our software — if you find something interesting, let us know!
This document focuses on configuring Juniper J Series and SRX Series devices for J-Flow v9, which is based on the RFC3954 (IPFIX) flow export standard (via UDP) and as such is consumable by any IPFIX-capable flow collector, including FlowTraq. J-Flow v9 exporting requires Junos OS 10.4 or later; this document is based on Junos 12.1. The following procedure provides an example of the J-Flow configuration for version 9:
The following commands correspond to nested templates, so that
is equivalent to defining the structure,
The first step in configuring the J-Flow v9 export is to set the template:
The next step is to configure sampling. Sampling rate is the number of packets (X:1) to sample. Junos allows the user to specifically select additional samples to follow a sampled packet, the run-length. So, if rate is 10 and run-length is 3, then the flow output would be based on 4-packet sequences such that the overall sampling rate is 10:4 — out of every 10 packets, 4 would be selected, and those 4 would be sequential. For full-fidelity recording, then, rate is set to 1, and run-length to 0.
NOTE: J-Flow v9 is based on a sampled-packet system. Flows are built using only a portion of the observed network traffic, according to sampling rules. When sampling is enabled, J-Flow should not be treated as a full-fidelity flow stream — individual flows may be missing from the record, or have incorrect packet and byte counts and session lengths. This enables a Juniper device to monitor more traffic without impacting performance, but once information is lost due to sampling selection it cannot be recovered. Nevertheless, this analysis comes at a computational cost which must be considered when
Next, configure the external flow collector (here, 10.0.0.100) and its UDP port address (the default NetFlow port is used here, as a convenience) and attach the previously defined flow template.
Note that this is equivalent to defining, in two commands, the structure:
(For a full description of the flow-server command, see Juniper’s documentation)
Configure the inline-jflow, so that the sampling and the J-Flow service thread are implemented in the forwarding engine. The source address determines the address to use for generating monitored packets, and will appear as the Exporter IP address.
Configure the sampling filter on an interface (or interfaces) in the direction to be monitored. “input” corresponds to Ingress traffic on other devices, and “output” to Egress. Per common convention, monitoring all Ingress interfaces covers all traversing packets exactly once.
At this point, your Juniper device should be exporting J-Flow datagrams to your flow collector. J-Flow v9 is a template-based format (like NetFlow 9) and it may take 5-15 minutes for the first flows to appear. If traffic fails to arrive at your collector, there are a few things to check:
First, make sure that your J-Flow collector is listening on the correct port (UDP 2055 above) and that any firewalls in between (particularly on the host running the collector) allow the J-Flow packets to pass.
Second, verify the flow of session records using a packet capture utility such as Wireshark or TCPdump. (J-Flow datagrams will appear as CFLOW in Wireshark) Verify the destination IP address and port. If they are correct, Log back into your Juniper device and verify that the correct interfaces are being monitored.
If none of the above troubleshooting methods yielded an obvious error, contact your vendor’s support if J-Flows are not being sent. If they are being sent but not received, contact FlowTraq technical support.
No security tool can be used in a vacuum. Context is key to any investigation, to determine what is ordinary and what is of interest. Security analysts rely on a suite of tools to generate security information and to investigate suspicious incidents. Among these tools, Security Information and Event Managers (SIEMs) provide a higher-level centralized repository for events coming from a variety of sources: firewalls, application logs, intrusion detection systems, network behavioral intelligence analysis, etc.
Cisco’s Flexible NetFlow technology is a powerful but sometimes complicated way to customize your flow collection. Here are some tips how to configure Cisco routers for NetFlow export. It can take a little bit of time to understand and set up, but is well worth the effort.
For simple pre-defined NetFlow output, most Cisco devices offer a fast and straight-forward monitoring solution. The specific commands required depend on your version of IOS, however: This document is for Cisco IOS Releases 12.2(14)S, 12.0(22)S, 12.2(15)T, or later. Many of these devices offer the option of Flexible NetFlow configuration instead, using the ip flow command to add a configured flow monitor.
For simple pre-defined NetFlow output, most Cisco devices offer a fast and straight-forward monitoring solution. The specific commands required depend on your version of IOS, however:
You know they exist, and you can picture them: They’re on shop floors with grease-covered CRTs and keyboards. On the counters of mom-and-pop grocery stores. On the desks of tenured professors who still resent having had to upgrade from Windows 95. In a wiring closet somewhere, probably, though you only know it’s there because you can see NetFlow sessions from it and it still responds to pings. All throughout Asia, South America, and much of the rest of the world. And judging by Netcraft’s Web Server Survey, more than a few active servers in your own environment and in those of services your users access and receive email from.