Detecting Data Staging & Exfil Using the Producer-Consumer Ratio

In their FloCon 2014 presentation PCR - A New Flow Metric, Carter Bullard and John Gerth introduced the idea of the Producer-Consumer Ratio (PCR) for measuring and tracking shifts in the typical pattern of network communication for each host. PCR is calculated on a per-host basis, like this:

(bytes sent - bytes recvd)

-----------------------------

(bytes sent + bytes recvd)

This is an interesting metric, because it gives a good indication of the traffic pattern yet ignores many details that tend to complicate understanding, such as the actual volume of data sent or received, the number of flows, the amount of packets, etc. It boils everything down to one simple number in the range [-1.0,1.0]. They provided the following chart to give a rough idea how to interpret the PCR values:

PCR	host role
1.0	pure push - FTP upload, multicast, beaconing
0.4	70:30 export - Sending Email
0.0	Balanced Exchange - NTP, ARP probe
-0.5	3:1 import - HTTP Browsing
-1.0	pure pull - HTTP Download

The idea is that you can track the PCR for each host over time and look for shifts to identify significant changes that might indicate possible data exfiltration. I recently came across this technique as I was reviewing contributions to The ThreatHunting Project and though it sounded like a fun thing to play around with. I decided to give it a try on some test data I had laying around, just to see how it would work.

The Hypothesis

By comparing a host's baseline PCR to it's current PCR and looking for large shifts, we should be able to identify hosts that are exfiltrating the data to the Internet. By extension, we should also be able to use the PCR shift to identify central staging points for data, where threat actors gather it in preparation for exfiltration.

The Data

The data I used comes from a test lab which features a realistic corporate Windows environment in small scale (about 40 hosts) as well as a simulated user population that does things like access file servers, send/receive email, browse the web, etc. There's also a simulated Internet. 

In this lab, we monitor our network with Bro, so I used Bro's connection logs (conn.log files) as my data source. The exact format of these files doesn't really matter here, and you can easily adapt this to any flow data you happen to have (argus, SANCP, etc).

I should also point out that in this attack scenario, the same host was used for both data staging and data exfil. This isn't much of a problem when calculating PCR, since the staging and exfil detections each calculate PCR on a different subset of the data (those flows traversing the Internet perimeter for the exfil, and those staying only internal for the staging). Therefore, the combination of big inbound and outbound data transfers don't interfere with each other. Were I to ignore this and just compute PCR across all flows in the dataset, I'd probably have gotten a much more balanced PCR ratio, and therefore the staging and exfil on the same host would cancel each other out. This all just goes to show that PCR-based methods should always take the network vantage point(s) into account, or risk missing things that are anomalous in both directions.

Dealing With Production Datasets

Since this is a test lab, I have both a "clean" dataset (no threat actor activity) and one that contains a mixture of legitimate use and attack traffic (we'll call this the "dirty" dataset). Most readers, though, probably aren't so lucky. If you're trying to do this with your own data pulled from a production network, try defining the dirty data as everything during the previous 7 days and the clean data as anything before that (perhaps to a maximum of 30 or 60 days). Even though your "clean" data may not actually be totally clean, the more you have, the less likely any transient fluctuations are to distort your baseline PCRs.

Exfil Detection

Exfil is data going from inside our network to the Internet, so I started by filtering my flows to select only those where the source (orig_h) is an internal IP and the dest (resp_h) is an Internet host. In plain language, I selected only at flows that crossed the network's perimeter (transit to/from Internet).  Then I simply summed up the bytes sent and bytes received for each host (Bro's orig_ip_bytes and resp_ip_bytes columns, respectively).  

Note that since Bro records bi-directional flows, I had to calculate PCR for any host that appeared as either a source or a destination.  Further, each "destination" host not only received bytes, but sent some of its own, so I had to sum the resp_ip_bytes twice: once as the bytes received by the src host and once as the bytes sent by the dest host.  Ditto for the orig_ip_bytes, but in reverse.  In psuedo code, it would look something like this:

srchost_bytes_sent = sum(orig_ip_bytes)
srchost_bytes_recvd = sum(resp_ip_bytes)

dsthost_bytes_sent = sum(resp_ip_bytes)
dsthost_bytes_recvd = sum(orig_ip_bytes)

I was looking for hosts that have a fairly large PCR shift in the positive direction. In the extreme case, the baseline value would be -1.0 (a pure consumer) and the dirty value would be +1.0 (a pure producer). To make those hosts show up nicely, I calculated the shift as (dirty PCR - baseline PCR). In the best case, the shift would therefore be 2.0.

I constructed a scatter plot of PCR shift for each host, showing the baseline PCR on the X axis, the "dirty" PCR on the Y axis, and the amount of PCR shift as the color. The trend line provides an easy reference to see where "no PCR shift" would fall on the graph, and makes it a bit easier to eyeball the distances for the outliers.

Most hosts will tend to adhere closely to their own baselines, given enough data points. Therefore I would expect that the plot should look very much like a straight diagonal line (the graph of the line y=x). Furthermore, most colors should come out as a nice neutral grey.

Red hosts are more likely to be exfiltration, since those hosts shifted the most in a positive direction (towards being a producer). Theoretically, very blue hosts could indicate the exfil destinations (they suddenly consumed a lot more data than normal). However, being Internet hosts, we can't count on that. I only plotted hosts that have PCR values in both the baseline and in the "dirty" dataset. If I'd never seen a particular Internet host before (as is probably the case with threat actor infrastructure), it wouldn't show up. In practice, once you know the exfil host it's probably not too difficult to identify the host(s) that received the data, but if this were an issue you could try to do something smarter here.

As you can see in the graph above, the most red host isn't actually all that red (the exfil package wasn't very large), but it is the host that exfiltrated the data in our attack scenario.

Staging Detection

Data staging is defined as data moving purely internal to a network. This should show up as a hosts consumer ratio becoming more negative (the staging host). It may also show as some hosts (the hosts from which the data was stolen) becoming more positive if a lot of data was stolen from them.

For this part of the analysis, I specifically filtered the flows to only those  that both originated and terminated within the internal network.  As before, I calculated the PCR shift for each src & dest host in the dataset.  I calculated the PCR shift slightly differently here, since I was looking for different activity. In fact, I was trying to find the inverse of what I was looking for before, so I inverted the shift calculation, too. That is, in the best case, a system would go from a pure producer (PCR 1.0) to being a pure consumer (PCR -1.0). I calculated PCR shift here as (baseline PCR - dirty PCR), which would again mean a PCR shift of 2.0 for a host staging data.  I could have skipped this inversion, but it's an easy way to make the most significant shift look red on the graph, which I like.

I then constructed a new scatter plot of PCR shift for each host, showing the baseline PCR on the X axis, the "dirty" PCR on the Y axis, and the PCR shift as the color, just like the previous graph.

Again, hosts adhering to their own baselines would tend to fall near the diagonal y=x line, and be colored grey. Red hosts are more likely to be unusually large consumers of data relative to their own baselines, and therefore may represent systems being used as data staging points. In fact, as indicated above, the most red host is actually the host that was used to stage the stolen data. It's very red, indicating that it experienced a large shift.

Unlike in the previous graph, where blue points weren't likely to be very useful, they could very well indicate the sources of the stolen data in this graph. If the attacker stole a significant amount of data from any given system, it may be that the act of that victim host transferring it's data to the staging point caused a significant PCR shift in the producer direction. If so, those hosts would tend to fall out of the expected diagonal and be more blue than grey. In fact, most of the blue points here actually were the ones from which data was stolen. Most only lost a little data, it seems, though one host is quite blue, indicating that it may be the source of the bulk of the data.

Conclusions

Based on my test data, it seems like this has promise. In both cases, the "most red" point on the graphs corresponded to the data exfil or staging host. For staging, we were actually able to derive some additional information about the likely sources of the data that was stolen. We may not be able to rely on this in all cases, and it's likely to be much more complicated in a real enterprise environment, but where it's present, it may be quite useful. At least in my small dataset, tracking PCR shift proved an effective method for identifying both data staging and data exfiltration.

A Simple Hunting Maturity Model

UPDATE 2015-10-15: I received some questions about the roles of automation in HMM0 and HMM4, which I addressed in a new section.  Also, I named the levels to make them easier to reference.

It's been conference season for me lately. I've attended several in the last few months, and most of them have had at least a few sessions on "hunting".  Since that's a research interest of mine, of course I attended all of them that I could possibly fit into my schedule.  I've gotten some great ideas from these presentations, and have begun to notice some patterns in what's being presented.  I've also spoken with a number of people at the conferences who have made some variation of the remark, "Sure, we'd like to hunt, but how do we figure out where to start?"

Recently, I had the opportunity to discuss with my colleagues the common types of activities that a security team does, and how they relate to hunting.  I drew from the presentations I've seen and the conversations I've had to make some generalizations about the different levels of capability within hunting organizations I've encountered.  These were literally just some bullet points on a whiteboard, but one of my colleagues said something along the lines of "That's kind of like a maturity model" and that got me to thinking.  I realized that I had never actually seen anything like that, but if one existed, then it'd be a useful way to measure the state of a hunting program and to help drive improvement.

So I went away and thought about it for a few days.  What follows is an expansion of the original idea, which I am making available in the hopes that the community will find it useful.

What is Hunting?

Before we can talk about hunting maturity, though, we need to discuss what, exactly, we mean when we say "hunting".  I usually define hunting as the collective name for any manual or machine-assisted techniques used to detect security incidents.  I refer to it as a "collective name" since there are many different techniques hunters might use to find the bad guys, and no single one of them is always "right".  In fact, it's good to be familiar with many different methods so you can choose the best one depending on the type of activity you are trying to find.

I also refer to hunting as being "manual or machine-assisted" as opposed to being automated.  I'm a big believer in automated alerting, but it can't be the only thing your detection program relies on.  In fact, one of the chief goals of hunting should be to improve your automated detection capabilities by prototyping new ways to detect malicious activity and turning those prototypes into production detection capabilities.

The Hunting Maturity Model

With that definition of hunting in mind, let's consider what makes a good hunting program.  There are three factors to consider when judging an organization's hunting ability: the quality of the data they collect for hunting, the tools they provide to access and analyze the data, and the skills of the analysts who actually use the data and the tools to find security incidents.

Of these factors, the analysts' skills are probably the most important, since they are what allows them to turn data into detections.  That's why each level of the Hunting Maturity Model (HMM) starts with a statement of the level of analytic capability that is typically found at that level.

The quality of the data that an organization routinely collects from its IT environment is also a strong factor in determining the HMM level.  The more data (and the more different types of data) you provide to an expert hunter, the more results they will find.  The toolset for collecting and analyzing the data is a factor as well, but a less important one.  Given a high amount of analyst skill and a large amount of good quality data, it's possible to compensate for toolset deficiencies, at least to a degree. For this reason, each HMM level discusses the quality and amount of data that's routinely collected across the enterprise, but does not directly address the analysis toolset.

The Hunting Maturity Model (HMM)

The Hunting Maturity Model describes five levels of organizational hunting capability, ranging from HMM0 (the least capability) to HMM4 (the most).  Let's examine each level in detail.

HMM0 - Initial

At HMM0, an organization relies primarily on automated alerting tools such as IDS, SIEM or antivirus to detect malicious activity across the enterprise.  They may incorporate feeds of signature updates or threat intelligence indicators, and they may even create their own signatures or indicators, but these are fed directly into the monitoring systems.  The human effort at HMM0 is directed primarily toward alert resolution.

HMM0 organizations also do not collect much information from their IT systems beyond the bare minimum needed to drive their automated alerting.  Thus, even if they were to somehow acquire hunting expertise (perhaps by hiring a consultant or by making a strategic hire), their ability to actually hunt would be severely limited.

Organizations at HMM0 are not considered to be capable of hunting.

HMM1 - Minimal

An organization at HMM1 still relies primarily on automated alerting to drive their incident response process, but they are actually doing at least some routine collection of IT data.  These organizations often aspire to intel-driven detection (that is, they base their detection decisions in large part upon their available threat intelligence).  They often track the latest threat reports from a combination of open and closed sources.  

HMM1 organizations routinely collect at least a few types of data from around their enterprise, and some may actually collect a lot.  Thus, when new threats come to their attention, analysts are able to extract the key indicators from these reports and search historical data to find out if they have been seen in at least the recent past.  

Because of this search capability, HMM1 is the first level in which any type of hunting occurs, even though it is minimal.

HMM2 - Procedural

If you search the Internet for hunting procedures, you will find several great ones.  These procedures most often combine an expected type of input data with a specific analysis technique to discover a single type of malicious activity (e.g., detecting malware by gathering data about which programs are set to automatically start on hosts across the enterprise and using least-frequency analysis to find suspicious binaries).  Organizations at HMM2 are able to learn and apply procedures developed by others, and may make minor changes, but are not yet capable of creating wholly new procedures themselves.

HMM2 organizations routinely apply these procedures, if not on a strict schedule, then at least on a somewhat regular basis. 

Most of the commonly available procedures rely in some way on least-frequency analysis (as of this writing, anyway).  This technique is only effective if there is data from many different hosts. Therefore, HMM2 organizations usually collect a large (sometimes very large) amount of data from across the enterprise.

HMM2 is the most common level of capability among organizations that have active hunting programs.

HMM3 - Innovative

HMM3 organizations have at least a few hunters who understand a variety of different types of data analysis techniques and are able to apply them to identify malicious activity.  Instead of relying on procedures developed by others (as is the case with HMM2), these organizations are usually the ones who are creating and publishing the procedures.  Analytic skills may be as simple as basic statistics or involve more advanced topics such as linked data analysis, data visualization or machine learning. The important point is that the analysts are able to apply the techniques to create repeatable procedures, which are documented and performed on a frequent basis.

Data collection at HMM3 tends to be similar to that at HMM2, though perhaps more advanced, since the continual focus on developing new techniques tends to drive the analysts into new data sources over time. 

HMM3 organizations can be quite effective at finding and combating threat actor activity.  However, as the number of hunting processes they develop increases over time, they may face scalability problems trying to perform them all on a reasonable schedule unless they increase the number of available analysts to match.

HMM4 - Leading

An HMM4 organization is essentially the same as one at HMM3, with one important difference: automation.  At HMM4, any successful hunting process will be operationalized and turned into automated detection.  This frees the analysts from the burden of running the same processes over and over, and allows them instead to concentrate on improving existing processes or creating new ones.  

HMM4 organizations are extremely effective at resisting adversary actions.  The high level of automation allows them to focus their efforts on creating a stream of new hunting processes, which results in constant improvement to the detection program as a whole.

Automation and the HMM

It may seem confusing at first that the descriptions for both HMM0 and HMM4 have a lot to say about automation.  Indeed, an HMM4 organization always has automation in the front of their minds as they create new hunting techniques.  The difference, though, is that HMM0 organizations rely entirely on their automated detection, whether it’s provided by a vendor or created in house. They may spend time improving their detection by creating new signatures or looking for new threat intel feeds to consume, but they are not fundamentally changing the way they find adversaries in their network. Even if they employ the most sophisticated security analytics tools available, if they are sitting back and waiting for alerts, they are not hunting.

HMM4 organizations, on the other hand, are actively trying new methods to find the threat actors in their systems.  They try new ideas all the time, knowing that some won’t pan out but others will.  They are inventive, curious and agile, qualities you can’t get from a purely automated detection product.  Although a good hunting platform can certainly give your team a boost, you can’t buy your way to HMM4.

Using the HMM

A CISO that's heard that her organization needs to "get a hunt team" may legitimately be convinced that an active detection strategy is the right move, and yet still be confused about how to describe what the team's capability should actually be.  It is my hope that by providing a simple maturity model, anyone thinking of getting into hunting will be able to get a good idea of what an appropriate initial capability would be (I recommend HMM2 to start with, BTW).  

Maybe more importantly for those organizations who already hunt, the HMM can be used both to measure their current maturity and provide a roadmap for improvement.  Hunt teams can match their current capabilities to those described in the model, then look ahead one step to see ideas for how they can develop their skills and/or data collection abilities in order to achieve the next level of maturity.

Although there's starting to be a lot of good hunting information available in the open literature, I believe that much of the confusion over what hunting is and how you should do it comes down not to processes but to the lack of a publicized hunting framework.  In order to get anywhere, you must first know where you are and where you want to be.  I hope the HMM will be a useful tool to help us all figure these things out.