Triage Any Alert With These Five Weird Questions!

(OK, so I went all "BuzzFeed" on the title.  My alternate was going to be "What kind of alert are you?  Take this quiz and find out!" so be thankful.)

Introduction

There are few things more frustrating to users than using a tool which doesn't support (or may even be at odds with) their processes.  Tools should be designed to support our workflows, and the more often we perform a workflow, the more important it is that our tools support it.  As analysts, our most commonly-exercised workflow is probably alert triage.  Alert triage is the process of going through all of your alerts, investigating them, and either closing them or escalating them to an incident.  Escalated incidents lead directly to incident responses (IRs), of course, and there's not always a distinct handoff where triage ends and response begins.  Therefore, some of the basic IR tasks are part of this workflow as well.

About five years ago, I was tasked with training a group of entry level security analysts to do alert triage.  Previously, I'd really just "done it" and never really thought much about how it worked.  After mulling it over for a while, though, I realized that the entire process really boiled down to a set of questions that the analyst needs to have answers for.

1. Was this an actual attack?
2. Was the attack successful?
3. What other assets were also compromised?
4. What activities did the attacker carry out?
5. How should my organization respond to this attack?

If you start from the assumption that the analyst sitting in front of an alert console is going to be answering these five questions over and over again, it seems pretty clear that we need to make sure that console makes it as easy and as quick as possible for the analyst to get these answers.  We may not need to answer every one of these questions every time, nor do we always tackle them in the same order.  In general, though, this is the process that we start from, and then adapt it on-the-fly to meet our needs.

I thought it might be interesting to examine these questions in a little more detail.

Was this an actual attack?

Of course, this is the first answer we need.  You could restate this question as "Is this alert a false positive?" and it would mean the same thing.  As an industry, no one has yet figured out how to eliminate all the false positive (FP) alerts while still keeping all the true positive (TP) alerts.  Given that we know that there will be FPs (probably a substantial percentage of them), we need to make it as easy as possible for the analyst to distinguish between FP and TP, and to do it quickly.

The keys here are:

• Providing the user with the context around the alert (what scenario is it intended to detect, what do actual examples of the TPs look like, etc)
• Identifying what other information (stuff that's not already in the alert) the analyst needs to see, and providing quick easy access to this (e.g., pivot to examining the PCAP for an alert)

Was the attack successful?

If you've ever monitored the IDS/NSM console of an Internet-facing asset, you know that the vast majority of exploit attempts probably fail.  These are mostly automated scan, probes or worms.  They attack targets indiscriminately, without regards to OS, software stack or version numbers.

Unfortunately, if you alert on exploit attempts, dealing with these alerts is a substantial burden to the analysts.  This is due entirely to the sheer number of alerts they have to deal with.  This is one reason I tend to focus my attention on detections further down the Kill Chain: it cuts down on the number of unsuccessful attack attempts the analysts have to deal with.

Even though we try to avoid them as much as possible, there are still plenty of situations where we are alerting on something that may or may not have been a success.  For example, if we alert on a drive-by download or a watering hole attack, the analyst then has to see if the browser reached back out to download the malicious payload.  The key information they need will be much the same as before:

• Context
• Quick & easy access to related information

If the answer to this question is "Yes, the attack was successful" the next step is usually to escalate the alert to a full-blown incident.

What other assets were also compromised?

This is where things start to get really interesting.  Assuming the alert indicates a successful attack, then you have to start doing what we call scoping the incident.  That is, before you can take any actions, you need to gather some basic information with which to plan and make response decisions.

The first step when determine the scope of the incident is to assemble a list of assets involved.  "Assets" in this case can be any object in your organization's IT space: computers, network devices, user accounts, email addresses, files or directories, etc. Anything that an attacker would want to attack, compromise or steal would be considered an asset.

For example, you may see an intruder try a short list of compromised usernames and passwords across a long list of hosts, trying to find out which credentials work for which hosts.  In this case, your asset list would include every one of the compromised user accounts, but also all of the hosts they tried to use them on.  If they performed this action from within your corporate network, the source of those login attempts would also be on the list.

It's also worth noting that the asset list will almost certainly change over the course of the incident.  For smaller incidents, you may be able to both assemble the asset list and filter out the ones which weren't actually compromised, all in one step.  For larger incidents, it's pretty common to create the list and then parcel it out to a group of incident responders to verify.  As the analysts determine assets are not compromised, they may drop off the list.  Similarly, as new information comes to light during the course of the investigation, new assets will probably be added to the list.  The asset list is one of the most volatile piece of information associated with any active incident response, and keeping your list current is absolutely critical.

The keys to helping analysts deal with asset tracking are:

• Creating a template "asset list" that can be cloned for each alert/incident to track compromise status.  At a minimum, you probably need to track the following info for each asset
• Asset name (hostname, username, email address, etc)
• Date/time the attacker was first known to interact with that asset
• Date/time the attack was last known to interact with that asset
• Date/time an analyst detected the activity
• Date/time the asset was added to the list
• Date/time the analyst investigated the asset and determined it to be compromised or not
• The results of that investigation (often COMPROMISED, NOT COMPROMISED, PENDING ANALYSIS)
• Brief notes about the investigative findings 
• Sharing this list with other responders, in such a way as to make it easy to interact with (e.g., via a wiki)

What activities did the attacker carry out?

The next big piece of scoping the incident is just trying to come up with what I call the "narrative" of the attack:  What did the attacker try to do, when, and to what?  The asset list answers the "to what" question, but even that needs to get put into context with the other stuff.

To help answer these questions, incident responders typically create timelines of the significant events that occur during the attack and the IR.  For example, you might start with a simple chart that calls out the highlights of the incident, with dates and times for:

• The attacker's first exploit attempt
• All the alerts generated by the attack
• When the alerts were triaged and escalated to incidents
• When the affected asset(s) were contained
• When the affected assets were remediated
• When the incident was closed

As your investigation gathers more details about what happened, the timeline grows.  As more confirmed malicious events are discovered, and incident response milestones achieved, these are added to the timeline.  Just like the asset list I mentioned before, the timeline typically changes a lot during the course of the investigation.

Once you have gathered the timeline data, you may want to display it in different ways, depending on what you're trying to accomplish or who you are sharing it with.  For example, a table view is common when you're editing or managing the events, but some sort of graphical view or interactive visualization is a much nicer way to show the narrative to others or to include in your incident reports.  There are some timelining tools available out there, but to start with, I recommend trying just a plain spreadsheet (or similar format) and seeing how that works before getting too complicated.

The keys to helping the response team track activities are:

• Creating a timeline template that can be cloned for each incident.  Key fields to track here might include
• A description of the entry (attacker action, milestone reached, etc)
• Date/time that entry occurred
• Sharing this list with other responders, in such a way as to make it easy to interact with (e.g., via a wiki)

How should my organization respond to this attack?

This is the big question!  Once you have assembled the list of compromised assets and the timeline of events, you exit the scoping phase and are ready to begin planning your incident response strategy.  In most a cases, a typical strategy involves containment (removing the attacker's access to the compromised assets) followed by remediation (bringing the compromised assets back into production state).  The topic of incident response strategies is far too detailed to get into here, but it's worth noting that this is not a one-size-fits-all situation.  Different types of incidents require different plans (sometimes called playbooks).  A well-managed CIRT will have a number of standard IR plans ready to go, but even then they often need to be tailored to the individual incident.  And it's still not uncommon to find incidents that don't fit any of the standard plans exactly, in which case the response teams need to create an entirely new plan on the fly (using pieces of existing plans, if they're lucky).

The creation and application of these plans still requires a high degree of skill and experience, something that a lot of organizations don't have enough of in-house.  If this is an issue, you may consider engaging a consultant with experience building CIRT teams to help guide you through this process.

The keys here are:

1. Identifying who in your organization has the experience necessary to come up with good playbooks, or getting outside help.
2. Creating playbooks that are specific to your organization's IT environment, policies and security goals.
3. Sharing these playbooks among all the incident responders
4. Training with and testing the playbooks on a regular basis

What does this all mean?

To a large extent, an organization's ability to detect and respond to security events depends on the quality of their tools.  It's not enough to just go out and get a bunch of "solutions" if they don't support the way you work.  We need to be able to leverage those tools to improve our ability to get our work done faster and better.  By identifying the common workflows we go through, we can design our toolset to help us be more productive. This translates into more effective responses that happen faster, thus helping us protect ourselves much better. 

I know that other very experienced incident responders will read this.  I'd love to hear your feedback, so please leave a comment!

The Defense Chain

Intro

If you're reading my blog, you're probably already familiar with the Kill Chain (KC).  Briefly, it's a generalized model of the stages an adversary has to go through to carry out a targeted attack.  It's been around for several years now, and just seems to become more popular over time.  It's a great model that captures a complex subject and presents it in a simple way.  I'm a big fan.

I've been thinking recently about the process that we as defenders have to go through to protect our networks against attacks (targeted or otherwise).  Many papers and articles have been written on the subject, and most of us probably know the highlights: we make policies, we enforce the policies using both technical and non-technical means, we monitor our networks and we respond to incidents.  The process is actually quite complex, though, and I began to think of how I would create a model to show it visually.

After a bit of pondering, it struck me that I was essentially trying to create the defender's counterpart to the Kill Chain.  That is, we already know the stages an attacker has to go through to carry out an attack; now we need a model that shows the stages a defender must go through to protect against these attacks.  One of the reasons the KC is so popular is that's it's very straightforward.  I mean, literally, it's a line.  So I thought, what would it look like if I applied it to defense?

The Defense Chain

The result is what I (rather clumsily) termed, "the Defense Chain" (DC).

The Defense Chain

Like the Kill Chain, the Defense Chain has seven phases (pure coincidence, but I admire the symmetry).  I should probably mention that the DC is not just for detection and response.  It includes all of your management and protective controls as well.  Of course, detection and response each get their own separate phases, which I think underscores their importance.  You can't get to the end of the chain if you're missing a link!

Let's examine the phases in more detail.

Plan

Before you can begin to protect your network, you first must figure out some key things, like what exactly you wish to protect, and what you're trying to protect it from.  In the Plan phase, you do things like identifying your assets and creating your security and incident response policies to help protect them.  This is also where you begin to decide what types of protective controls you will need (firewalls, endpoint protection, network proxies, etc), how you will deploy them, how you will monitor the entire system (because prevention always fails), and who's going to do all this work.

If you've ever been involved in the creation of a security program before, you'll know there are a lot of things to plan here.  So many things, in fact, that I'm not even going to try to list them all.  Just know that the planning phase is probably the most important piece of the Defense Chain, because everything else depends on it. 

Build

Compared to planning, building is often fairly straightforward.  During this phase, you assemble teams, learn skills and create or acquire the technical tools necessary to carry out your plans.  

Did you catch what I just did there?  It's vitally important that you build teams and skills *before* you try to build the technical parts of the solutions. Not everyone needs to be an expert, though you certainly need a few of those to guide you, but everyone involved needs to have enough of a background to know what they're doing and why they're doing it.  

It's also worth pointing out that the "Build" phase isn't something you just do one time and then forget about it.  Rather, you should be constantly growing your teams' skills and experience.  You also need to have someone looking over your controls to be sure they are operating efficiently, and to update and improve them as needed.

Monitor

The monitor phase is where you actually operate the technical solutions and perform periodic reviews and drills to exercise your policies and plans.  This could include just making sure the endpoint security solution you chose is working well, ensuring that packet loss on the NSM/ESM systems is within acceptable levels, or running table top incident response exercises to make sure everyone knows what to do.  

This phase is probably where you spend the majority of your time.

Detect

I probably don't need to explain this phase much to my readers.  The detect phase is where you check the output of your NSM/ESM systems, validate the alerts or do some proactive hunting through the data to find evil.  This could also include fielding user queries about "weird" things on their computers or suspicious emails they received. 

Respond

This is another phase I probably don't have to explain much here.  Once you have found evil, you need to exercise those incident response plans you developed in the Plan phase.  Investigate, contain and remediate!  Kick out the bad guys and bring the affected assets back into normal operation.

Report

The Report phase is all about gathering information about your successes and failures, analyzing it to make recommendations for improvement, and communicating this to the right people.  Typically, reporting is a followup to an incident response, but you would also do this for other reasons (e.g., to review a red team engagement or an auditors' findings).  

Not everything is a formal report, either.  Sometimes your "report" might be a presentation, a post to an internal blog, or even an email.  The key is that you communicate findings and recommendation to the right people in whatever way makes it easiest for them to digest the information.

Improve

Security programs are not static!  You need to constant improve you skills, your tools and your procedures to keep ahead of the bad guys.  That's what this phase is all about.  After the successes, failures and recommendations have been documented and reported, you need to make sure you act on them. So many organizations skip this step, and although it might make less work in the short term, it makes more work in the long term as they play keep-up with threats that have advanced beyond the organization's ability to protect themselves.

Conclusion

There you have it.  In 10 minutes or less, my thoughts on a model of how an organization successfully defends itself against attacks.  I don't claim that the Defense Chain model really contains anything new.  Rather, I hope to just provide a simple visual guide to all the things you need to do, in rough order, and layed out in a way that makes it easy to visualize how all the phases flow and work together.  

I know my readers deal with these sorts of things every day.  Please, leave a comment below to let me know what you think.  I'm eager to hear comments, questions and criticisms!

Use of the term "Intelligence" in the RSA 2014 Expo

I attended RSA 2014 this week, and one of the things that struck me was the recurrence of the term "intelligence" on many vendor booths.  I decided it would be a fun exercise to go through the expo halls to ask the vendors to clarify what their uses of the term "intelligence" meant.

Methodology

First, the parameters of the exercise.  I chose an arbitrary starting expo hall (honestly, never once in the whole week was I able to accurately remember which was North and which was South) and walked the aisles from one side to the other, examining each booth to see if they referenced either "intel" (not the chip maker) or "intelligence".  I did not consider booths that only had related terms like "information" or "sharing", nor did I consider booths for vendors I know play in the intel space but neglected to include the term in their display (I did stop at a couple of non-vendor booths that fell into this category, though.  More on this below.).

At each booth, I explained who I was and what I was doing, and asked if there was anyone there who could answer a few quick questions about their use of the term "intelligence".  Some of the vendor representatives were more well-equipped to answer my questions than others, but in all cases I let them decide who I should talk to, in order to try to avoid polluting the results with my own personal biases about who would make a "good" representative.

After establishing contact, I then pointed out their use of "intelligence" on their display and asked, "Can you explain what you mean by that?" If their answer seemed to roughly line up with the idea of "using information to detect malicious behavior" I then asked followup questions listed below, otherwise I thanked them for their time and ended the interview.

• What types of information do you consider to be "intelligence"?
• Are some types more valuable than others, either inherently or in certain circumstances?
• How can your customers know they're getting the maximum value out of their intelligence?

Vendors

In total, I visited 10 vendors and 2 non-vendors.  In fact, I had planned to visit more, but the expo halls are so large that I didn't even complete the tour of one hall during the short amount of time they were open on Thursday (the day I conducted this exercise).  

The vendors I visited were:

1. Webroot
2. LogRhythm
3. R-sam
4. IBM
5. NetIQ
6. Arbor
7. Solutionary
8. AlienVault
9. Securonix
10. BAE Systems

The non-vendors I visited were special cases, because although neither mentioned "intelligence" on their displays, I felt the organizations had enough expertise in the intelligence field that their perspectives might have been useful:

1. Homeland Security
2. National Security Agency

Unfortunately, neither of these entities were able to discuss their thoughts about intelligence, as that was not the purpose of their booths and neither brought any experts in that area, so I include them here only as an interesting sidenote.

Full disclosure:  I work for the Mandiant division of FireEye, both of which are well known for their threat intelligence.  I purposely left them out of this survey.  I did this not to throw criticism on other vendors, but because I have a much deeper knowledge of what FireEeye considers to be "intelligence" and I couldn't effectively include them in the survey without biasing the results towards my employers.

Uses of the term "Intelligence"

Out of the 10 vendors, I found variations of 8 different uses of the term "intelligence".

1. Threat intelligence (4 vendors)
2. Security intelligence (2 vendors)
3. Identity intelligence (2 vendors)
4. File intelligence (2 vendors)
5. Application intelligence (2 vendors)
6. Risk intelligence (1 vendor)
7. Applied intelligence (1 vendor)
8. Insider threat intelligence (1 vendor)

Most vendors stuck with one of the above, or a close rewording that meant the same thing.   One vendor (Securonix) actually used several different variations in their display.  Their representative explained this by saying "we add 'intelligence' to the end of everything."  In fact, if I had taken additional uses of "intelligence" from our conversation, I would have added several more to the list above.  This would have been breaking my own rules, though, so I omitted them.

Different meanings of "Intelligence"

The definitions of "intelligence" broke down into three categories:

1. "Intelligence" in the sense "doing something smart with input data to achieve a result" (i.e., what is also often referred to as "analytics", "anomaly detection" or just "correlation").  Terms used this way included "security intelligence" and "risk intelligence".
2. Enriching input data to allow security decisions to be based on more organizational context than was originally present in the data set (e.g., adding user identity information to incoming log events).  Terms used this way inlcuded "identity intelligence", "file intelligence"and "application intelligence".
3. Consuming information about adversaries, tools or techniques and applying this to incoming data to identify malicious activity.  The term "threat intelligence" was the most commonly used phrase in this category, although "insider threat intelligence" also applied.

Readers of my blog will note that definition #3 most closely matches up with what I consider to be "intelligence".  It's worth noting that Chris Sanders defines #2 as "Friendly Intelligence" in chapter 14 of his book, Applied Network Security Monitoring (Full disclosure, I was a contributing author, though not of that chapter).

Types of "Intelligence"

For those vendors who's use of the term "Intelligence" fell in line with definition #3 above, I then asked about the types of intelligence they deal with. By far, the most common were IP addresses and domains, though URLs were also sometimes mentioned.  The recurring ideas of "file" and "application" intelligence strongly implies the existing of file hash values as well.  I will not attempt to summarize how many vendors mentioned each type, primarily because many of the vendor representatives I spoke to either weren't willing or weren't able to go into detail about the types of intel data they dealt with.

Applying the Pyramid of Pain model to responses shows that the respondents' use of threat intelligence still falls mainly into the bottom half of the pyramid.  The few vendors that mentioned URLs may be working at least partially in the artifacts level as well.  No vendor in my survey mentioned any type of indicators that would fall into the upper levels of the pyramid ("Tools" or "TTPs").

It's interesting to note that most vendors who use definition #3 cited primarily network-based data types (IPs, domains, URLs).  A few mentioned "file" or "application" intelligence, which implies more of a host-based orientation, but no one mentioned traditional host-based indicators such as file names, registry keys or processes names.  (One vendor did mention file names, but in conjunction with definition #1).  This may indicate a gap in our industry's thinking about what types of information can be useful in detecting malicious activity, it may be a function of the types of products that the vendors in this survey are selling, or a combination of the two factors.

Other interesting things of note

I mentioned earlier that both Homeland Security and the National Security Agency had booths, but weren't able to comment on their ideas of intelligence.  In fact, I did get one quote from the NSA representative, which I thought was interesting:  "Information doesn't become intelligence until it is useful to someone."  I interpret this to mean that the information also has to be consumable (information buried in a PDF report isn't that useful; it needs to be put into detection mechanisms).  Since there is often a lot of confusion about the difference between information and intelligence, I think this is a nice way to phrase the difference so that people can understand.

Some of the individual vendor representatives also touched on a similar theme, drawing the distinction between "information" or "facts" and "intelligence".  For example, Webroot mentioned that they have databases of "facts" like IP or file reputation, but that they have a process that combs through those databases to try to find connections and correlations and place them in context with other related facts.  The ouput of this process is what they consider "intelligence": facts in context with each other.

The representative from Solutionary also had an interesting point of view.  He described a hierarchy of "technical indicators" which are facts about the state of something, independent of possible security concerns (e.g., "the system is out of memory"), "threat indicators" which do have security implications, and "threat intelligence" which is a combination of the two, with additional higher-level context.  The hierarchy goes something like:

technical indicators < threat indicators < threat intelligence

I've seen the "threat indicators < threat intelligence" before, and I think there is broad agreement on this among actual intelligence analysts, but I was unfamiliar with the concept of lower level technical indicators, although they seem pretty obvious in hindsight.

Conclusions

This turned out to be a pretty interesting exercise.  The sample set is by no means large enough to constitute a reliable study, but I do think it has some valid things to say about our industry's approach to "intelligence" in general.  I draw the following conclusions:

1. "Intelligence" is a buzzword that can mean anything you want it to mean.  In my sample of 10 companies, there were 3 separate definitions (broadly speaking, probably more if you scope the definitions more narrowly).  That's a lot of variance given the small number of respondents.  It'd be interesting to expand this to a much larger set of vendors to see how many other definitions we can collect.
2. There is a valid case for the concept of "friendly intelligence".  Of the three definitions, two of them actually did refer to the use of some sort of information to make it easier to detect malicious activity.  Definition #2 is what Sanders calls "Friendly Intelligence" though none of the vendors I spoke to used this term. It does a good job at disambiguating the term "intelligence" and clearly indicates the idea of intelligence based on information you generate about yourself versus information about your adversaries.  This is an important concept, and by naming it, we make it easier to identify and understand.
3. We are focused on the wrong types of threat intelligence. Most of the vendors' concepts of threat intel were solidly on the bottom half of the Pyramid of Pain, which suggests that the indicators we're focused on are the ones that are the least valuable to the adversaries.  This, in turn, means that our incident detection and response operations are purely following the adversaries' lead and playing to their strengths.  Instead, we should be developing tools and techniques to allow us to develop and apply intel near the top of the pyramid, where we can increase the attackers' costs of doing business against us and make them work harder (and expend more resources) to accomplish their missions.
4. No one has any idea if we are using intelligence effectively.  Or even what "effectively" means in this context.  Although I had three followup questions prepared (listed above), I rarely got to ask the final one.  The questions were designed to follow each other in logical succession, so if the vendor couldn't or wouldn't answer one of them, I skipped the succeeding questions.  No vendor was able to successfully provide an answer to the second followup question, "Are some types more valuable than others?"  I was definitely not expecting anyone to parrot back the Pyramid of Pain or anything like that, but was hoping for some indication that certain types of indicators had different characteristics in terms of false or true positives, applied to more specific or more broad classes of attacks, or at least in general that not all types of data were of exactly equal use in detection or response.  I didn't get that from any vendor, which leads me to believe that the idea of "throw it all at the wall and see what sticks" may still be the dominant paradigm in many of today's security solutions.

I wish I could say I was more surprised by these results.  I suspected that result #1 was probably true, which was the original reason for this exercise.  If you have read my previous blog articles, you will know that I have been saying #3 and #4 for some time, and these findings to tend to confirm my views.  Result #2, though, I offer as a constructive finding.  The term "intelligence" itself is neutral; it is neither malicious nor benign.  We are used to the phrase "threat intelligence" as the kind that deals with malicious activity, and there is broad acceptance of this term in the industry. However, there is no equivalent term for information about oneself that can be used to help identify malicious activity, even though many vendors are clearly expressing this concept in their own different ways.

I had a lot of fun doing this: I got to meet a lot of new people, have some interesting conversations and even walk off some of this fantastic San Francisco food!  My thanks to all the folks I talked to while doing this research.  Perhaps I will try this again next year and see how the results compare.

BSidesAugusta

Update 2014-03-14
Here's a link to the video: https://www.youtube.com/watch?v=SVKcFhyGqcY.

Update 2014-01-17
Holy wow, I just found this in my draft's folder!  I obviously meant to publish it a few months ago, but somehow didn't. Normally I'd just skip it and not bother, but it does have a link to my ESM presentation video, which I think some people might find useful.  It's better if I just fess up.  Yup, I was an idiot.

---

Yesterday, I was lucky enough to attend the inaugural BSidesAugusta in Augusta, GA.  This was a fantastic high energy event with a lot of great talks.  I spoke on the Blue Team track about many of the themes I've blogged about already, plus a lot more that have been baking.

My talk was entitled "Enterprise Security Monitoring", and covers not only the themes I've blogged here already, but a lot of other stuff that I've been working on but haven't yet had a chance to write up. 

I had several people ask me if I could make my slides available, so here they are.

The talk was also recorded, so I'll post a link to the YouTube video when it's available.  You can also expect a blog post sometime in the next few days to explain this concept a bit more.  Finally, I'll also be giving an updated version of this talk next month at BSidesDC, so if you didn't catch me in Georgia, come to Our Nation's Capitol and see me there!

On the Misuse of Indicators

This being only my third blog post, you might think it's too early to make predictions about my most commonly-used terms, but I'm ready to go out on a limb.  You can pretty much rest assured that you'll be reading the word indicator pretty frequently in these posts.  You would think that most of us in the DFIR field probably already know what an indicator is, but I'm not so sure.  There are at least two different but confusingly-similar types of indicators we deal with every day, and the meaning of "indicator" has a lot to do with who is using the word, and the context in which they are using it.

What is an Indicator?

"Finger-pointing-icon" by debivort,
http://commons.wikimedia.org/wiki/File:Finger-pointing-icon.png,
19 July 2013

In non-technical terms, an indicator is a piece of information that points to a certain conclusion.  An individual indicator may or may not be enough to effectively support the conclusion to which it points, but as you collect more indicators that agree with each other, the conclusion becomes more likely.  Given enough of these indicators, you may end up with a defensible statement about the likelihood of your conclusion being true.  Your indicators serve as the data points by which you prove your argument. 

An indicator can be almost any type of data that you think captures some sort of repeatable occurrence or pattern.  It could be a simple domain name or IP address known to be used by a piece of malware, a piece of recurring data in a transaction, a combination of adversary actions that make up a distinct behavior, or nearly anything else.  Both of my previous posts dealt heavily with indicators, so please refer to them if you need specific examples. 

Attribution vs. Detection

In my experience, there are at least four different types of indicators:

1. Attribution Indicators are used to distinguish activity or artifacts traceable to a specific threat actor.
2. Detection Indicators (often called "Indicators of Compromise" or IOCs) are observables that you look for to help find security incidents.
3. Prediction Indicators are behavior patterns that foreshadow other events ("We just announced a major new business initiative, therefore we can expect recon from this adversary within 30 days.").
4. Profiling Indicators help predict which of your users, facilities or projects are likely to be the subject of targeted attacks.

Although all four types are definitely interesting, almost no one collects types #3 or #4, so let's ignore those for now.

Attribution indicators are used primarily for doing intelligence analysis to determine the actor behind an attack or artifact (e.g., "Who wrote this malware?" or "Which actor is most likely responsible for this set of recon scans against my web server?").  Attribution indicators attempt to answer the question "Who?"  This is a pretty difficult question, and there's a lot of ambiguity in the attribution process.  You typically need several indicators that all agree with each other pretty well to even arrive at the ballpark of a successful attribution, and even that is an oversimplification.  It leaves out the vital contribution of the human analyst to make decisions, weigh evidence and arrive at defensible conclusions despite ambiguous and possibly conflicting information.  Still, indicators are at the heart of the attribution process.

Detection indicators are linked to observable events on your hosts or network.  You can monitor for these indicators, and if you find them, you may have a security incident.  Detection indicators attempt to answer the questions "Is?" (e.g., "Is this web transaction a SQL injection attack?" or "Is the XYZ trojan active on my network?").  

So Which Is It?

Confusingly, both attribution and detection indicators share many of their data types.  A domain name could be an attribution indicator, a detection indicator, or both.  

For example, consider the once-notorious (but now defunct) Chinese Dynamic DNS site 3322[.]org.  In most networks I've ever monitored, any traffic to *.3322[.]org domains was at least highly suspicious, if not outright malicious.  These domain were pretty good detection indicators, because they were highly likely to serve drive-by downloads or act as C2 nodes for banking trojans (just to give two examples).  However, the simple fact that it was a very popular DDNS site made it nearly useless for attribution.  There were probably hundreds of threat actors active on those domains at any one time, and except for a few who were lazy enough to re-use their subdomains, it was basically impossible to tell who was who. Without additional information, *.3322[.]org usually wasn't a very good attribution indicator.

The opposite situation is also a frequent problem.  Suppose the PANDA BALLS group is known to have a fondness for the popular grammar website eats-shoots-and-leaves.com, which they deface and use to serve malware drivebys in a watering hole attack.  That domain would be a great attribution indicator; when you have an artifact from a confirmed compromise and it references eats-shoots-and-leaves.com, it's a little piece of evidence that begins to support a conclusion about which adversary is responsible.  On the other hand, if you treat that domain name as a detection indicator, you're going to run into trouble when your IDS throws a constant stream of alerts on legitimate traffic from your technical writing staff!

What To Do About It?

An indicator is not an indicator is not an indicator.  Print that out and paste it to the monitors of your entire intel and detect staff.  There are different types of indicators, with different purposes according to the type of work at hand.  If you have an "indicator database" you're probably already in trouble, because you are likely mixing your indicator types indiscriminately.

At the very least, you should start tagging your indicators according to their purpose.  Consider how you would dump a list of all the detection indicators for signature-generation purposes, or how you could take an artifact and compare it to only your attribution indicators.  If you can't do this, you may need to rethink your indicator management strategy.

What Do You Get When You Cross a Pyramid With A Chain?

In my last post, I described what I call the Pyramid of Pain, a simple model to visualize the effect that your use of different types of indicators can have on an adversary's operations.  I think that single post probably got more positive feedback than anything I've ever blogged about before.  I got a lot of great ideas for future posts by reading the comments people made on Twitter, so thanks everyone for the feedback.

One recurring theme has to do with the interaction between the Pyramid and the concept of the Cyber Kill Chain put forth by Hutchins, Cloppert and Amin.  If you're not familiar with the idea behind the Kill Chain, I recommend you take a break right now and go read these articles before proceeding.

How the Pyramid and the Kill Chain Fit Together

Gizah Pyramids ["All Gizah Pyramids.jpg", Liberator, Ricardo,
http://commons.wikimedia.org/wiki/File:All_Gizah_Pyramids.jpg,
Checked 2013-03-06]

Let me start by making a clear statement:  The Pyramid is not a replacement for the Kill Chain, it is a complement.  The Kill Chain model shows the various states an adversary must move through to complete their objective(s).  At each phase, you have the opportunity to detect their actions using certain indicators.  This is where the Pyramid comes in: it serves as a guide for knowing how to prioritize your limited detection resources in order to achieve the maximum benefit.

The Cyber Kill Chain ["Security Intelligence: Attacking the Cyber Kill Chain", Cloppert, Michael, http://computer-forensics.sans.org/blog/2009/10/14/security-intelligence-attacking-the-kill-chain/, Checked 2013-03-06]

To see how you might combine these two concepts, let's examine some use cases from one of the Kill Chain phases.

Bringing the Pain to Reconnaissance

The first phase of the Kill Chain is all about information gathering to plan the attack.  This can involve anything from probes and scans to find potential weak entry points to examining publicly available information to map out the sorts of valuable information they might expect to find once they are inside. 

Let's examine that latter use case.  Since we are talking about indicators you might be able to detect, that probably means we are limited to discussing the adversary's direct interactions with our own systems (e.g. our web servers or other Internet facing infrastructure.  Things like checking the Google caches or examining SEC filings are not likely to leave indicators where we can see them, so they don't really count here.)

Starting at the lower level of the Pyramid, you could try to keep track of all the IPs that you know the adversary uses for reconnaissance operations, but this is not likely to be very effective.  You may potentially be tracking hundreds of individual IPs, each of which may only be used for a short time.  This is a lot of effort on your part to manage that data, and it is likely to generate a significant amount of false positive alerts which will consume analyst time.  

The adversary has effective countermeasures against this, too.  They can rent access to a botnet, giving them thousands of throwaway IPs, or they could simply use any of a number of anonymizing services such as Tor, I2P or even a VPN.  The best part (the worst for you) is that not only are these easy to use, but they are usually free or very low cost. It costs you a lot of resources to effectively track at this level, and it's free and easy for the attacker to counter your efforts.  Sound like a win?

Moving up the Pyramid, though, makes things a bit moe complicated for the attacker.  Detecting the network artifacts of their tools (e.g., a distinctive URL pattern or User Agent string they use when spidering your website) puts the burden on them to change up their toolset.  Once your monitoring structure is in place, creating new signatures for new tools is usually not too difficult, and can be done at only an incremental cost to you.  

If you can move even further up, perhaps operating at the TTP level, you're forcing the adversary to rewrite their playbook, something that is extremely time consuming for them.  Again, there will be a cost associated with gaining the ability to operate at this level, but once you pay it, the costs for future detections are likely to be incremental.  In other words, once you reach that plateau, you can stay there with modest resources.  On the other hand, every time you force the adversary to relearn new TTPs, they pay the cost.  Again, and again and again.

Integrating the Models

The preceding is really just a cursory review of a couple use cases, but it shows the value in combining the Pyramid and the Kill Chain.  Ultimately, though, you must do the actual integration yourself, using your own detection program's capabilities along with your own threat intelligence.  

I recommend starting with a single threat domain (a specific actor or group if you're dealing with targeted attacks, or for something more general, a topic like "Banking Trojans").  Comb through your intel for that domain and organize your indicator types by Kill Chain phases.  Some indicator types may fit in more than one phase, and that's fine.  You may discover gaps or ambiguities in your data, and that's OK, too.  Note these issues and follow up later.  

Once you have your indicator data arranged by Kill Chain phase, you essentially have a dossier on how that threat acts as it tries to accomplish its missions.  

Now revisit each Kill Chain phase in your dossier, and rank the indicator types according to their position on the Pyramid.  This is not entirely a precise mapping, so don't be discouraged if you have to make some best guesses.  Also, don't worry so much right now about whether you have any detection platforms in place that cold actually detect those types in those phases.  That's the next step.

After prioritizing the indicators for each phase according to the Pyramid, go back through and mark the ones you are currently detecting.  Also take note of the ones you have the technology to detect now, but may not actually be detecting (e.g., you have access to DMZ web server logs, but you are not currently mining them for indicators of hostile activity).  What you have marked are both your current detection stance and your likely candidates for short term improvements.  Anything that is not marked is a potential candidate for medium or long term improvement (depending on your detection needs and resources).

The End Result

If you've been following closely, you may have already figured out what all this is driving to, but just in case, I'll spell it out.  By sorting and prioritizing your Kill Chain dossier on a threat and applying the Pyramid to the potential indicators you have, you have developed a detection plan for that threat across your Enterprise.  

I cannot overstate the importance of this plan.  As a living document, you should constantly revisit it to update the intel and revise the priorities and detection capabilities that went into the analysis as your own detection program changes over time.  The document captures not only your current state, but also your goals and gaps.  It's a reference for where you are and a roadmap for where you want to go.  What's more, the compilation of detection plans for all your key threats will make your detection program leaner and more effective, because you'll be operating on actual data about your detection posture rather than guesses and generalizations.

The Thrilling Conclusion

Ok, I lied.  It's not really that thrilling.  But it is important. The combination of the Kill Chain methodology to organize your threat indicators and the Pyramid of Pain to prioritize your detection of them is an extremely powerful one.  The resulting document not only shows you a "snapshot in time" of your DFIR effectiveness against that threat, but also points out gaps which are potential areas for growth and improvement.  It is these plans which help an organization realize the full potential of the intel-driven model of detection and response.  Everything else is just a guess.

The Pyramid of Pain

Update 2014-01-17
I'm updating this post to include a slightly revised version of the Pyramid.  The only real change I made was that I added a new level for hashes.  I also updated the text to account for this.  

---

On February 18th, Mandiant put a major hole in the APT intelligence dam when they released their APT1 report profiling a group commonly referred to as Comment Crew.  There followed a small flood of reports from other entities like the Symantec (and this) and the DHS/FBI.  Most of the furor over the APT1 report was regarding it's findings suggesting that APT1 is actually the PLA's Unit 61398.  This is solid work, and a real breakthrough in the public policy area, but I was a lot more excited about the detailed technical information included in the reports' seven appendices (not including the video appendix).

After seeing how these indicators were being applied, though, I came to realize something very interesting: almost no one is using them effectively.  

I put that statement in bold, because it's a little bit of a challenge, and I'm sure it will surprise many readers.  The entire point of detecting indicators is to respond to them, and once you can respond to them quickly enough, you have denied the adversary the use of those indicators when they are attacking you. Not all indicators are created equal, though, and some of them are far more valuable than others.

The Pyramid of Pain

To illustrate this concept, I have created what I like to call the Pyramid of Pain.  This simple diagram shows the relationship between the types of indicators you might use to detect an adversary's activities and how much pain it will cause them when you are able to deny those indicators to them.  Let's examine this diagram in more detail.

Types of Indicators

Let's start by simply defining types of indicators make up the pyramid:

1. Hash Values: SHA1, MD5 or other similar hashes that correspond to specific suspicious or malicious files.  Often used to provide unique references to specific samples of malware or to files involved in an intrusion.
2. IP Addresses:  It's, um, an IP address.  Or maybe a netblock.
3. Domain Names: This could be either a domain name itself (e.g., "evil.net") or maybe even a sub- or sub-sub-domain (e.g., "this.is.sooooo.evil.net")
4. Network Artifacts: Observables caused by adversary activities on your network. Technically speaking, every byte that flows over your network as a result of the adversary's interaction could be an artifact, but in practice this really means those pieces of the activity that might tend to distinguish malicious activity from that of legitimate users.  Typical examples might be URI patterns, C2 information embedded in network protocols, distinctive HTTP User-Agent or SMTP Mailer values, etc.
5. Host Artifacts: Observables caused by adversary activities on one or more of your hosts.  Again, we focus on things that would tend to distinguish malicious activities from legitimate ones.  They could be registry keys or values known to be created by specific pieces of malware, files or directories dropped in certain places or using certain names, names or descriptions or malicious services or almost anything else that's distinctive.
6. Tools: Software used by the adversary to accomplish their mission.  Mostly this will be things they bring with them, rather than software or commands that may already be installed on the computer.  This would include utilities designed to create malicious documents for spearphishing, backdoors used to establish C2 or password crackers or other host-based utilities they may want to use post-compromise.
7. Tactics, Techniques and Procedures (TTPs): How the adversary goes about accomplishing their mission, from reconnaissance all the way through data exfiltration and at every step in between.  "Spearphishing" is a common TTP for establishing a presence in the network.  "Spearphishing with a trojaned PDF file" or "... with a link to a malicious .SCR file disguised as a ZIP" would be more specific versions.  "Dumping cached authentication credentials and reusing them in Pass-the-Hash attacks" would be a TTP.  Notice we're not talking about specific tools here, as there are any number of ways of weaponizing a PDF or implementing Pass-the-Hash.

The Pyramid Explained

Now that we have a better idea what each of the indicator types are, let's take a look at the pyramid again. The widest part of the pyramid is colored green, and the pinnacle of the pyramid is red.  Both the width and the color are very important in understanding the value of these types of indicators. 

Hash Values

Most hash algorithms compute a message digest of the entire input and output a fixed length hash that is unique to the given input.  In other words, if the contents of two files varies even by a single bit, the resultant hash values of the two files are entirely different.  SHA1 and MD5 are the two most common examples of this type of hash.

On the one hand, hash indicators are the most accurate type of indicator you could hope for.  The odds of two different files having the same hash values are so low, you can almost discount this possibility altogether. On the other hand, any change to a file, even an inconsequential one like flipping a bit in an unused resource or adding a null to the end, results in a completely different and unrelated hash value.  It is so easy for hash values to change, and there are so many of them around, that in many cases it may not even be worth tracking them.  

You may also encounter so-called fuzzy hashes, which attempt to solve this problem by computing hash values that take into account similarities in the input.  In other words, two files with only minor or moderate differences would have fuzzy hash values that are substantially similar, allowing an investigator to note a possible relationship between them.  Ssdeep is an example of a tool commonly used to compute fuzzy hashes.  Even though these are still hash values, they probably fit better at the "Tools" level of the Pyramid than here, because they are more resistant to change and manipulation.  In fact, the most common use for them in DFIR is to identify variants of known tools or malware, in an attempt to try to rectify the shortcomings of more static hashes.

IP Addresses

IP addresses are quite literally the most fundamental indicator.  Short of data copied from local hard drive and leaving the front door on a USB key, you pretty much have to have an network connection of some sort in order to carry out an attack, and a connection means IP Addresses.  It's at the widest part of the pyramid because there are just so many of them.  Any reasonably advanced adversary can change IP addresses whenever it suits them, with very little effort.  In some cases, if they are using a anonymous proxy service like Tor or something similar, they may change IPs quite frequently and never even notice or care.  That's why IP Addesses are green in the pyramid.  If you deny the adversary the use of one of their IPs, they can usually recover without even breaking stride.

Domain Names

One step higher on the pyramid, we have Domain Names (still green, but lighter).  These are slightly more of a pain to change, because in order to work, they must be registered, paid for (even if with stolen funds) and hosted somewhere.  That said, there are a large number of DNS providers out there with lax registration standards (many of them free), so in practice it's not too hard to change domains.  New domains may take anywhere up to a day or two to be visible throughout the Internet, though, so these are slightly harder to change than just IP addresses.

Network & Host Artifacts

Smack in the middle of the pyramid and starting to get into the yellow zone, we have the Network and Host Artifacts.  This is the level, at last, where you start to have some negative impact on the adversary.  When you can detect and respond to indicators at this level, you cause the attacker to go back to their lab and reconfigure and/or recompile their tools.  A great example would be when you find that the attacker's HTTP recon tool uses a distinctive User-Agent string when searching your web content (off by one space or semicolon, for example.  Or maybe they just put their name.  Don't laugh.  This happens!).  If you block any requests which present this User-Agent, you force them to go back and spend some time a) figuring out how you detected their recon tool, and b) fixing it.  Sure, the fix may be trivial, but at least they had to expend some effort to identify and overcome the obstacle you threw in front of them.  

Tools

The next level is labelled "Tools" and is definitely yellow.  At this level, we are taking away the adversary's ability to use one or more specific arrows in their quiver.  Most likely this happens because we just got so good at detecting the artifacts of their tool in so many different ways that they gave up and had to either find or create a new tool for the same purpose.  This is a big win for you, because they have to invest time in research (find an existing tool that has the same capabilities), development (create a new tool if they are able) and training (figure out how to use the tool and become proficient with it).  You just cost them some real time, especially if you are able to do this across several of their tools.

Some examples of tool indicators might include AV or Yara signatures, if they are able to find variations of the same files even with moderate changes.  Network aware tools with a distinctive communication protocol may also fit in this level, where changing the protocol would require substantial rewrites to the original tool.  Also, as discussed above, fuzzy hashes would probably fall into this level.

Tactics, Techniques & Procedures

Finally, at the apex are the TTPs.  When you detect and respond at this level, you are operating directly on adversary behaviors, not against their tools.  For example, you are detecting Pass-the-Hash attacks themselves (perhaps by inspecting Windows logs) rather than the tools they use to carry out those attacks.  From a pure effectiveness standpoint, this level is your ideal.  If you are able to respond to adversary TTPs quickly enough, you force them to do the most time-consuming thing possible: learn new behaviors.  

Let's think about that some more.  If you carry this to the logical extreme, what happens when you are able to do this across a wide variety of the adversary's different TTPs?  You give them one of two options:  

1. Give up, or
2. Reinvent themselves from scratch

If I were the adversary, Option #1 would probably look pretty attractive to me in this situation.

Effective Use of APT1 Indicators

Now that we've covered all that background, we can finally turn our attention back to the APT1 indicators.  I said that almost no one was making effective use of them.  What did I mean by that, and what constitutes "effective use"?

What I meant was that I see a lot of discussion about the long list of domain names included in Appendix D (and to a lesser extent, the domains and IPs included in the DHS/FBI and Symantec reports as well).  Seth Hall of the Bro-IDS project even put out a neat Bro module that you could use to search for those domains in your network traffic.  

This is all good and proper, but I haven't seen a lot of discussion centering around the data they provided on the behavior of the Comment Crew tools themselves.  Appendix A is a giant data dump of some very good information regarding host and network artifacts for 40+ tools known to be in use by this group.  

Emerging Threats has published a set of Snort signatures that cover many of these, and I'm quite sure many of us have produced our own, but I find the lack of attention to these curious.  Maybe the ET rules are already serving everyone's  needs and so there's no need to talk about them?  More likely, though, I think a lot of organizations have not properly reviewed the reports for detection indicators.

Whenever you receive new intel on an adversary (whether it be APT1/Comment Crew or any other threat actor), review it carefully against the Pyramid of Pain.  For every paragraph, ask yourself "Is there anything here I can use to detect the adversary's activity, and where does this fall on the pyramid?"  Sure, take all those domains and IPs and make use of them if you can, but keep in mind that the amount of pain you cause an adversary depends on the types of indicators you are able to make use of, and create your plan accordingly.