Use of the term "Intelligence" in the RSA 2014 Expo

I attended RSA 2014 this week, and one of the things that struck me was the recurrence of the term "intelligence" on many vendor booths.  I decided it would be a fun exercise to go through the expo halls to ask the vendors to clarify what their uses of the term "intelligence" meant.

Methodology

First, the parameters of the exercise.  I chose an arbitrary starting expo hall (honestly, never once in the whole week was I able to accurately remember which was North and which was South) and walked the aisles from one side to the other, examining each booth to see if they referenced either "intel" (not the chip maker) or "intelligence".  I did not consider booths that only had related terms like "information" or "sharing", nor did I consider booths for vendors I know play in the intel space but neglected to include the term in their display (I did stop at a couple of non-vendor booths that fell into this category, though.  More on this below.).

At each booth, I explained who I was and what I was doing, and asked if there was anyone there who could answer a few quick questions about their use of the term "intelligence".  Some of the vendor representatives were more well-equipped to answer my questions than others, but in all cases I let them decide who I should talk to, in order to try to avoid polluting the results with my own personal biases about who would make a "good" representative.

After establishing contact, I then pointed out their use of "intelligence" on their display and asked, "Can you explain what you mean by that?" If their answer seemed to roughly line up with the idea of "using information to detect malicious behavior" I then asked followup questions listed below, otherwise I thanked them for their time and ended the interview.

• What types of information do you consider to be "intelligence"?
• Are some types more valuable than others, either inherently or in certain circumstances?
• How can your customers know they're getting the maximum value out of their intelligence?

Vendors

In total, I visited 10 vendors and 2 non-vendors.  In fact, I had planned to visit more, but the expo halls are so large that I didn't even complete the tour of one hall during the short amount of time they were open on Thursday (the day I conducted this exercise).  

The vendors I visited were:

1. Webroot
2. LogRhythm
3. R-sam
4. IBM
5. NetIQ
6. Arbor
7. Solutionary
8. AlienVault
9. Securonix
10. BAE Systems

The non-vendors I visited were special cases, because although neither mentioned "intelligence" on their displays, I felt the organizations had enough expertise in the intelligence field that their perspectives might have been useful:

1. Homeland Security
2. National Security Agency

Unfortunately, neither of these entities were able to discuss their thoughts about intelligence, as that was not the purpose of their booths and neither brought any experts in that area, so I include them here only as an interesting sidenote.

Full disclosure:  I work for the Mandiant division of FireEye, both of which are well known for their threat intelligence.  I purposely left them out of this survey.  I did this not to throw criticism on other vendors, but because I have a much deeper knowledge of what FireEeye considers to be "intelligence" and I couldn't effectively include them in the survey without biasing the results towards my employers.

Uses of the term "Intelligence"

Out of the 10 vendors, I found variations of 8 different uses of the term "intelligence".

1. Threat intelligence (4 vendors)
2. Security intelligence (2 vendors)
3. Identity intelligence (2 vendors)
4. File intelligence (2 vendors)
5. Application intelligence (2 vendors)
6. Risk intelligence (1 vendor)
7. Applied intelligence (1 vendor)
8. Insider threat intelligence (1 vendor)

Most vendors stuck with one of the above, or a close rewording that meant the same thing.   One vendor (Securonix) actually used several different variations in their display.  Their representative explained this by saying "we add 'intelligence' to the end of everything."  In fact, if I had taken additional uses of "intelligence" from our conversation, I would have added several more to the list above.  This would have been breaking my own rules, though, so I omitted them.

Different meanings of "Intelligence"

The definitions of "intelligence" broke down into three categories:

1. "Intelligence" in the sense "doing something smart with input data to achieve a result" (i.e., what is also often referred to as "analytics", "anomaly detection" or just "correlation").  Terms used this way included "security intelligence" and "risk intelligence".
2. Enriching input data to allow security decisions to be based on more organizational context than was originally present in the data set (e.g., adding user identity information to incoming log events).  Terms used this way inlcuded "identity intelligence", "file intelligence"and "application intelligence".
3. Consuming information about adversaries, tools or techniques and applying this to incoming data to identify malicious activity.  The term "threat intelligence" was the most commonly used phrase in this category, although "insider threat intelligence" also applied.

Readers of my blog will note that definition #3 most closely matches up with what I consider to be "intelligence".  It's worth noting that Chris Sanders defines #2 as "Friendly Intelligence" in chapter 14 of his book, Applied Network Security Monitoring (Full disclosure, I was a contributing author, though not of that chapter).

Types of "Intelligence"

For those vendors who's use of the term "Intelligence" fell in line with definition #3 above, I then asked about the types of intelligence they deal with. By far, the most common were IP addresses and domains, though URLs were also sometimes mentioned.  The recurring ideas of "file" and "application" intelligence strongly implies the existing of file hash values as well.  I will not attempt to summarize how many vendors mentioned each type, primarily because many of the vendor representatives I spoke to either weren't willing or weren't able to go into detail about the types of intel data they dealt with.

Applying the Pyramid of Pain model to responses shows that the respondents' use of threat intelligence still falls mainly into the bottom half of the pyramid.  The few vendors that mentioned URLs may be working at least partially in the artifacts level as well.  No vendor in my survey mentioned any type of indicators that would fall into the upper levels of the pyramid ("Tools" or "TTPs").

It's interesting to note that most vendors who use definition #3 cited primarily network-based data types (IPs, domains, URLs).  A few mentioned "file" or "application" intelligence, which implies more of a host-based orientation, but no one mentioned traditional host-based indicators such as file names, registry keys or processes names.  (One vendor did mention file names, but in conjunction with definition #1).  This may indicate a gap in our industry's thinking about what types of information can be useful in detecting malicious activity, it may be a function of the types of products that the vendors in this survey are selling, or a combination of the two factors.

Other interesting things of note

I mentioned earlier that both Homeland Security and the National Security Agency had booths, but weren't able to comment on their ideas of intelligence.  In fact, I did get one quote from the NSA representative, which I thought was interesting:  "Information doesn't become intelligence until it is useful to someone."  I interpret this to mean that the information also has to be consumable (information buried in a PDF report isn't that useful; it needs to be put into detection mechanisms).  Since there is often a lot of confusion about the difference between information and intelligence, I think this is a nice way to phrase the difference so that people can understand.

Some of the individual vendor representatives also touched on a similar theme, drawing the distinction between "information" or "facts" and "intelligence".  For example, Webroot mentioned that they have databases of "facts" like IP or file reputation, but that they have a process that combs through those databases to try to find connections and correlations and place them in context with other related facts.  The ouput of this process is what they consider "intelligence": facts in context with each other.

The representative from Solutionary also had an interesting point of view.  He described a hierarchy of "technical indicators" which are facts about the state of something, independent of possible security concerns (e.g., "the system is out of memory"), "threat indicators" which do have security implications, and "threat intelligence" which is a combination of the two, with additional higher-level context.  The hierarchy goes something like:

technical indicators < threat indicators < threat intelligence

I've seen the "threat indicators < threat intelligence" before, and I think there is broad agreement on this among actual intelligence analysts, but I was unfamiliar with the concept of lower level technical indicators, although they seem pretty obvious in hindsight.

Conclusions

This turned out to be a pretty interesting exercise.  The sample set is by no means large enough to constitute a reliable study, but I do think it has some valid things to say about our industry's approach to "intelligence" in general.  I draw the following conclusions:

1. "Intelligence" is a buzzword that can mean anything you want it to mean.  In my sample of 10 companies, there were 3 separate definitions (broadly speaking, probably more if you scope the definitions more narrowly).  That's a lot of variance given the small number of respondents.  It'd be interesting to expand this to a much larger set of vendors to see how many other definitions we can collect.
2. There is a valid case for the concept of "friendly intelligence".  Of the three definitions, two of them actually did refer to the use of some sort of information to make it easier to detect malicious activity.  Definition #2 is what Sanders calls "Friendly Intelligence" though none of the vendors I spoke to used this term. It does a good job at disambiguating the term "intelligence" and clearly indicates the idea of intelligence based on information you generate about yourself versus information about your adversaries.  This is an important concept, and by naming it, we make it easier to identify and understand.
3. We are focused on the wrong types of threat intelligence. Most of the vendors' concepts of threat intel were solidly on the bottom half of the Pyramid of Pain, which suggests that the indicators we're focused on are the ones that are the least valuable to the adversaries.  This, in turn, means that our incident detection and response operations are purely following the adversaries' lead and playing to their strengths.  Instead, we should be developing tools and techniques to allow us to develop and apply intel near the top of the pyramid, where we can increase the attackers' costs of doing business against us and make them work harder (and expend more resources) to accomplish their missions.
4. No one has any idea if we are using intelligence effectively.  Or even what "effectively" means in this context.  Although I had three followup questions prepared (listed above), I rarely got to ask the final one.  The questions were designed to follow each other in logical succession, so if the vendor couldn't or wouldn't answer one of them, I skipped the succeeding questions.  No vendor was able to successfully provide an answer to the second followup question, "Are some types more valuable than others?"  I was definitely not expecting anyone to parrot back the Pyramid of Pain or anything like that, but was hoping for some indication that certain types of indicators had different characteristics in terms of false or true positives, applied to more specific or more broad classes of attacks, or at least in general that not all types of data were of exactly equal use in detection or response.  I didn't get that from any vendor, which leads me to believe that the idea of "throw it all at the wall and see what sticks" may still be the dominant paradigm in many of today's security solutions.

I wish I could say I was more surprised by these results.  I suspected that result #1 was probably true, which was the original reason for this exercise.  If you have read my previous blog articles, you will know that I have been saying #3 and #4 for some time, and these findings to tend to confirm my views.  Result #2, though, I offer as a constructive finding.  The term "intelligence" itself is neutral; it is neither malicious nor benign.  We are used to the phrase "threat intelligence" as the kind that deals with malicious activity, and there is broad acceptance of this term in the industry. However, there is no equivalent term for information about oneself that can be used to help identify malicious activity, even though many vendors are clearly expressing this concept in their own different ways.

I had a lot of fun doing this: I got to meet a lot of new people, have some interesting conversations and even walk off some of this fantastic San Francisco food!  My thanks to all the folks I talked to while doing this research.  Perhaps I will try this again next year and see how the results compare.