FireStarter: The Only Value/Loss Metric That Matters

@Patrick, Well, we certainly agree on one thing: No one has perfect data But I think we have worse data than, for instance, medical research field. I should've said "there's very little reliable information" in my last post. At least that is my impression. I perceive the $x/rec estimates that are published as mostly guesswork. Informed guesses, maybe, but still biased. I question the methods for coming up with these numbers more than I question the various risk calculation methods we put them into afterwards. That "much of the medical data that is published .. leaves a lot to be desired" is no surprise, this goes just as well for a lot of scientific articles. But I am assuming that they would not be published, if the reviewers had no faith in the collection methods. Can we trust the scientific validity of the TJX or Ponemon data? If not, then we might see them as interesting, and they might give useful insights or pointers in the right direction. But the minute we assume a certain accuracy, we're in trouble. And that false sense of accuracy occurs as soon a $ figure is on some manager's Powerpoint slide, or we put it into a risk calculation. We can never eliminate uncertainty, it just seems to me that the uncertainties are currently so large that our attempts to be exact fail.

@Anders - I think that we agree about many things. I haven't ever said anything much about being exact - I wouldn't think that exactness is attainable. Regarding accuracy, there are degrees. To drive from Dallas to Austin, I don't really need a high degree of accuracy - just a little bit of data will enable me to get the job done - get onto I-35, as opposed to I-45 or I-20, and go south as opposed to any other direction. Whether TJX or Heartland are misrepresenting their costs in the SEC filings or not? I don't think so - they are reporting what they have actually spent and estimating future expenses by creating reserves. If you read successive filings, you see that the reserves go up and down based upon how the cost estimates look in each quarter. Harder to know are the accelerations in CAPEX that may occur because of a breach. Also, cost per record may not be the most relevant common denominator fro breach cost - it doesn't work so well for loss of intellectual property, for example. The key here is reduction of uncertainty - if you don't think you can do that enough to make better decisions, then there isn't much else I can say. But it's the reason I advocate quantitative methods, probability distributions, and looking at data across a range of distributions and estimates. BTW - Have you read Doug Hubbard's books? "How to Measure Anything" and the "Failure of Risk Management"? If not, I highly recommend them for giving an interesting perspective on these issues. These are difficult problems - no question about it - Dan Geer is on record saying that infosec is one of the most intellectually challenging activities out there. But if we keep saying that nothing is possible, then nothing ever will be possible Best regards, Patrick Re: medical data - you would not be pleased to know, I don't think, how much junk gets past peer review - I have reviewed thousand's of peer-reviewed clinical articles in medicine - the data presented are often very problematic, but lots of times the article is accepted for publication just because of the author's name.

@Anders - I disagree - There is actually quite a lot of information available - check the SEC Filings from TJX, Heartland, and others (Forms 10-Q and 10-K). The numbers they report are informative. Or check the Maine Breach report - it breaks some things out for TJX and Hannaford. Even read the Ponemon reports and try to understand the context of those data. I don't think that the evidence supports Ponemon's estimate of approx $200/rec - even so, there is much to be learned there. Loss magnitude is one of the easier things to develop data for. Threat capability, threat frequency, and the effectiveness of controls/defenses is much more difficult, but still possible to model. I think that it depends upon what you think measurement means and is for. I think that it's for the purpose of reducing, not eliminating uncertainty. That's also what the statistical functions are for - modeling the uncertainty and variablity - two different things, by the way. I would suggest to you that your statement about statistical functions misses this point. With regard to your last statement that it doesn't matter, I also strongly disagree. No one has perfect data - doctors don't, engineers don't, scientists don't, and we don't. I worked in medical outcomes research for 17 years and will tell you that much of the medical data that is published in peer-reviewed journals leaves a lot to be desired. Best regards, Patrick

Patrick, since we have no sound method of actually measuring loss magnitude, methods like FAIR, FIRM or whatever are not going to work much better than others. Applying statistical functions successfully implies knowledge of statistical distributions, but as others have pointed out, there's very little information available, thus denying us such knowledge. So until we find that way of getting data that's both accurate and plentiful, which risk calculation you choose is somewhat insignificant.

@jeff - This is a very interesting line of inquiry. The law is still catching up with the reality, I think. As I recall, about a year ago, a Federal judge hearing a class action suit against Hannaford Bros. disqualified all but one plaintiff. The plaintiffs had alledged financial harm to do efforts and worries over possible identity theft. The judge initially took the point of view that since federal law limited financial loss to $50, and since only one of the plaintiffs (the one he allowed to stay in the suit) had demonstrated any financial damage, the other class plaintiffs had no grounds to sue. Basically, he seemed to be saying: "No harm - no foul - no right to sue" But then, some weeks later, the judge reversed his ruling and pushed the issue up to a higher court to decide. I asked my daughter, a lawyer, about this, and her response was helpful. She said that without any actual damages to work from, no matter how sympathetic a judge might be, there is simply no basis for pulling a number out of the air. I don't know where the Hannaford matter stands today - maybe someone else can update. Patrick

I don't think anyone would argue the premise that predictions should meet reality. And I think there are two basic points here: 1) our predictive models for expected loss stink 2) feedback into those decisions (data gathered from breeches) stink too Completely agree, and both are required for meaningful change, if I lay out a pretty good process: Step 1 is we try some predictive process: if a tornado hits, we'll lose X. Step 2: we get a pulse on reality: when tornado hits, count the nickels and dimes, compare to X. Step 3: revise original predictive process to deal with second tornado. The gotcha of course is that a tornado may never come so it would be great if we could learn from neighbors down the road, or update the predictive process for tornados hitting by learning from a tape backup failure (for example). I don't disagree at all with what's been said here, but I do think we're doing the right thing by a) trying stuff b) getting feedback and c) talking about trying stuff. Even something as silly as the ALE process provides us a place to start comparing our theory with our reality. I don't think the problem lies in risk models trying to be predictive and place value on loss, but the problem is that there's no feedback, no process for improvement, and in doing so make more claims to their accuracy than are not supportable by data.

>>> We don

Well, whether you call it ALE or something else, the basic idea is that we want to predict what something is going to cost us. While I agree that the (current) concept of ALE is misguided, if you replace "guess the value of your data" with "insert actual cost" then it's basically the same procedure you are going to follow to fulfill Rich's requirement. Because somewhere down the line, you are going to have to decide which risks to mitigate and which to ignore. That requires you make up your mind about what is going to happen to you and what is not, thus forcing some more or less accurate estimate of likelihood. And estimating the worth of your assets is going to be just as tough after the fact as before. In all likelihood, we must therefore settle for vague estimates for the foreseeable future. Preferably (in my book) without quantitative metrics. Having said that, the world is not entirely without risk frameworks that move in the right direction. IIRC, Information Security Forum has a risk analysis methodology (FIRM) that bases it's estimates for likelihood/probability on actual events. Ie. not "how often will this happen?", but "how often has this happened?". If you could extend that to also as the same question for actual loss, we would be getting somewhere. And whatever way you try to estimate your risk: the lack of real incident data is going to be Bump-In-the-Road #1. To change that requires widespread industry information sharing, which I guess is sort of a "holy grail" of infosec. And if we can achieve that, then I am not so sure that "traditional" risk analysis methods won't work well after all.

You've rather lost me... the post starts out as a criticism of ALE (fine, easy target), but then concludes with "...I have yet to see a risk framework with a value/loss model that meets this basic requirement for information assets." Rothman further adds on "But I

All of the concerns that have been raised about estimating impact are legitimate. Part of the problem with many approaches to-date, however, is that they've concentrated on asset value and not clearly differentiated that from asset liability. Another challenge is that we tend to do a poor job of categorizing how loss materializes. What I've had success with in FAIR is to carve loss into two components -- Primary and Secondary. Primary loss occurs directly as a result of an event (e.g., productivity loss due to an application being down, investigation costs, replacement costs, etc.), while Secondary loss occurs as a consequence of stakeholder reactions to the event (e.g., fines/judgments, reputation effects, the costs associated with managing both of those, etc.). I also sub-categorize losses as materializing in one or more of six forms (productivity, response, replacement, competitive advantage, fines/judgments, and reputation). With the clarity provided by differentiating between the Primary and Secondary loss components, and the six forms of loss, I find it much easier to get good estimates from the business subject matter experts (e.g., Legal, Marketing, Operations, etc.). To make effective use of these estimates we use them as input to PERT distribution functions, which then become part of a Monte Carlo analysis. Despite what some people might think, this is actually a very straightforward process, and simple spreadsheet tools remove the vast majority of the complexity. Besides results that stand up to scrutiny, another advantage is that a lot of the data you get from the business SME's is reusable from analysis to analysis, which streamlines the process considerably.

Ben, I have studied FAIR, OCTAVE, and whatever else I can get my hands on. Every framework has to have a loss/valuation component at some point. ALE is the simple example, but isn't alone.

Jack, Good point- I shouldn't lump FAIR in quite the same way since I like how you've split the losses and try to use multiple input points to develop the estimate. What's nice is that someone can break out the categories and loss types and then evaluate post-incident losses using the same framework. Have you thought about making this post-incident analysis part of FAIR? (Apologies if I've missed that part and it is already in there).

Rich - It's too bad you missed MiniMetricon 4.5 as we talked a bit about this very topic. Pete Lindstrom provided a good talk based on Douglass Hubbard's books (in particular, his "How to Measure Anything"). Ranges and confidence are key, and help shake out much of the concern you've expressed. fwiw.

Rich, I'm not sure that I follow the point of your response to Ben. Yes, as you state, every framework has to have a loss component at some point. So the question becomes whether that component of a framework is reasonably effective. Do you believe it's impossible to effectively characterize the loss component of a risk scenario, or do you just think the infosec profession has done a poor job of that to-date?

Rich, Sorry. Our posts seem to have "crossed in the mail" so to speak. You can delete my question about your response to Ben if you like. Very glad to hear that my categorization strikes a chord with you. We've had excellent buy-in from business management with the approach, and analysis of losses from actual incidents fits nicely within the framework, which helps validate the categories and allows us to do a decent job of leveraging empirical data where it exists. Unfortunately, I haven't had (made?) time to update the documentation I've made public about FAIR, so a lot of people aren't familiar with some of the improvements that have taken place since the original white paper was written. Thanks, Jack

Hi, Rich - It's nice to hear you say something somewhat nice about quantitative approaches. They next thing I hope to hear you say someday is that qualitative approaches are almost completely worthless and misleading. There are a number of ideas that I might suggest here. 1) Focusing on the value of assets is not always the right thing to do because it's not always where the real value/risk is - rather, the value/risk is sometimes the loss exposure, realized or as yet unrealized, of a compromised asset, the value of a lost or compromised business process, data store, protected information, etc. As I understand it, and some accounting types might wish to weigh in here, according to GAAP (Generally Accepted Accounting Principles), the book value of "information" is limited to the cost of creation and maintenance of the information. In the event of the sale of a company, additional value of information may be recognized as "good will". This value is in many cases far less than the "value/cost" of the information if it falls into the wrong hands. As we know from TJX ($170-250M so far), Heartland ($140M so far), and others, the costs of dealing with a large data breach are huge (even if not close to $200/rec that some assert.) I wonder which was greater for TJX or Heartland - the cost of creating and maintaining the information, or the loss exposure that came about because of the breaches? Just a question - I don't know the answer. With regard to a business process, maybe a company has a $10M investment in IT that generates $250M in revenues. The value of the asset may not even come close to the exposure created by losing the process. 2) Concerning models out there, you and I have talked about FAIR, which is one model that produces consistent, reproducible estimates. There are other ways to do this, too. I guess now that I am 60 years old, I might as well say what I think - the lack of broader experience that becomes evident when talking to many infosec practitioners is a big problem. An even bigger problem - really appalling in my view, is the willingness of many infosec practitioners to issue "pronouncements" based upon this state of ignorance. (I am not particularly shooting at you, here) And, the lack of intellectual honesty and curiosity that is apparent with many infosec "rock stars" is probably the biggest obstacle of all. Actuaries, insurance companies, oil and gas companies (even BP), and many others have for decades been doing the sorts of quantitative risk analyses that infosec says are impossible. We need to look outside, as Adam Shostack has advocated, and learn from others before deciding what is or is not possible. 3) Too many people are looking for "the answer", rather than a range of reasonable estimates that help to reduce uncertainty. In my view, the whole purpose of risk analysis is to reduce uncertainty in a way that leads to better decision making. If you wish to try to convince me that qualitative methods do a better job, I am willing to listen. 4) You are absolutely correct that every method needs to be tested against measurable outcomes. I know how to do this with a quantitative approach. It is not at all clear to me how this might be accomplished in a meaningful way with non-quantitative methods. Best regards, Patrick Florer

Patrick... I argue that the vast majority of quantitative risk assessments I've seen in infosec are little more than qualitative risk assessments with dollar signs added to wild ass guesses. Thus they are even more worthless and deceiving than a model that admits a guess is a guess. I didn't reiterate it in this post, but my philosophy on risk assessment is quantify as much as you can, qualify where you can't accurately quantify, and combine them in a consistent fashion to communicate overall risk. I don't believe either is "right" on it's own. It's easy to say we should learn from other industries, but it isn't so easy. As I'll detail in the next response, information assets are fundamentally different than physical goods, which is why we have the problems we do.

This is turning into a great discussion. Bravo to all those participating. At the risk of being the wet blanket man, I don't think we've addressed Rich's point about going back and comparing **actual** loss to the loss predicted by the (various) models. As evidenced by the discussion, there are lots of ways to estimate potential losses. Many will be defendable and pass muster of the business folks. The real question is the accuracy of the estimates. We can provide ranges and confidence levels all day and night, but unless we close the loop and actually figure out the real accuracy of the model, we are still practicing black magic. Not science. I'm not familiar with any attempts to compare estimated loss to actual loss. Can anyone share an example? Mike.

I want to clarify something that isn't as clear as it could be from my original post, which responds to a couple of comments... 1. Calling for a model to validate predicted losses with experienced losses applies to any model or type of loss. This is the non-controversial part of the post, other than almost no one does it. 2. I do believe you can measure a number of loss vectors- costs to replace physical items, response costs, legal costs, etc. 3. I do not believe there is any way (currently) to consistently measure the dollar value of the information asset itself. We can equate its value with the loss categories we *can* measure, but that's not the real value of the asset. That's the part we can't measure, but it's also where I see a lot of infosec risk assessments get completely derailed as people make up numbers which are essentially qualitative expressed as quantitative. I only slightly called out point 3 directly in the post since if you agree with the prediction/experienced tenet, 3 emerges naturally.

Mike nailed it, and brings us back to what I intended from the start. Show me a data valuation model where the predicted value matched the measured value after a loss event. I'm not saying that's impossible, but I haven't seen it done.

Actually, Mike, I have (unfortunately?) had an opportunity to validate loss estimates for a couple of events where I've worked. The estimates fit quite well. If I were still working there, and if you were under NDA, I'd be happy to share the details. Those who've worked with me though, can at least corroborate my assertion. Of course, one of the challenges with any relatively new method is that it takes time to establish enough history in their use to strongly substantiate (or not) their effectiveness. In the meantime, we're left with evaluating them based on the logic/reasonableness of their approach and whatever data are available and become available over time.

With regard to actual loss - In the cases of the big breaches I have followed from 10-K filings as well as press reports, it's clear to me that these costs unfold over time - years, in fact - and that the estimates and set-asides change and go up and down. Just take a look at the TJX 10-K's for 2007, 2008, and 2009. In addition, there are capex components that may be involved - accelerated spending, delayed spending, etc., that make it hard to tell what the costs are. I don't really agree about learning from other disciplines - although I have worked in IT for 30 years, what really turned on the lights for me with regard to risk was the 17 years of that 30 years that I spent part-time in clinical outcomes research - that's where I learned my statistics, Bayesian techniques, and decision analysis stuff. Since then I have accelerated my studies in statistics and quantitative risk analysis. Monte Carlo techniques and probability distributions are not hard to use correctly, even if you cannot do the math by hand. Mike, very few things in our modern world aren't black-box-like, wouldn't you agree? Modern cars are a completely mystery to me, as are iPods and even refrigerators. But they work. I am probably an exception to the rule - an old-fashioned generalist with fairly deep skill. I never saw this coming 40 years ago when I graduated from UT Austin with a degree in Classical Greek.

This is a fascination discussion, but one thing jumps out at me. There has been quite a bit of discussion here about determining loss, the probability of bad things happening, and other associated factors when trying to determine risk, but one comment/question was made about determining the value of our information. Not the value of loss, but the value of the information to the organization. Call me naive, but shouldn't our business partners be able to tell us what the value of their information is? Notice I said their information. It would seem to me that since we see constant forecasts for revenue and expectations for profit, we (they) should be able to tie that back to the value of the information they maintain in some meaningful manner. I'm not saying it should be or is easy, but it seems imminently reasonable to me, not that I have accomplished this in my own organization. It is definitely giving me something to think about though. Kevin

Hi Kevin, I believe you're right that our business colleagues should be able to tell us (at least roughly) what the business value of the information is. That said, that information is only relevant if the scenario we're analyzing involves either the loss of that data (as in, it goes away) or damaged integrity of the information. If the data is still in our possession and we're still able to generate/realize its value in our business processes, then losses tend to be associated with liability (i.e., secondary loss from stakeholder reactions) and the costs associated with responding to the event. Consequently, it becomes important in our analyses to distinguish between the different types of events (confidentiality vs. integrity vs. availability). Jack

Kevin, Very interesting! I would submit that information has no value at all except in its use or mis-use. On the positive side, it's the value of the business process that the information enables that matters. On the loss side, maybe it's a bit more complicated - your business process could be compromised due to loss of information or processing capability. Or, your information could fall into the wrong hands and create any number of liability scenarios. Or both. IT hardware assets, given the rapid pace of change, also have little value, except in their use or mis-use. Forget about what is carried on a company's fixed asset ledger as book and depreciated value. Once you install a server or a data center, what is it really worth if it isn't doing anything to support business processes? Very little. Have you ever tried to sell a used server or a data center or software? This point was driven home to me about 25 years ago when the company I worked for shut down suddenly. It was a small service bureau that had about $1M worth of medium scale mainframe and DEC minicomputer hardware - that was $1M cost carried on the books. At auction, the $1M of hardware fetched less that $40k. Patrick

As per usual, context is everything, eh? Letters have little to no value until formed into words. Words have some value, but not nearly as much (generally) as when they're chained to form sentences and paragraphs and so on. It's not the representation, but the contextual interpretation or use that is important.

Ben, I really like that - it speaks to the linguist in me. Patrick

This fire has turned into a blaze, I haven't seen this lively a discussion in a while. Fun! I think I see two points playing together here. The first is a model to predict the value at risk and the second is a model to predict the probability of a loss occuring. Predicting the probability of a loss feels easier, and in the physical world it is. Hence, I disagree with Mike's comment: >> I agree that we can

@ds - There already is a framework for doing this. You don't need two models. It's called FAIR. It addresses both of the issues you bring up: event frequency and loss magnitude. I don't think you have risk without both of these. FAIR uses Monte Carlo simulation, the structured solicitation of subject matter expert opinion, actual data when available, and probability density sampling functions to provide ranges of estimates for a variety of parameters. The Jack Jones who has been active in this discussion is the originator of FAIR. But, you don't have to use FAIR. Just go to Wikipedia and do a little reading on risk, Monte Carlo simulation, and probability distributions. Or, buy and read one of Doug Hubbard's books ("How to Measure Anything" and "The Failure of Risk Management"), download his Excel sample files, and start fooling around. You don't have to have a PhD in statistics or Ops Research to build effective models that will help to reduce uncertainty and lead to better decisions. It will be a good thing when more people realize how easy it is to do this kind of stuff, how defensible it is, and how useful it is. With regard to sharing information, I wonder? Do insurance companies share data? Perhaps someone else can clear this up. Patrick

"value/loss model that meets this basic requirement for information assets' how do you value the loss/damage of a breach of a persons privacy impacting reputation ,employment or simple right to privacy etc once the information is "out there" , eg health information- is it persistent for ever?

Rich points out the difficulties of valuing assets for the purposes of an ALE-type of analysis. But I think getting close to the "annual rate of occurrence" is even harder than getting to asset value. Yes, there are some events (like a lost laptop) where we have plenty of data. Then we can model those out for both value and occurrence. But what about those events/incidents, which cannot be modeled? Your proverbial black swans like a massive data breach or a weaponized zero day attack. Just as it is hard to estimate the value of the asset being impacted, it's even harder to figure out when/if those kinds of events (massive potential loss, seemingly very small chance of occurring) will occur. Which is another way of stating why we think ALE is crap. You don't know the value of the asset, and you don't know how often a certain event is going to happen. Hmmm. Seems like a risk analysis foundation built on quicksand to me. But I'm sure the risk modelers out there will tell me Bayesian estimation factors all this in, eh? Mike.

> But what about those events/incidents, which cannot > be modeled? Your proverbial black swans like a massive > data breach or a weaponized zero day attack. Putting a dollar value on the loss experienced by a zero-day attack is like putting a dollar value on the loss experienced by a tornado. You simply can't do that, nor should you try. The thing you need to try to measure is the impact. A tornado can do a lot of damage, but if it misses your building, your loss is zero. (Perhaps you can measure lost productivity during the tornado drill, but that's not what we're talking about here.) If the tornado takes your entire building out, now you can measure. Because you just lost building, and that has a real cost. Same for hardware, stolen laptop, compromised system you have to rebuild, etc. I think the real tough question here is "How much is your data worth?" And I wish I had a good model for that. Perhaps my business users can shed light on it...

John, You've nailed the problem, and what inspired this post. We don't have a way to value the vast majority of IP/data in any consistent fashion. It's like every piece of data is a work of art or family photo.

@john, I agree that we can't put a dollar value on the loss of a tornado, but most of the risk models try to do exactly that, which was my point. Agree that valuing the loss is challenging as well. You can paint a worst case scenario, but what about direct costs (replacement, clean-up, disclosure, etc.) vs. indirect costs (brand damage, etc.). How much do you model in there? Again, no one knows the answer here and that's the point of the post. Mike.