Conferring about conferences.

There's a great discussion going on right now on Twitter about what's wrong with security conferences:  do we have too many?  Are they focusing on the wrong things? 

Josh Corman threw out the figure that more than 60% of conference paper submissions these days were on Android security issues.  This sounds pretty excessive when you consider all the other security topics out there.  However, let's not forget that there are many different audiences for security talks, just as there are different sub-communities within the security industry.  For "breakers," Android security is a hot topic these days, and you would expect to see a lot of talks on mobile security in general at conferences "by breakers, for breakers."  And because that's a hot topic among breakers, you'll see defenders and builders eyeing it as well, because in the security ecosystem, what's getting targeted the most is what everyone will tend to focus on.

That's not to say that security conferences are homogeneous.  There is a very different culture and flavor at work at a conference for defense-related security (law enforcement and military, and to some extent critical infrastructure), as opposed to a meeting of financial services CISOs, or civilian government, or academia, or "hacker ethos" tribal gatherings.  Even if the hot topics are nominally the same, the perspectives and timbre of discussions will be very different.  And a conference that features roundtable discussions will bring out information exchanges that aren't as readily forthcoming at classic "stand up and present" functions (even if you count the hallway track).

So even though the sheer number of security conferences these days is dizzying, I think the variety is healthy.  We need the grass-roots B-Sides just as much as the vendor-oriented RSA, or the raucous Shmoocon, or the Chatham House Rules-driven CISO roundtable.  If anything needs to be changed or tweaked, I simply think that we need to make sure that the same speakers aren't touting the same perspectives at all of these different venues.  Everyone wants to hear a sexy war story about mobile every so often, but I really admire the efforts to bring in first-time and local speakers to certain events as well.  The "democratization" of security conferences is a trend that I'd like to see continue.

Too many questions.

As an analyst, I have too many things I'd love to research and can't.  I'm in a target-rich environment (then again, so was Custer).  It doesn't stop me from coming up with questions, though, and hoping someone else will want to answer them.

Take the discussion I just had on Twitter with @jeremiahg, @chriseng, @attritionorg, @dakami, @rybolov and others.  I objected to the claim that everyone in the Fortune 500 is hacked, in the absence of two things:

1. A clear definition of "hacked," and
2. Some data supporting the assertion that everyone in the F500 fit that definition.

So we got to talking about what data would constitute proof, and I suggested that having one host in your IP range detected as being a member of a botnet could qualify as "hacked."  This could theoretically be straightforward to determine, if you had access to enough threat intelligence feeds and/or had enough sensors to compile a list yourself.  Now, there are some open source feeds, but for the most part companies that create their own feeds want to monetize them. (One laudable exception is Microsoft, which has been testing a feed that it would offer free of charge to law enforcement, CERTs, foreign governments and private corporations.)  If you have one machine on a botnet at some point in time, that could designate you as hacked, at least until you scrubbed it. 

But is it the tip of the iceberg?  Does having a bot automatically mean that more nefarious things are going on besides just selling V1agr4 or perhaps DDoSing the Anonymous target of the week?  This is the risk calculation that we need more data to perform, and it's one that the C-suite would really appreciate.

So I'd love for someone to comb through their incident response data and present statistics on what, if anything, followed after an initial malware infection.  If you could say that (for example) 70% of the time, it was simply used to grab CPU without necessarily trying to grab passwords or data, and 20% of the time it led to password compromise for financial theft, and 10% of the time it led directly to IP theft, those would let us infer probability.  It would depict in a more concrete way just why being part of a botnet is a symptom of something more dangerous.

By association, any company that found itself with membership in a botnet could reasonably suspect that it was even more compromised than that.  It might take the time to look further.  (There are plenty of enterprises that just wipe the affected machine, re-image it, and go back to work.)

The other question is whether membership in a botnet should be considered public data.  If anyone on the Internet can discover it, you could argue that it's the kind of compromise that anyone can report.  The fact of an enterprise's system interacting with another host on the Internet isn't confidential; it (like a public posting) is just assumed to go unnoticed.  Would a company have grounds to complain if its membership in a botnet were revealed, based entirely on publicly available information outside of its private network?  I am not a lawyer, but sometimes I want to ask lawyerly questions like this.

Following this chain of thought, anyone could set up sensors, collect data on botnet membership, and publish it widely.  Someone could collect statistics on just how many of a company's systems were in a botnet at any given time.  In the absence of any other data, could this be used as a poor man's Compromise Index?  It would be like someone noting how many broken windows you could see in a building: one indication of a breach, but without any way to know what, if anything, happened or was taken after the windows were broken.

And armed with that data, someone could actually make a substantiated claim that the whole Fortune 500 is hacked, without hearing the clackety-clack sound of thousands of eyes rolling.

After that comes the question, "So what?"  Would this kind of naming and shaming prompt any additional diligence on the part of these organizations?  Would it make regulators sit up and take notice?  Call me a skeptic, but I suspect that botnet membership is so widespread that people would assume it happens to everybody -- just like ant invasions -- and it wouldn't be condemned except within the security echo chamber.  I could be wrong.  Either way, I'd love to find out.

[DISCLAIMER: I am not encouraging anyone to compromise any systems themselves without the permission of the affected organizations.  I am not suggesting that anyone collect data that can only be gathered directly from those systems.  I am certainly not recommending that anyone leak confidential data, even if it's with the best of intentions.  Do not try this at home.  Ask your parents before calling.  And so on.]

For great justice. I mean security.

The Verizon Data Breach Investigations Report (available here) was basically another year of "all your POS are belong to us."  Which is depressing, but not at all surprising.  As you know, I talk a lot about what I call the Security Poverty Line, and how smaller organizations that are IT-poor tend also to be security-poor.  Moreover, because security and IT are so often separate, security becomes optional, a luxury and an omission for the small business that doesn't know it has something to lose -- or even if it does, it hasn't got the faintest idea of how to go about addressing it.

Enter the DBIR, and what I think is one of the most helpful steps ever taken to address this security-poor population.  On page 62, the redoubtable Verizon Risk Team has created a cutout sheet that you can hand out to your favorite retail, hospitality and food establishments.

Greetings. You were given this card because someone likes your establishment. They wanted to help protect your business as well as their payment and personal information.
It may be easy to think “that’ll never happen to me” when it comes to hackers stealing your information. But you might be surprised to know that most attacks are directed against small companies and most can be prevented with a few small and relatively easy steps.

 And the cutout doesn't get too fancy or preachy; it basically recommends two main things:  change your default passwords, and make sure you have a firewall.  And if you're not the one who is in charge of these things, make sure your vendor does them.

The beautiful simplicity of this is hard to overstate.  The cutout doesn't invoke FUD; it just says, "Hey, we've seen a lot of this and you might want to be careful."  The language makes it accessible to someone who is busy running a business, and who doesn't have time to delve into arcane IT concepts.  It tells them the most important things they need to do, and puts it in a digestible format.

I hope people will go to the trouble of making copies of this cutout and giving them to as many franchises and local businesses as possible.  It would also help to have a simple and cheap answer to the question, "How do I find out more about this?" if the business owner should ask.  I know of at least one security professional who makes a point of going to speak about security at chamber of commerce meetings, and we need more of this kind of outreach.

For the security-poor organizations, the best thing we can start with is to arm them with information -- the kind of information that is useful to them.  If we made a concerted effort to reach out to this underserved population, I'm hoping the DBIR numbers would get smaller over time.

Going back to the stack.

In the spirit of trying to suggest solutions, here are a couple of thoughts about what an enterprise can do first off to make security a little better.

It's bothered me that infrastructure is being administered more horizontally than vertically these days.  Everyone specializes in a different layer: network, OS, utilities (such as Exchange), middleware, applications, etc.  And this gets worse when you outsource one or two layers to "the cloud" (think IaaS, PaaS and SaaS), so that you have to coordinate with a third party to troubleshoot something.

Back in the Pleistocene era, system administrators knew their systems like they were their babies.  They knew everything that was running on them, how they were configured, who they talked to, and they knew when something was "off."  I know sysadmins that would regularly help a developer debug code, and they were often better at it than the developer, because they also understood the underlying environment better.  They could troubleshoot all the way up and down the stack, and you went to one source to do it instead of having to get a conference call together with 3rd level engineers from four different companies.  (Seriously, I know of a data center that had five different networks owned by five separate entities.  Think you could figure out what happened to a packet?  Think again.)

So one thing that enterprises can do is simply to get control of their layers as much as possible.  Know what you have, know where it is, and be able to cause changes to it when you want to.  That sounds so obvious as to be not worth saying, but I don't know of any admins who know more than about 500 hostnames by heart, and many times the environment is so dynamic that boxes come and go without any centralized tracking keeping up with it.  (And I'm not even talking about VMs.)

If you already have parts of your infrastructure outsourced, go over your contracts and strengthen your relationships with your providers.  You want them to be able to give you logs, for example, within a few minutes of the request.  You also need to have the right technical level support people on call without having to fight your way through first-level script-readers.

And finally, go back to designating "stack admins," who are generalists rather than specialists in one particular technology.  It should be their job to know as much as possible about any given system.  You can fit this into DevOps if the developers truly know the lower layers.  A stack admin is your best hope for knowing what normal operation is, and for alerting you when something doesn't smell right; they're also the best at understanding the implications of any given planned change (such as changing the ports an application uses without creating the corresponding firewall rules).

Start with knowledge, and then work your way to control.  Notice we haven't really touched on security yet; that'll come later.  But knowledge and control are basic building blocks of security.

In 50 gigabytes, turn left: data-driven security.

I love Scott Crawford's research into data-driven security.  I agree with him that IT operations and development can both benefit from the right security data -- where "right" means at the appropriate level and relevant to what they're doing.  It also has to be in the right mode:  an alert should be based on a conclusion drawn from the analysis of data (20 failed logins per second = someone is using automation to try to break in), based on an event or confluence of certain events.  Once someone in IT needs to perform an investigation, the need changes to looking at more atomic data (exactly which logins are being targeted, whether they're active or disabled, etc.).  In other words, the details need to be available on demand, but they shouldn't be shoved at the IT staff in lieu of useful alerts.

Another kind of data that is useful is situational data:  how things are configured and what is happening during "normal" operation.  Viewing all the responses from a database is too much to ask of a developer -- but the developer would benefit a lot by knowing that some queries are taking 25 minutes to return (do you suppose that would have some effect on application performance?).  This is the sort of data that is incredibly useful, but setting up every possible abnormal situation to trigger an alert is way beyond the scope of an overworked operations team.  Sometimes you just have to sit down and do some exploring every so often, to find out these sorts of operational problems.  Packet captures can teach you things you can't learn any other way -- if you have the time and skills to read them.

Because detection is expensive.  It requires the luxury of having staff both knowledgeable in the technology and in the context of those particular systems, and having them devote a lot of their time just to sitting and looking at things, sorting out what's normal from what's not.  Those are the kind of costly eyeballs that have been transferred so frequently to managed security service providers.  It's the kind of thing you pay consultants to do, because if your staff weren't completely occupied with keeping the infrastructure running, you wouldn't be allowed to keep them.  Data analysis today is expensive, and it's a one-off deal unless you can find economies of scale somewhere.

Yes, automation is getting better, but it's not there yet.  There are still too many alerts taking up too much time to sort through (particularly in the tuning phase).  IT staff get hundreds of emails a day; they can't handle more than two or three alerts that require real investigation.  (By the way, this is why operations often can't respond to something until it's down -- it's the most severe and least frequent kind of alert that they receive all day, and they don't have time to chase down anything lower-level, like a warning message that hasn't resulted in badness yet.)

If you break security events down, you're generally looking for two kinds of things:  normal activities that are being done by the wrong people (as in, from a Tor exit node through your administration console), or abnormal activities that are being done by the "right" people (internal fraud, or someone has taken over an authorized account).  And by "people," of course, I also mean "systems," but at first glance it's sometimes hard to tell the difference. 

This determination of "wrong" and "right" is a security activity, and for the reasons I listed above, operations people may not care that much until it makes something happen that they have to fix.  If someone wipes a database, they'll care a whole lot, but if there's some unusual encrypted traffic leaving the enterprise on port 80, not so much.  A fully leveraged (i.e. overworked) ops team doesn't have time to analyze alerts at that level.

"Wrong" and "right" to the business is on a completely different stratum, and it's one that's hard for automation to reach today.  Executives care when it gets to the level where they have to do something about it, like fire someone for looking at patient data, or talk to the press about a breach.  They care when an event starts to present the risk of legal liability or increased cost.  But you can't bring them alerts like that until you have digested everything at a lower level and put together enough evidence to reveal a business issue.

And finally, historical data can be extremely useful in determining what works in security and operations and what doesn't.  But that kind of data has to be analyzed in a different way from real-time operational data or situational data.  It requires a different model that caters to the requirements of risk analysis -- and that, too, is expensive, even assuming you know how to do it today.  (Hi, Chris.)

My point here is to say that data-driven security is where we need to go, absolutely.  But there is no single path to take with the data we have; there are a number of divergent paths that are all needed in the enterprise.  We also need to be able to drive the data in the right delivery directions -- which means that we need a really good data navigation system.

Security: ur doin it rong.

As I mentioned before, a lot of security work consists of telling people they're doing something wrong.  There are all the "thou shalt nots" in security policies, there's the "scanning and scolding" of vulnerability assessment, and there's the "Ha! Got you!" inherent in penetration testing and exploit development.

In other words, it takes a lot of moxie (pun intended) to stand up to a security professional.

Rob Lewis, aka @Infosec_Tourist, made the comment yesterday:

You're right. Nobody says "we're screwed!" with as a sincere and calm demeanour as @451wendy.

Which I appreciate, but it's been bothering me lately that that's almost always how we discuss security.

In his preso at Security B-Sides London last year, David Rook (aka @securityninja) made a great point about application security:  if we taught driving the same way we taught secure development, we'd make a whole big list of different ways you could crash the car, but never actually tell the student how to drive safely.

A good number of talks at security conferences focus on what we (or other people) are Doing Wrong.  Very, very few are about how to do something right.  Part of the reason for this, of course, is that practitioners are afraid to stand up in front of an audience and talk about how they're defending themselves, for fear that someone in the audience will take it as a challenge and de-cyber-pants them before they've even gotten to the Q&A session.  (I've heard tell of presenters' laptops being hijacked in the middle of a presentation.)  I know a lot of practitioners are doing very cool things that their management would never let them say publicly.

But when we focus too much on what people are doing wrong, there's a danger of our talks turning into pompous lectures.  "We need to do something different from what we're doing today."  Okay, but what, exactly?*  This is why I admire those who are proposing alternative solutions, such as Moxie Marlinspike's Convergence.  These folks might be right, or they might be wrong, but at least they're trying to make things better.

So, lest this turn too Gödel, Escher, Bach on us, I'll stop lecturing too, and talk about what I plan to do about it.  I'm going to do more talks about what I think works in security.  I've done a few before on topics such as how to bootstrap an infosec program, what multi-contextual identity and access management looks like, and how to dicker on the contract with third party providers.   I won't aspire to #sexydefense; I'll leave that to the ones who show up all the time on the Top Ten Infosecsiest lists.  But I'll encourage people to turn that frown upside down, and try not to bring up a problem without also proposing a solution.  


Maybe this way, we can get invited to a few more non-security parties instead of having to throw them all ourselves.


*No, the answer is NOT "use our product."  Thanks for playing, though.

Insecure at any speed.

With the release of breach data reports, such as the one from Trustwave SpiderLabs that came out recently and the highly anticipated one from Verizon Business, inevitably comes a wave of data dissection and then disbelief.  Security pundits moan at the statistics, such as the one this year that 78% of organizations that Trustwave investigated had no firewalls at all.  The report itself takes an incredulous tone as it describes the pervasive use of unencrypted legacy protocols (one highlighted case study described a breach involving an X.25 network), insecure utilities such as telnet and rsh, and more.

Security pros who specialize in this sort of thing may be surprised at how big the problem is, particularly among smaller enterprises, but anyone who has actually tried to implement security in these organizations isn't surprised at all.  You can tell by the faces in the audience when one of these talks goes on:  it's the difference between "ZOMG!" and "Yup, *sigh*."

It's not that these organizations don't care about security.  You'd have to know about security first in order to care about it.  The next time you go to a sandwich shop or a gas station, ask the manager about the security in the POS system they're using.  It should be an interesting, but very brief, exchange.

Should everyone be able to manage their own security?  It's very much out of reach for those below the security poverty line; when you think about it, the level of security management needed for technology today reaches the equivalent of having to rebuild and restock grocery shelves on a weekly basis, or requiring an accountant to know construction, electricity and plumbing for the office.  Just reading through the Trustwave report, and all the myriad ways that systems are breached, I can't help but imagine the look on a manager's face if I made it into a checklist and handed it out.  Who outside of the clannish IT industry knows how to spell ftp, much less knows that it's insecure?  Who would know the better options and be able to implement them? Who has the time to examine and reconfigure computers on a regular basis?

What this indicates to me is that our IT infrastructure -- from the networks to mobile -- is inherently, badly insecure.  And we're so far down the road in its widespread implementation that it will be decades before the problem is substantially fixed, even assuming we started today with all software developers and manufacturers.  Nobody is going to pay to replace what's running just fine today -- until someone loses a figurative eye.

As technology advances, organizations have to deal with an ever-widening range of technology that they have to try to get secured.  Yes, there are still X.25, COBOL, VMS, DOS, NT, SunOS, Sybase, and token ring out there. At the same time, iOS and Android are coming into play, along with "the cloud" and Hadoop and NoSQL and everything else that's new.  A CIO needs to know about all these; a CISO has to know how to secure them all -- especially when older systems aren't compatible with updated software.  The complexity grows year by year, and the inertia of the legacy environment weighs more heavily on it.

And make no mistake: security is disruptive.  It's enormously disruptive.  Getting the network architected correctly, every version of software patched and every configuration right, especially after the system has been in use for a while, is as disruptive to the business as migrating to a completely new system or platform.  Ask anyone who has tried to manage a security initiative in an enterprise.  Even assuming the enterprise wants to do it, it's a major undertaking.  All this shows how badly security is designed today; you shouldn't have to keep reconfiguring your systems on a weekly or monthly basis in-flight just to keep the security entropy at bay.

It's an intractable problem, and frankly, it's one that the enterprise shouldn't have to solve.  People are trying to work with the equivalent of a pencil, and it's not their fault that their pencils are fragile, complicated, and prone to exploding at inopportune moments.  They shouldn't have to know or care why the pencil isn't working; they want a new one without any delay, and without hearing long stories about how the graphite in this type of pencil isn't backwards-compatible with all the erasers in the firm.

So when we read about how bad security is getting, we shouldn't be pointing the finger at the compromised enterprises.  We should be pointing it at their IT providers, who really ought to know better; but more fundamentally, we should be pointing it at ourselves.  We should stop demanding that the user be responsible for security; those of us who are building this stuff to begin with should fix it ourselves, and build it in to all future technology.  Today security is an afterthought, and a bad one at that.  As long as it remains separate from the systems it's supposed to protect, instead of being simply an attribute, and as long as it requires users to maintain an abnormal height of awareness as they go about their daily jobs, security is going to continue to be as bad as it is today.