So much of security is about doing it, not just having it. The best products can be useless in the wrong hands: you have something that's supposed to be blocking, but you have it only in logging mode. Or you have it in blocking mode, but you only enabled a few of the available rules. You have it installed in the wrong location on your network because that's the only place you could put it. It tells you what you need to know, but you never have time to look at it. And so on.
I believe that most of security relies on detecting and controlling change. And there are so many aspects to change that have to be considered.
- Recognizing change. Do you know how your systems, operations and usage patterns are supposed to be? This is probably the biggest challenge, believe it or not. Just knowing your baseline, in everything that happens in your environment, will be impossible for any one group or team. This baseline knowledge is institutional, and will be spread across staff at every level. The network team may know what normal traffic looks like, but they may not know what makes it normal.
- Detecting change. This might sound like it's the same as the item above, but it's not. Detection implies both recognition and timeliness. It requires finding what has changed, when, in what way, and by how much.
- Understanding change. This is the next step in the line: if you know what has changed, and the details, then you should be able to understand the root cause of the change (someone was trying to restore a database) and the implications of the change (it clobbered the production version and now it has to be rebuilt, before the stock markets open in the morning). Or you realize that the change happened for a particular reason, and it will likely be followed by other changes (a connection is made to an unknown external system, and your sensitive data is about to head in that direction).
- Initiating change. Making a change happen is often interwoven with politics. Why do you want to initiate the change? Are you allowed to request it? Who will execute the change? These are all big questions when, for example, you are trying to get a security hole plugged ("Turn off that SMTP relay! Now!!").
- Designing change. If you understand the effects of a change, you may need to make sure it happens with certain timing, in a certain sequence, on certain systems, done by certain people. You have to design the change, especially if it's a complex one -- say, a migration off of Windows XP.
- Controlling change. Making a change happen the way you designed it to happen, within the time frame that you need, without political or organizational fallout, is harder than you think. It may take longer than you hoped for that application's vulnerabilities to be patched. If changes need to be approved by other entities, you may need to persuade them to give that approval. The person assigned to execute the change doesn't actually know how to do it, and is going to do it wrong. Reverting a change could be more complicated than making the original change.
- Preventing change. Most people think that security is all about this part, but it's not. Any business requires change, and it's up to the security team to help ensure that change happens in the right ways. In some cases it's still important to stop some changes from occurring (such as malware being deposited), but the prevention has to be fine-grained enough to address only those change cases. For example, this setting can be changed, but it should never be done on Sundays. This rule can be changed, but only by people in this role. Or something can be changed, but it has to create a notification for someone else who needs to know about it. These types of changes should not be made unless they are logged.
Business rules, as well as risk management decisions*, will dictate how changes are initiated, designed, approved, executed, recorded and evaluated. And if you don't have a good handle on these business and risk requirements, you'll be severely hindered in detecting unauthorized changes and responding to them.
You can't use malware detection if it takes too much work to figure out what a false positive is. Rather than being able to understand those changes, you'll ignore alerts about them until a bigger change happens as a result. (Or until you get a call from the Secret Service.)
There's no point in running a vulnerability scanner if you can't cause fixes to be made based on the findings. Moreover, you shouldn't use a vulnerability scanner as a "compliance detection" tool: it may tell you that your systems are configured exactly the way your policies specified and they haven't changed, but you may have stupid policies.
If you can't figure out the effects of a change, then you will either try to prevent it out of fear, or you will execute it wrongly. If you don't know how a change is made, you can't design a process that will limit collateral damage.
Luckily, you can start mastering all these aspects of changes without spending a lot of money. Just knowing what your systems, applications and users are supposed to do is a huge start towards effective security -- and it takes time, but it's cheap. After that, the road will be different for every organization, because the changes, effects and control points will be different. There will be things you can't change, changes you can't detect, or business processes that constrain (or mandate) change. Look at what power you have over change, to figure out how secure you can be.
*Notice I didn't say "security best practices."
