Evaluating Security Tools

There's no stock answer to the question of whether or not a sysadmin should install and run a given security tool. It is advisable, as one certification firm suggests, to "Choose Your Weapons Wisely."¹ While evaluating various tools for coverage in this book, I kept the following attributes in mind:

• Is source code available for this tool?
  There's no way to audit or thoroughly evaluate a security tool without source code. While there are many fine commercial tools that don't provide source, I eliminated them from coverage in these pages. Closed proprietary products require a high level of trust in the supplier. Without access to source code, there's no way to know for sure that holes aren't introduced by the product and that the product doesn't contain back doors. For more discussion of this issue, see The Benefits of Open Source. In general, security safeguards -- from door locks and safes to a cryptographic method -- shouldn't rely on design and implementation secrecy.
• Is this tool easy to install?
  The concept of "easy" is tricky in this context because none of the tools in the entire UNIX security corpus are no-brainers to install. They all require some forethought and effort. In many cases, tools that I thought might be very useful would not install at all on my reference machines. In other cases, tools were very elegantly designed and implemented -- installation was painless enough that I could devote time and mental energy to configuration and use rather than compiler errors and such.
• Can this tool be configured to specific circumstances?
  No two UNIX systems are identical in every way. A good tool is flexible -- the designers considered the various environments the tool would be used in and provided for customization.
• Is this tool reliable?
  An unreliable security tool can be worse than no security tool at all since it may provide a sysadmin with a false sense of security.
• Is this tool reasonably easy to use?
  The regular operation of a tool should be a relatively simple matter. Human nature resists using complex or overly difficult tools -- a tool that is too cumbersome won't get used and thus won't provide needed protection. Also, the tool should produce readable output. Succinct reports are preferable to voluminous ones since the data reduction capabilities of any one person are limited. The Crack tool is a good example of a tool that provides relevant output: account so-and-so has a weak password, here it is. This question illustrates the general principle that security safeguards must be accepted in order to be effective.
• Is this tool cost effective?
  Both time and money are valuable. Every tool provides some sort of benefit. The benefit needs to be weighed against the cost of implementing and running the tool. In some cases, a tool might degrade performance to an unacceptable extent. This is a common problem, for example, with some firewall implementations.
• Is this tool maintained?
  There are a number of tools available that for one reason or another are no longer maintained by the original authors. In general, security safeguards should be sustainable over time -- an old tool that's been "orphaned" may not be a good fit in a global security environment that's constantly in flux. On the other hand, sometimes old tools can be very useful, particularly on old systems -- Dan Farmer's COPS tool was first written in 1990 but I was still able to use it to find several holes in one of my vintage 1990 NeXT cubes. Occasionally, old tools are picked up and improved by new maintainers. An example is the SATAN network probing tool, which was written by Dan Farmer and Wietse Venema and then updated by programmers at an outfit called World Wide Digital Security, Inc. (and re-christened "SAINT").
• Does this tool depend on other (possibly insecure) programs?
  The UNIX way is to combine different programs to achieve a desired result. This can backfire when a security tool depends on complex UNIX programs that were not designed for security purposes. Wietse Venema tells the story of setting a booby trap for an intruder that was breaking into and deleting systems at Eindhoven University of Technology.2 The trap relied on fingering the host the attacker was using and mailing the result to root. Later, Wietse realized that relying on the finger and mail programs (neither of which was designed for security) could open up security holes. The attacker could have included shell escapes in his .plan file which could have been picked up by the booby trap and sent on for execution by root. The moral of the story: good security tools either stand alone or minimize their trust when relying on other programs.
• Is this tool portable across different UNIX implementations?
  Many sites run a variety of UNIX systems. A cleanly coded tool should be portable across systems. If written for platform A, it should be possible to port it to platform B. Of course, there are fine platform-specific tools, so this is only a general rule.
• Does this tool do any harm?
  This should go without saying. "Do no harm" should be operating assumption number one for any security professional. A number of security tools have recently come to market that feature "strikeback" provisions. Modeled after Department of Defense systems, they can detect an intrusion attempt and then automatically launch a denial-of-service counter-attack. I believe the widespread deployment of this kind of harmful tool is a bad idea. It will be all too easy for one of these systems to attack an innocent party. In fact, it opens new lines of attack for crackers. A cracker on system C could masquerade as system B and then attack system A, knowing that A will automatically retaliate against B. This is known as "killing two systems with one packet."