The first and most important point that we feel a need to convey is that
we REALLY ARE saying that sniffer-driven intrusion detection systems
are fundementally flawed, and (from what we can tell at this point) cannot
work in the manner they are advertised to.
The next important statement we're making is that ALL the systems we
reviewed have bugs that will currently allow an attacker to silently slip
past them. The bugs we conclusively identified and tested in our paper
are not due to fundemental flaws in protocol analysis, but rather to an
apparently remarkable lack of software testing applied to these products.
Some of the problems we identified can be fixed. For instance, one way to
evade all of the ID systems we tested was to send traffic in IP-fragmented
packets. This is due to the fact that (with the exception of Network
Flight Recorder), none of the vendors we tested thought to implement IP
fragmentation reassembly in their products. We expect that, in light of
the seriousness of these issues, most IDS vendors will have working
fragmentation reassembly implemented soon.
However, we do not see a way in which sniffer-driven ID systems can
accurately detect SPECIFIC TYPES of attacks in IP traffic. We are not
contesting the fact that it is possible to detect traffic that is likely
"malicious", and we are not saying that it is impossible to detect the
fact that a network is being attacked. The issue is that sniffers cannot
isolate (most types of) specific attacks from any other type of attack.
The reasoning behind this (clearly bold) statement is that there are
ambiguities stemming from the meaning of individual packets that can only
be resolved with secondary sources of information. For instance, there are
situations in which you can only determine conclusively what a packet
means if you know what operating system is running on the destination
host.
The technical basis for this (particular) problem is that different
operating systems have different, slightly nonstandard TCP/IP
implementations. The inconsistancies between different operating system
TCP/IP implementations and the implementation of the IDS can be exploited.
This is one of the fundemental technical points of our paper.
Another misconception being propagated by one of the vendors mentioned in
our paper is that the paper is either a "commercial review" of ID systems
(with rankings, point scores, and purchasing recommendations), or a
"comprehensive assessment" of the tested ID systems. Neither of these
statements are accurate, and both are immensely damaging to the security
community.
The reason we mentioned specific ID systems in the paper was because we
felt our points would be driven home best by providing empirical evidence
of the types of bugs we demonstrated. Because the paper is a technical
report, meant to provide information to the community about intrusion
detection technology, we provided some comparitive analysis between
different systems (ie, "it is interesting that system X implemented this
function in this manner, as opposed to function Y").
However, one thing we specifically did NOT do was recommend any one system
over another. The only technically credible statement that can be made
about the results of our (cursory) comparitive analysis is that we
recommend Network Flight Recorder as a result of their (excellent) policy
of providing free source code to the public. In terms of actual
performance and reliability, no system did "better" or "worse" than any
other. All had basic, devastating bugs, and all systems (when used as
network intrusion detection systems) have fundemental problems that we
don't expect to see fixed any time soon.
Another thing we did NOT do was provide a comprehensive security analysis
of different ID systems. This is always an important point to make when
discussing the publication of security flaws in software, because it is
easy to think that the release of security vulnerability information means
that there are no other problems that we could have identified in the
product other than those we explained.
This was a technical paper, not an audit report. There are likely numerous
other problems with all the systems we tested, just as it is likely that
there are numerous problems in any piece of network software. What the
industry needs to see happen is comprehensive testing of products; when
this happens, some claims about the overall reliability and security of
these intrusion detection systems will be credible. Until then, we can
only continue to point out the specific problems.
Thanks for listening to this. The feedback we've received has been
encouraging and appreciated, and we welcome as much input as you can
possibly provide us with. One thing the security community certainly
cannot be harmed by is public discussion, scrutiny, and peer review. Just
as (we feel) the community has benefited from our contributions to the
review of security products, it will benefit from your contributions to
the review of the validity of our results.
Have a great weekend.
-----------------------------------------------------------------------------
Thomas H. Ptacek Secure Networks, Inc.
-----------------------------------------------------------------------------
http://www.enteract.com/~tqbf "mmm... sacrilicious"