RootSense is the first holistic intrusion detection system for Linux. This daemon is the first attempt at detecting at-entry intrusions at the scope of four subsystems—the process, filesystem, memory, and networking subsystems. This is done by having an inner and outer monitoring layer. The outer layer monitors activity based on known bad behavior and the inner layer monitors for behavior that deviates from normal behavior that on its own would seem like malicious activity. These activities are sequences of events. These events may be system calls, or other types of events like load spikes in network bandwidth. These sequences can describe all system actions and were chosen as being the main ingredients to the signatures utilized by the rootsense daemon. Signatures are described using a special "attack sequence language" that allows describing known vulnerabilities at the granularity of system calls and specific events. This makes it rather straightforward to design and describe vurnulbilities if one could monitor the stream of events that occur doing a system comprimise.

Introduction

An intrusion detection system (IDS) is software that is capable of detecting malicious activity caused by a misuser. With the present state of security software, most IDS focus on one subsystem. These single subsystem centric IDS's can at best hope to know everything that is going on in their domain. For example, how can Tripwire detect a buffer overflow? Or how can Snort detect a rootkit being installed locally? The answer is as simple as they cannot. In fact, despite the popularity of some of these IDS they often will report false positives when no intrusion has occurred and even worse will sometimes report false negatives when a cracker is already having his/her way with their system. The solution is to monitor the global state of an operating system. Only by knowing as much as possible can a system make the best decision as to what is most likely occuring in a system at any given time. Rootsense achieves this while keeping cpu utilization low.

Architecture

The rootSense IDS has a centralized architecture where one to many sensors gather events and pass them along to a central analyzer object. Each sensor is a specialized object that gathers events from a particular subsystem. Once a sensor has its events it has the ability to filter out events and pass them along to either another sensor or directly to the central analyzer object. The sensor interface(base class) used a policy patten to define the filtration and escalation policy for a sensor instead of using inheritance. This was chosen because no two sensors should or most likely will share filtration and escalation polcies. Furthermore, it allows polymorphism without the performance cost of inheiritance. Moreover, because events are always being created and destroyed, their memory management is conrolled through an object pool patten that has protected member creation/destruction methods in the sensor base class. Once events reach the central analyzer, they are quered and later dispatched to a state machine that will monitor a good or bad signature object. When a state machine reached its final state it notifies the central analyzer and appropriate actions are taken. It should noted that state machines are not terminated in the solution. Although this is a problem it is solvable.

Events

Events are described with 6 fields. The first field classifies the subsystem that the event is associated with. This can be a P for process, F for filesystem, M for memory, and a N for Network. The next field in the event is the process id, followed by the user id field, and then the effective user id field. The next field is a name that describes the event (usually a system call name), followed by any parameters that pertain the name field. These fields for the most part can have lowercase letters that are variables, fields that can be prefixed with an exclamation mark to negate them, and fields may even have an asterisk (*) to denote a wildcard. Furthermore, events can utilize sets either as set variables or using curly braces ({,}).

These are example events,

P p * * fork * This is a fork system call for any process p. F p * 0 { open , creat } f This is an open or creat system call for any process p.

Example Signature and Interpretation

This is a "time of check to time of use" exploit (race conditions).

F p * 0 { open , creat } f F !p * * unlink f F !p * * { symlink , link } f * F p * 0 { read , write , chown , lchown } f

In this signature we have 4 events (one on each line). The first line says that we are looking for any process p that opens or creates a file f. Second, we look for any other process (not p) that deletes f. Third, we again look for any other process (not p) that creates a symbolic link or link to a file f with any parameters. Finally, if the process p reads, writes, chowns, or lchowns a file f then we have detected malicious behavior.