The Architecture
The Architecture Document for IPSec, RFC2401, defines the base architecture upon which all implementations are built. It defines the security services provided by IPSec, how and where they can be used, how packets are constructed and processed, and the interaction of IPSec processing with policy.
The IPSec protocols—AH and ESP—can be used to protect either an entire IP payload or the upper-layer protocols of an IP payload. This distinction is handled by considering two different "modes" of IPSec . Transport mode is used to protect upper-layer protocols; tunnel mode is used to protect entire IP datagrams. In transport mode, an IPSec header is inserted between the IP header and the upper-layer protocol header; in tunnel mode the entire IP packet to be protected is encapsulated in another IP datagram and an IPSec header is inserted between the outer and inner IP headers. Both IPSec protocols, AH and ESP, can operate in either transport mode or tunnel mode.Because of the method of construction, transport mode can only be used to protect packets where the communications endpoint is also the cryptographic endpoint. Tunnel mode may be used in place of transport mode, and in addition may be used by security gateways to provide security services on behalf of other networked entities (for example, a virtual private network). In this latter case, the communications endpoints are those specified in the inner header that's protected and the cryptographic endpoints are those of the outer IP header. A security gateway decapsulates the inner IP packet upon the conclusion of IPSec processing and forwards the packet to its ultimate destination.
As noted, IPSec may be implemented in end systems or on security gateways such as routers and firewalls. Typically this is done by directly modifying the IP stack to support IPSec natively. When access to the IP stack of a machine is not possible, IPSec may be implemented as a "Bump in the Stack" (BITS) or "Bump in the Wire" (BITW). The former is typically a shim that extracts and inserts packets from the IP stack. The latter is typically an external, dedicated crypto device that may be independently addressable.
Security Association
To properly process IPSec packets it is necessary to have a way to associate security services and a key, with the traffic to be protected, and the remote peer with whom IPSec traffic is being exchanged (in other words, how to protect the traffic, what traffic to protect, and with whom the protection is performed). Such a construct is called a "Security Association" (SA). An SA contains the state necessary to do IPSec processing on an IP packet.
An IPSec SA is unidirectional. That is, it defines security services for one direction, either inbound for packets received by the entity, or outbound, for packets that are sent by the entity. SAs are identified by a Security Parameter Index (SPI)—which exists in IPSec protocol headers, the IPSec protocol value—either AH or ESP, and the destination address to which the SA applies—which dictates the direction. Typically, SAs exist in pairs, one in each direction. They may be created manually or dynamically. SAs reside in the Security Association Database (SADB).
When created manually, an SA has no lifetime. It exists until it is manually deleted. When created dynamically, an SA may have a lifetime associated with it. This lifetime is generally negotiated between the IPSec peers by the key management protocol. A lifetime is important because the amount of traffic protected by a key, or similarly the time that a key remains active and in use, must be carefully managed. Excessive use of a key can give an attacker an entry into your work.
Policy
The IPSec Architecture defines the granularity by which a user may specify his or her policy. This allows for certain traffic to be identified coarsely and have one level of security applied while allowing other traffic to be identified more finely and have a completely different level of security applied. For example, one may specify IPSec policy on a network security gateway that requires all traffic between its local protected subnet and the subnet of a remote peer be encrypted with AES and authenticated with HMAC-MD5, while all telnet traffic to a mail server from the remote subnet requires encryption with 3DES and authentication with HMAC-SHA, and all Web traffic to another server requires encryption with IDEA and authentication with HMAC-RIPEMD.
IPSec policy is maintained in the Security Policy Database (SPD). Each entry of the SPD defines the traffic to be protected, how to protect it, and with whom the protection is shared. For each packet entering or leaving the IP stack, the SPD must be consulted for the possible application of security. An SPD entry may define one of three actions to take upon traffic match: discard—do not let this packet in or out; bypass—do not apply security services to an outbound packet and do not expect security on an inbound packet; and protect—apply security services on outbound packets and require inbound packets to have security services applied. SPD entries that define an action of protect will point to an SA or bundle of SAs that identifies the state used to protect the packet.
IP traffic is mapped to IPSec policy by selectors. A selector identifies some component of traffic and may be either coarse or fine. IPSec selectors are: destination IP address; source IP address; name; upper-layer protocol; source and destination ports; and a data sensitivity level (if an IPSec system also provides for flow security). The values of these selectors may be specific entries, ranges, or "opaque." A selector in a policy specification may be opaque because that information may not be available to the system at that time. For example, a security gateway that has an IPSec tunnel with a remote security gateway peer may specify that (some of) the traffic that goes through that tunnel is IPSec traffic between two hosts behind the gateways. In this case, neither gateway would have access to, say, the upper-layer protocol or ports, since they would be encrypted by the end hosts. Opaque may also be used as a wild card, indicating the selector applies to any value.
If an SPD entry defines protect as an action and does not point to any existing SAs in the SADB, those SAs will have to be created before any traffic may pass. If this rule is applied to inbound traffic and the SA does not exist, the IPSec Architecture requires the packets to be dropped; if this rule is applied to outbound traffic the SAs can be created dynamically using the Internet Key Exchange (IKE).
The IPSec Architecture defines the interaction of the SPD, the SADB, with the IPSec processing functions—encapsulate, encrypt, integrity protect and decapsulate, decrypt, integrity verify—and defines how various IPSec implementations may exist. It does not, though, define how the base IPSec protocols operate. That is left for two different documents, one to define the Encapsulating Security Payload (RFC2406) and one to describe the Authentication Header (RFC2402).
Anti-Replay
Both IPSec protocols provide an antireplay service to prevent against a denial of service attack in which old packets are resent by an attacker to cause the recipient to waste CPU cycles processing them. This protection is not explicitly part of the architecture but is germane to both protocols and, as such, will be described here. IPSec packets are protected against replay attacks by using a sequence number and a sliding receive window. Each IPSec header contains a unique and monotonically increasing sequence number. When a SA is created, the sequence number is initialized to zero and prior to IPSec output processing the value is incremented. New SAs must be created prior to the sequence number wrapping around back to zero—prior to 232 packets since the sequence number is 32 bits long. The receive window can be any size greater than 32 but 64 is recommended. For performance reasons, the window size should be a multiple of the size of a word on the computer on which IPSec is being implemented.
The left end of the window represents the sequence number of the beginning of the window and the right end is window-size packets in the future. Received packets must be new and must fall either inside the window or to the right of the window, otherwise they are dropped. A packet is new if it has not yet been seen in the window. If a packet is received that is to the right of the window, it may be dropped if it fails an authenticity test (more on that later). If it passes the authenticity check the window is advanced, to the right, to encompass that packet. Note that packets may be received out of order and still be properly processed. Also note that a packet received too late—that is, received after a valid packet with a sequence number greater than the size of the window—will be dropped.
The replay window is in only 16 bits and is therefore illegal, but for the sake of illustration will suit us fine. The left end of the window at sequence number N, the right end is therefore at sequence number N+15. Packets N, N+7, N+9, N+16, and N+18 onward have not been received. If recently received packet N+17 is authenticated the window is advanced such that the right end is at N+17 and the left end is at N+2. This would cause packet N to be irretrievably lost since it's now to the left of the receive window. Notice, though, that packet N+7 can still be received provided that packet N+23 is not received and authenticated first.
It's important to note that the window must not be advanced until the packet that would cause its advancement has been authenticated. Doing otherwise would allow an attacker to generate bogus packets with large sequence numbers that would move the window outside the range of valid sequence numbers and cause us to drop valid packets.
