Skip to main content

IPsec (CSEC)

IPsec (IP security) is a protocol suite for the network layer in the 5-layer internet model. It provides confidentiality, integrity and authenticity for IPv4/IPv6 traffic at the IP layer.

IPsec can be understood as a framework rather than a single, self-contained protocol. Therefore, the specification spans various RFCs. An architectural overview is given in [📖RFC4301].

The original specification for IPv4 did not include native protection mechanisms. However, due to growing concerns about network security in the 1990s, the IETF intended to make security mechanisms mandatory for the upcoming IPv61. Graziani points out that this mandate was later relaxed in [📖RFC6434] ([📖Gra17, p. 78]): Support for IPsec is recommended ("should implement") for IPv6 devices2.

There are two protocols commonly used with IPsec when end-to-end communications needs to be secured: The Authentication Header Protocol and the Encapsulating Security Payload Protocol, respectively. Both differ in the security objectives they achieve. Furthermore,

"both AH and ESP offer integrity and authentication services, but the coverage differs for each protocol and differs for transport vs. tunnel mode." ([📖RFC4301, Section 4.2 ]).

The following provides a brief overview.

(Note that for IPsec to be used various parameter shave to be negotiated. This is realized with the help of Security Associations. The concept is described in [📖RFC4301, Section 4.1 ] and [📖RFC4306] respectively.)

Authentication Header Protocol (AH)​

Authentication Header is specified in [📖RFC4302]. It is designed to provide origin authentication and integrity for IPv4- and IPv6-datagrams.

Transport Mode​

In transport mode, an IPsec datagram gets created as follows: An HMAC hh is constructed over the payload of the IP datagram, i.e. the encapsulated TCP packet ([📖TFW21, p. 823]), and over immutable fields of the IP header. The result is prepended as the AH to the original payload of the datagram. It contains the cryptographic data used to verify the authenticity and integrity of the datagram by the destination host. Thus, it the payload of the "new" datagram (see Figure 1) contains the original TCP packet and the AH.

The original IP header from the authenticated datagram is reused for sending the datagram.

The destination host recalculates the HMAC h′h' over the payload of the datagram and specific fields of the IP header and compares it to the HMAC hh embedded in the Authentication Header. If both values match, the datagram was verified.

The symmetric key used for the HMAC was previously established via the negotiated Security Association and the implemented Internet Key Exchange (IKEv2) Protocol ([📖RFC4306]).

Figure 1 Illustration of the IPsec Transport-Mode. Note that when using AH in transport mode, not all fields of the IP header are authenticated, since certain fields - such as the TTL or destination address - must be read or modified by devices not IPsec-aware.

Tunnel Mode​

In tunnel mode, the entire original IP datagram is encapsulated and becomes the payload of a new datagram. An HMAC is computed over this payload (see Figure 2).
The outer IP header is constructed independently, often by copying selected fields from the original header. However, since the HMAC does not cover fields from the outer header, intermediate network devices like router and switches may modify those fields if required (e.g. to support Network Address Translation).

As a result, the verification process performed by the destination host excludes fields from the outer header, in contrast to transport mode.

Establishing Authenticity and Integrity with AH​

  • Integrity is ensured by verifying whether h=?h′h \overset{?}{=} h'.
  • Authenticity is guaranteed by the fact that only sender and receiver possess the secret shared by using the Security Association and negotiated over the IKE Protocol.

If verification of h=?h′h \overset{?}{=} h' fails, then the datagram cannot be trusted.

Encapsulating Security Payload Protocol (ESP)​

Encapsulating Security Payload Protocol (_ESP) is specified in [📖RFC4303]. In addition to AH, it provides confidentiality. Fall and Stevens point out that, if ESP is used, but only authenticity and integrity if required, ESP can employ a NULL-encryption method[^null-encryption] ([📖SF12, p. 858]).

[^null-encryption] [📖RFC2410]

Transport Mode​

In transport mode, ESP encrypts the TCP packet. In contrast to AH, the HMAC for integrity check is appended to the encrypted payload, after the ESP trailer, which contains Pad, Pad Length and Next Header fields (see Figure 1).

"The ESP ICV is a variable-length trailer used if integrity support is enabled and required by the integrity-checking algorithm. It is computed over the ESP header, payload, and ESP trailer." [📖SF12, p. 859]

Tunnel Mode​

In tunnel mode, ESP encrypts the whole original IP datagram. Just like with AH and tunnel mode, a new IP header is then constructed, and intermediate devices may modify the header fields as required (see Figure 2).

Figure 2 IPsec Tunnel-Mode.

Combination of AH and ESP​

Tanenbaum et al. point out that AH's integrity check uses some of the IP header's fields, which makes it "impossible for an intruder to falsify the origin of a packet". ([📖TFW21, p. 823])3. For this reason, AH and ESP are sometimes combined to add an additional layer of protection:

  • AH first, then ESP: The AH data is encrypted. The origin is only visible to the receiving peer after decryption.
  • ESP first, then AH: AH covers the encrypted payload, providing a stronger form for origin authenticity.

Comparing IPsec to SSL/TLS​

While SSL/TLS operates as an application-layer protocol, IPsec secures end-to-end communication at the network-layer.

Support for SLS/TLS must be provided by the applications that wish to provide secure communications, e.g. web browsers or mail clients.

In contrast, IPsec is implemented at a lower abstraction layer in the IP stack and requires system wide support, affecting devices and operating systems. For example, when tunnel mode is used, receiving endpoints must be able to properly demultiplex incoming packets and extract the payload accordingly.

Due to the (architectural) complexity of IPsec, it is most commonly deployed in Virtual Private Networks (VPNs).

Footnotes​

  1. Tanenbaum outlines briefly the historical discussion that led to IPsec being addressed at the network layer instead of relying on application layer protocols in [📖TFW21, p. 820] ↩

  2. "Previously, IPv6 mandated implementation of IPsec and recommended the key management approach of IKE. This document updates that recommendation by making support of the IPsec Architecture [RFC4301] a SHOULD for all IPv6 nodes." ([📖RFC6434, Section 11 ]). Conversely, Fall and Stevens state in [📖SF12, p. 841] that IPsec implementation must be present fpr IPv6 implementations, which is possibly due to the close date of the pblication date of the 2nd edition of this book (2012) and the RFC's publication date (December 2011) ↩

  3. Additionally, the authors mention the historical motivation for the existence of the two different protocols: AH was responsible for integrity, ESP for encryption, while integrity was later on added to ESP (ibid., p. 824824). ↩

References

  1. [RFC4301]: Seo, Karen and Kent, Stephen: Security Architecture for the Internet Protocol (2005), RFC Editor, 10.17487/RFC4301 [BibTeX]
  2. [RFC6434]: Jankiewicz, Edward and Narten, Dr. Thomas and Loughney, John A.: IPv6 Node Requirements (2011), RFC Editor, 10.17487/RFC6434 [BibTeX]
  3. [Gra17]: Graziani, R. and Safari, an O'Reilly Media Company: IPv6 Fundamentals: A Straightforward Approach to Understanding Ipv6, 2nd Edition (2017), Cisco Press [BibTeX]
  4. [SF12]: Fall, K.R. and Stevens, W.R.: TCP/IP Illustrated (2012), Addison-Wesley [BibTeX]
  5. [TFW21]: Tanenbaum, Andrew and Wetherall, David and Feamster, Nick: Computer Networks Global Edition (2021), Pearson Deutschland [BibTeX]
  6. [RFC4306]: Kaufman, Charlie: Internet Key Exchange (Ikev2) Protocol (2005), RFC Editor, 10.17487/RFC4306 [BibTeX]
  7. [RFC4302]: Kent, Stephen: IP Authentication Header (2005), RFC Editor, 10.17487/RFC4302 [BibTeX]
  8. [RFC4303]: Kent, Stephen: IP Encapsulating Security Payload (ESP) (2005), RFC Editor, 10.17487/RFC4303 [BibTeX]
  9. [RFC2410]: Glenn, K. Robert and Kent, Stephen: The NULL Encryption Algorithm and Its Use with Ipsec (1998), RFC Editor, 10.17487/RFC2410 [BibTeX]