IPSEC is a name that's been given to a suite of security protocols used to secure IP traffic between computers. I'll talk about these individual protocols shortly, but for now let's just talk about the main ideas behind this initiative as a whole. Recall from my discussion of CIA in WhatIsCIA that to secure a channel we want to start with an authenticated key exchange, during which two things happen: The communicating peers develop trust in each other's identity, and they discover a session key that the bad guys won’t know. Based on this session key, the peers then provide integrity protection over the channel via MAC protocols (usually HMAC), and the channel is encrypted as well. These countermeasures mitigate lots of different attacks, including spoofing, tampering with packets, connection hijacking, eavesdropping, and replay.
Most IPSEC implementations also provide some form of authorization. For example, in Windows you can do meaningful filtering of incoming IP packets based on the source IP address. This is normally easy for an attacker to skirt around because IP source addresses can be spoofed when not protected. Under IPSEC, however, the source IP address is authenticated, giving these sorts of filters some real teeth. In fact, IPSEC policies are chosen based on the peer's IP address, so it's possible to have a client that communicates with a secure server over IPSEC but also communicates over an unsecured channel with other machines on the network. IPSEC is not an all-or-nothing proposition.
With IPSEC, the peers are not user principals; rather, they're machines. So when two machines use IPSEC to communicate, they authenticate to one another, ultimately proving that they know some secret that validates their IP address. In other words, with Alice logged into machine MAC1 and communicating with Bob, logged into MAC2, if IPSEC is the only technique securing their communication, Alice and Bob aren't authenticating each other; MAC1 and MAC2 are. If MAC2 requires all incoming traffic to be secured with IPSEC, the only thing Bob knows about any given message from Alice is that the IP address on that message wasn't spoofed, that the data wasn't tampered with, and that the plaintext wasn't seen by an eavesdropper on the wire (assuming that both encryption and integrity protections are in place). Bob has no proof that it was Alice who actually sent the message. He just knows that it originated from MAC1 (see Figure 68.1). IPSEC is thus very useful for protecting communications between machines, but it doesn't help a server implement any form of user authentication or authorization.
Figure 68.1 Authenticating machines
When is IPSEC useful? When you don't have any better alternative. For example, two COM servers talking to one another over DCOM can use COM security to communicate, which under the covers uses SSPI (WhatIsSSPI), typically using Kerberos authentication. The endpoints of the authentication in this case are the user principals running the processes in question, as shown in Figure 68.2. This is better because the server will get a token (WhatIsAToken) for the client and will be able to make authorization decisions based on the client's group memberships as well as to impersonate the client if the need arises (WhatIsImpersonation).
Figure 68.2 Authenticating user principals
But what if you're not using COM? What if you're using something less mature, like version 1.1 of .NET Remoting, which doesn't provide a secure channel? This is where IPSEC can come in handy. At least you'll have a secure channel between the machines that are communicating, even if you won't necessarily be able to do user authentication and authorization. While there are ways of adding user authentication to .NET Remoting, such as hosting in IIS, all of the techniques I know of are somewhat kludgy (HowToAddCIAToDotNETRemoting) . Until version 2.0 of the .NET Framework ships and this support is built in to remoting from the ground up, IPSEC may be the best alternative in your toolbox.
What protocols make up IPSEC? Too many, if you ask Niels Ferguson or Bruce Schneier (Ferguson and Schneier 1999). There are two protocols for channel security: Authentication Header (AH) and Encapsulating Security Payload (ESP). They can be run individually or on top of one another. In addition, they both support two distinct modes of operation: transport and tunnel. AH provides only authentication and integrity protection; it doesn't encrypt the channel. ESP provides full CIA on the channel, but costs a bit more in terms of bandwidth. Ferguson and Schneier (1999) recommend simplifying the protocol suite by eliminating AH and transport mode completely, which would leave ESP in tunnel mode as the only option. I'm going to compromise a bit in HowToUseIPSECToProtectYourNetwork and suggest that you use ESP in transport mode for securing communications within your organization, as it's the most manageable for LAN communication and still provides full CIA security on the channel.
Key exchange is described by a couple of layered standards: Internet Security Association Key Management Protocol (ISAKMP), and Internet Key Exchange (IKE). These specs talk about how to perform authenticated key exhange to get the channel ready for operation. One last term you should be aware of is the Security Association (SA), which is a little bit like a TCP association, only unidirectional. An SA is identified by a triplet that consists of a peer's IP address, a protocol identifier (AH or ESP), and an index to a set of parameters (such as what encryption and hash algorithms should be used to protect packets). One of the things that the key exchange protocols do is help establish an SA between two machines. In other words, there's lots of negotiation involved here: what protocol to use, which algorithms to use, and so on. Unfortunately none of these IETF standards are easy to read, so if you want to learn more about IPSEC, you might want to pick up a book to guide you, such as IPSEC: Securing VPNs (Davis 2001).
On Windows, key exchange can be done one of three ways: Kerberos, X.509 certificates, and a hardcoded master key. Of these three, Kerberos is by far the simplest to use (assuming you’re using Active Directory), very secure, and is thus the clear preference. But it will work only if the two machines communicating have a path of trust between them (e.g., they're members of the same forest or their forests are linked with trust relationships). If no trust path exists, you can't use Kerberos; you'll need to use one of the other two options instead, with certificates being the preferable mechanism.
In HowToUseIPSECToProtectYourNetwork, I'll show how to implement IPSEC using a very simple, secure configuration for locking down communication between machines in your organization, where a lot of negotiation isn't really necessary. I'm not going to attempt to tell you how to implement a VPN-secured extranet, where you'll need to deal with multiple platforms, firewalls, NAT, and so on, because that's not only beyond the scope of this book but also beyond the scope of my own expertise.