How IKEv1 IPSec VPNs Work?

If you are not familiar with the basics of cryptography, please read A Note on Cryptography Fundamentals & Algorithms before this. All of them will be used here to explain this.

IPSec VPN is just a logical tunnel between 2 VPN peers across a public network like internet. It does not create a separate network with a different IP range between peers. This post is about the IKE version 1 IPSec tunnels..

There are 2 types of IPSec VPNs according to their behavior. 

1. Site-to-Site VPNs

2. Remote Access VPNs

Basically Site-to-Site VPNs are formed between 2 gateways across the internet while Remote Access VPNs are created between a gateway and a client software.

Benefits:

1. Confidentiality through Encryption

2. Integrity through Hashing

3. Authentication through PSK (symmetrical keys) or RSA/DSA (asymmetrical keys)

4. Anti-replay packets by Counting Packets

Two Phases of VPN

Actually there are 2 tunnels in the process of creating a IPSec tunnel. We call them IKE (Internet Key Exchange) Phase 1 & Phase 2.

IKE Phase 1 is used to exchange control information between 2 VPN peers.

IKE Phase 2 is used to transport the real traffic.

Actually the IKE Phase 2 is the real IPSec tunnel which use IPSec parameters. IKE Phase 1 is formed using ISAKMP (Internet Security Association & Key Management Policy) parameters..

So IKE Phase 1 must be formed 1st in order to start IKE Phase 2 to send user traffic..

IKE Phase 1

STEP 01. Negotiate Phase 1

When PC1 wants to send a data packet to PC2, it hits R1 1st and R1 sends ISAKMP negotiation parameters to R2 to form IKE Phase 1. R2 sends his ISAKMP negotiation parameters to R1.

Following are the ISAKMP parameters which will be negotiated from both peers (R1 & R2)

Hashing Algorithm which will be used.. (MD5/SHA)

Authentication Type which will be used.. (PSK or RSA)

Group of Deffie Hellman Keys which will be used to generate shared key for encryption algorithms like DES/3DES/AES later.. (Group 1/2/ 5)

Life Time which the IKE Phase 1 tunnel must active.. (default is 1 day)

Encryption Algorithm which will be used.. (DES, 3DES, AES)

From all of the above 5 parameters only the life time can be different from both ends. They will agree to the minimum life time value from both ends.

STEP 02. Setup DH Keys

After the negotiation is completed, they will run Deffie Hellman Algorithm to generate shared secret key material. Following image will explain how the shared secret key is generated by DH Algorithm.

R1 and R2 creates public & private key pairs from their ends..
Both exchange their public keys with each other..
R1 creates his shared secret DH Key from his private key and R2's public key..
R2 creates his shared secret DH Key from his private key and R1's public key..

According to the DH Algorithm, final out put from both sides (DH Keys) are same. Following capture explains how the keys of both sides becomes equal mathematically.

Now using the shared secret key as the symmetrical key of the agreed encryption algorithm in step 1 (DES, 3DES, AES), both R1 and R2 can exchange encrypted data between peers.

STEP 03. Authenticate

Now R1 and R2 can authenticate by exchanging identities and certificates securely using the authentication method negotiated in step 1 and finally form IKE Phase 1 tunnel.

Now it's the time to form IKE Phase 2 tunnel which is the actual IPSec tunnel which will be used to transport user traffic from PC1 to PC2 using the security of IKE Phase 1 tunnel.

IKE Phase 2

STEP 01. Negotiate Phase 2

Like the ISAKMP parameters in IKE Phase 1, R1 & R2 will negotiate following IPSec parameters which will be used in IKE Phase 2 tunnel. Authentication is not necessary because IPSec peers are already authenticated in IKE Phase 1.

Hashing Algorithm which will be used.. (MD5/SHA)

Group of Deffie Hellman Keys which will be used to generate shared key for encryption algorithms like DES/3DES/AES later.. (Group 1/2/ 5). Here R1 & R2 can use the DH Key which is used in IKE Phase 1 or create a new DH Key just for IKE Phase 2. Creating a new DH Key for IKE Phase 2 is called Perfect Forward Secrecy (PFS)

Life Time which the IKE Phase 1 tunnel must active.. (can be a clock tick or amount of data)

Encryption Algorithm which will be used.. (DES, 3DES, AES)

STEP 02. DH Keys if PFS is configured

If PFS is configured R1 & R2 will run DH Algorithm to create a new shared secret key for the encryption algorithm which will be used in IKE Phase 2. Otherwise this step is omitted.

So after the negotiation or after the DH key creation (if PFS is configured) they form the fully working IPSec VPN from both ends. Now PC1's traffic can be securely sent to the PC2.

Note:-

According to the number of packets exchanged in the process of forming IKE Phase 1 & IKE Phase 2 tunnels, 3 modes are defined. But the overall process is same as described above.

Modes of IKE Phase 1:
1. Main Mode which use 6 packets to form IKE Phase 1 tunnel between R1 & R2
2. Aggressive Mode which use 3 packets to form IKE Phase 1 tunnel between R1 & R2

Modes of IKE Phase 2:
1. Quick Mode which use 3 packets to form IKE Phase 2 tunnel between R1 & R2