June 6, 2022
Protecting data online - how does a VPN actually work?
A VPN is a virtual private network that creates an encrypted connection between one point on a network to another, but how does it work?

This blog has been written by digital forensics expert, Mike Wallis, as part of our series in the run up to Infosecurity Europe where we’ll be uncovering how commonly misunderstood cyber security technical practices really work. At the conference, you can find our team running practical demos on our stand looking at how several malicious processes really work in practice.

VPN Connections

A VPN connection is identified under two different methods known as transport or tunnel modes. Under the transport method a client connects to a site, an example would be an employee at their home address connecting to an office, and a tunnel refers to site to site, where an office connects to another office. The method describes the type of connection. A third type of connection also exists but is far less common which is client-to-client, where two devices can connect to each other. The connection itself is not limited to just the internet; in fact, a VPN can be leveraged over any private network.

Why use a VPN?

Many of us will use a VPN because it satisfies our goal of confidentiality, where the VPN is used to encrypt the data whilst in transit. A request from your device in the form of plain text is encrypted by the VPN hardware or software. This encrypted message is sent to the end point at which moment in time it is decrypted. Application of this over the internet on a public Wi-Fi therefore prevents an anonymous person reading the data. Apply this to a private network and you have decreased the ability for anyone to either plug in to that network or gain knowledge of data content passing through it.

How does a VPN work?

Implementing IPSec

Internet Protocol Security (IPSec) is a standard for establishing VPN connections and offers two modes. The Authentication Header (AH) mode of IPSec can provide data integrity and authentication but not confidentiality. For this the Encapsulating Security Payload (ESP) mode is employed and it can do everything AH mode does with the added feature of encryption. Built into IPSec is also some advanced anti-replay prevention. This is important as replay attacks occur when an attacker copies a message and retransmits it. In the instance of a VPN using ESP where the message is encrypted, whilst the content may not be known, the result could be a replication of a bank withdrawal for example. ESP encrypts all the information in a packet above the network level, including the protocol header and message data. There are variations when it comes to how AH and ESP perform integrity and authentication as to why you may implement one over the other. To simplify what IPSec using AH or ESP looks like we will start with a simple TCP/IP packet. The IP Header contains details around source IP, destination, the protocol in this case TCP, type of service and more. The TCP header used to understand the conversation between source and destination and finally the packet contains the data that we transmit.

Authentication Header

The Authentication Header works by adding a hash known as an Integrity Check Value (ICV) to the packet. The hash is comprised of the source address, destination address, length, and the data itself. When the destination receives the packet, it goes through the same motions and thus can verify the integrity of the packet. However, how do both sides know how to compute the hash? This is where authentication comes in to play with the source and destination agreeing on a key to compute this hash prior to the start of communications. Also included in the AH Header as we have mentioned is anti-replay prevention. This is achieved by adding a 32-bit sequence number that increments with each packet.  

Authentication Header - Transport Mode

AH in transport mode uses the IP Header for the ICV that is then added to the AH Header.  

Authentication Header – Tunnel Mode

The addition in tunnel mode is the new IP Header at the start of the packet. In short, the original packet is packaged inside a tunnel packet with this header containing the details of where the tunnel packet is being sent to. This new tunnel IP Header is what is used when computing the ICV rather than the original IP Header.

Now, looking at this when using IPSec ESP mode we can fully start to understand where encryption actually occurs.

Encapsulating Security Payload

We have briefly mentioned what information ESP encrypts and that it differs from AH in implementation. ESP focuses on the message contents itself.

Encapsulating Security Payload – Transport Mode
Encapsulating Security Payload – Tunnel Mode

The tunnel packet is packaged similarly to the AH tunnel packet by adding a new IP header to the start. In both examples, authentication and integrity in the form of the ICV is created using the message contents. The value is then placed in the ESP Auth part of the ESP mode. The next header is placed in the ESP trailer and the ESP Header has the sequence number in.  

In the above we have shown how using IPSec modes can meet the goals of confidentiality, integrity, and authentication. However, IPSec isn’t the only option.  


SSL stands for Secure Socket Layer, but in the case of a VPN is more of a dated terminology. An SSL VPN in fact provides a cryptographic equivalent to what IPSec can produce by using the latest methods available such as more up to date protocols like Transport Layer Security (TLS). The downfall of an SSL VPN is that it typically relies on a web browser in some fashion. This is where vulnerabilities are more likely to be found rather than the cryptographic methods. There are more touchpoints with a web browser requesting information from a server - the application, server, authentication mechanisms and the VPN portal itself are all potential risks.

The usage and how an SSL VPN works in the background isn’t too dissimilar to that of the IPSec modes above. In an SSL tunnel VPN, your IP traffic is protected by the tunnel it creates from the web browser to a VPN gateway. The VPN gateway would then allow the user to view services on a proprietary network , or specific corporate only use software, that is hidden from the internet. This differs to connecting to a VPN portal as the portal itself is web-based and resources will be controlled by the organisation.


Understanding that both can offer a similar service is fantastic, but how do you choose which one suits your organisation or needs?  

IPSec operates by using both hardware and software, requiring set up, maintenance and alongside that comes cost and scalability. The vulnerabilities are more limited than with an SSL VPN and the layered security approach is the main benefit of IPSec.

SSL VPN on the other hand has the benefit of being almost good to go on any device. Smartphones, tablets, laptops, and desktops all come with some form of browser pre-installed. Tunnelling allows access to specific applications rather than network wide access which can be ideal when dealing with a third-party contractor for example. The downside of course, as mentioned, is the inherent risks associated with the use of a web browser.

Both solutions offer protection from Man-in-the-Middle (MITM) attacks. IPSec is more difficult to operate, maintain and scale but ultimately provides better security than an SSL VPN. The choice should be dictated by your organisations’ threat model and risk appetite.

How can a VPN help my business?  


A VPN should be looked upon as another tool in the cyber security arsenal rather than the solution to all your problems. Even with in-depth research and auditing of VPN providers, flaws or leaks may still occur. Due diligence can help you to choose a more robust VPN provider, but to ensure you are protected, view a VPN as a layer of defence. It should be applied alongside other mechanisms. In the SSL VPN example this the extends to your browser. Also, incorrect assumptions around VPNs pose a cyber security risk, such as blindly trusting the VPN connection or not correctly managing VPN traffic. Remember the traffic is encrypted, this prevents protection systems from analysing the packets, meaning decryption is required beforehand at a VPN gateway. A firewall can be used as another layer of defence to restrict the VPN traffic and to protect your network.

Our team are experienced at identifying vulnerabilities and partnering with clients to strengthen their defensive cyber security process. If you have any questions about the blog, or would like to discuss your security set up, please contact Mike Wallis.  

Click here to read the first blog in series, The importance of safely wiping data from broken devices (and how to do it!).

Sign up to our newsletter to receive the latest updates