Hiding OpenVPN traffic from network opponents


#1

Introduction

While VPNs have been used by millions to route through hostile network territory (a government controlled ISP, or corporate environment), they are increasingly under threat from a border-firewall detection method known as Deep Packet Inspection, or DPI.

Perhaps no more popular example of this in action has been the so-called Great Firewall of China, which has very effectively managed to automate a detection and blocking mechanism for countless connections to VPNs outside of China, rendering the VPNs useless. On a more local level however (EU, in my case), supposedly net neutral providers are now routinely dropping VPN traffic.

For VPN users in Germany, a common complaint is the WiFi service on ICE trains that have semed to make using a normal VPN impossible. A great many hotels I have stayed in have blocked or throttled (to the point of being useless) exposed VPN traffic, alongside city-wide WiFi infrastructure, Eduroam networks at universities, and even major providers like Vodafone (in some cases) and Comcast (many cases).

Here I detail a method that I have found very reliable against DPI along the route. Not only does it ‘sheath’ VPN traffic in a layer of SSL encryption, it operates on port 443, and so appears as HTTPS traffic. It has been tested extensively worldwide in my own travels, in network contexts that I have personally known to fail before, and by a great many others using the Cyborg Unplug product (which employs the same method).

Who’s this for?

This guide is intended for those comfortable administering a UNIX or UNIX like server wishing to run a stealthy OpenVPN service scalable (with provided scripts) to hundreds of users. It will not defend against advanced end-point unmasking techniques like active probing, but it will punch through any firewall that doesn’t block SSL traffic, and doesn’t already know (somehow) that your server hosts a VPN.

Why OpenVPN?

While not the fastest and leanest (like Wireguard and some of the IPSec VPN implementations) it is the most popular and feature rich, with many free and open source clients for GNU/Linux and OS X systems.

Prerequisites

First things first, this guide assumes the following things:

  • You have full root access to a server running a Debian-based operating system, whether VPS or on-the-metal, it doesn’t matter.

  • Your server host has a DNS server and you know its IP. Alternatively, you’re happy trusting a public DNS server, and know its IP. This part is important: all your traffic might pass through the VPN but your DNS queries, each and every site you visit, are sent to your service provider’s DNS server unless you specify otherwise (DNS leaks).

  • You don’t mind typing in a few commands.

  • You know how to use an editor like nano, vim or (for some reason) emacs. This editor needs to be installed on your server.

  • Your server is running a Debian-based system (Ubuntu, Debian GNU/Linux, etc).

  • Your server has port 443 free for use. This, for instance, means it’s no use setting up a stealth VPN system on your web-server.

  • You’re willing to start again with your existing OpenVPN install, if it’s installed already.

  • You have a GNU/Linux, or OS X laptop.

Server setup

Install OpenVPN

SSH into your server on the standard port (22). For instance, if your username is you and your server is your.server:

ssh you@your.server

Now we need to install OpenVPN on the server.

sudo apt-get install openvpn easy-rsa

Create a special user for our OpenVPN process, with highly restricted privileges

sudo adduser --system --shell /usr/sbin/nologin --no-create-home ovpn
sudo groupadd vpn
sudo usermod -g vpn ovpn

Create a new config

Now we need to change into the (new) openvpn directory:

cd /etc/openvpn

Here we’ll create a new configuration file, specifically for your ‘sheathed’ tunnel.

I prefer Vim, but will use nano here as a humane baseline:

sudo nano server.conf

Copy in the following:

# OpenVPN will connect to the localhost
local 127.0.0.1 
# Define a port above 1000 for OpenVPN to connect to
port 34567 
tls-server
proto tcp 
dev tun 

# CERTS AND KEYS
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/sheath.crt
key /etc/openvpn/easy-rsa/keys/sheath.key  # This file should be kept secret
dh /etc/openvpn/easy-rsa/keys/dh4096.pem
tls-auth /etc/openvpn/easy-rsa/keys/ta.key 0 # This file is secret

txqueuelen 1000
mtu-disc yes
# Slower, more bandwidth, but I had stability problems with LZO inside a sheath 
comp-lzo no
push "comp-lzo no"

keepalive 10 120
persist-key
persist-tun
# You can allow for client-to-client interaction by uncommenting the below, but
# you'll need to then disable the related rules in the iptables/firewall script
#client-to-client
status /tmp/openvpn-status.log
verb 6
client-config-dir /etc/openvpn/ccd

server 10.11.12.0 255.255.255.0

# This ensures Internet-bound traffic to-from clients can pass through the VPN
push "redirect-gateway def1"
#TODO: EDIT DNS BELOW!!!!
push "dhcp-option DNS <DNS SERVER HERE>"
push "dhcp-option DNS <DNS SERVER HERE>"

# 1024 clients is represented as /22 in the firewall script (below). 
# To change max-clients to 128 clients would require using a subnet mask of /25
# there. See (https://www.aelius.com/njh/subnet_sheet.html)
max-clients 1024

# Important, otherwise OpenVPN will run as root, opening up a significant
# vulnerability.
user ovpn 
group vpn

Change the port and VPN IP range, as you prefer, in this config.

Generate certificates

NOTE: If you already have a working OpenVPN install you’ll need to be sure to backup your existing keys to another folder before continuing here. You’ll also need to then edit your existing configs to reflect the new location.

OpenVPN uses a familiar certificate system for client authentication, and provides helper scripts for building them.

Copy easy-rsa scripts

Copy in the scripts directory:

sudo cp -r /usr/share/easy-rsa/ /etc/openvpn
sudo mkdir /etc/openvpn/easy-rsa/keys

Customise certificate identity

Now we want to edit some variables that are resourced by the helper scripts:

sudo nano /etc/openvpn/easy-rsa/vars

The file will have some defaults as follows:

export KEY_COUNTRY="US"
export KEY_PROVINCE="TX"
export KEY_CITY="Dallas"
export KEY_ORG="My Company Name"
export KEY_EMAIL="sammy@example.com"
export KEY_OU="MYOrganizationalUnit"

Edit the above values to represent your setup.

Further down in the file you’ll see:

export KEY_NAME="EasyRSA"

Change it to read:

export KEY_NAME="sheath"

Save the file and exit.

The Diffie-Helman Parameters

Now it’s time to generate some parameters, known as the Diffie-Helman Parameters. IMO this is a great name for a crypto-folk band, but that’s another matter entirely.

4096 bits is sufficient here, at least until quantum computing breaks the world:

sudo openssl dhparam -out /etc/openvpn/easy-rsa/keys/dh4096.pem 4096

Prep for cert generation

Now we need to change to the easy-rsa directory

cd /etc/openvpn/easy-rsa

Import (source) the variables to our shell session for use by the scripts:

source vars

Clean out the directory of any existing keys. This should be unneccessary for a fresh install. Be warned that if you have an existing OpenVPN install, this will wipe out your keys! Backup first, if so.

sudo ./clean-all

Now we’ll build the main certificate authority (CA):

sudo ./build-ca

You’ll be pressing ENTER a few times here.

Generate certs/keys for the server

sudo ./build-key-server sheath

You’ll see something like the following, but just hit ENTER to accept defaults as a challenge password is awkward in practice, especially during reconnection, and doesn’t suit our setup:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Type ‘y’ when you see the following:

Sign the certificate? [y/n]
1 out of 1 certificate requests certified, commit? [y/n] 

If it all worked, you should see the following:

Write out database with 1 new entries
Data Base Updated

Generate an HMAC key, for extra security

Resourcing a ta.key ensures that a special HMAC signature is required during the authentication process. Without this signature, authentication attempts will be dropped.

sudo openvpn --genkey --secret /etc/openvpn/easy-rsa/ta.key

Certs and keys for clients

Configs can be distributed in the form of the older *.config files, with keys resourced from the given directory specified in the config itself. What is far more convenient however, is an ‘inline’ distribution of keys, whereby clients can simply load one file containing configuration variables and keys/certs therein. Such files are commonly distributed ending with .ovpn.

I have written a simple shell script to generate client *.ovpn files, with the addition that it can be used to create any number of such configs derived from a range supplied as execution arguments. The port and host assignment may seem confusing at first, but that will be explained later.

To start with, copy the below script into a new file in your keys directory:

sudo nano /etc/openvpn/easy-rsa/genkeys.sh

#!/bin/bash

OUTDIR=keys/clients
REMOTE=localhost
NAME=sheath
PORT=1194
source ./vars
START=$1; END=$2

OPTIONS="
client
dev tun
proto tcp
remote $REMOTE $PORT
resolv-retry infinite
script-security 2
nobind
persist-key
persist-tun
user nobody
group nogroup
tls-client
tls-auth ta.key 1
comp-lzo
verb 3
"

ovpn(){
        export KEY_CN=$NAME$1
        OVPN=$NAME$1.ovpn
        FN=$OUTDIR/$OVPN
        ./pkitool $NAME$1
        echo "$OPTIONS" > $FN
        echo "<ca>" >> $FN
        cat keys/ca.crt >> $FN
        echo "</ca>" >> $FN
        echo "<cert>" >> $FN
        cat keys/$NAME$1.crt >> $FN
        echo "</cert>" >> $FN
        echo "<key>" >> $FN
        cat keys/$NAME$1.key >> $FN
        echo "</key>" >> $FN
        echo "<tls-auth>" >> $FN
        cat keys/ta.key >> $FN
        echo "</tls-auth>" >> $FN
}

if [ ! -d $OUTDIR ]; then
        mkdir $OUTDIR
fi

for i in $(seq $START $END)
        do
                ovpn $i
                echo "$FN generated"
        done

Ensure you are in that directory:

cd /etc/openvpn/easy-rsa/keys

Now we need to make the script executable:

sudo chmod +x genkeys.sh

Create a directory to store our client configs

 sudo mkdir /etc/openvpn/easy-rsa/keys/clients

Fix an issue with local OpenVPN provided openssl.cnf file in Debian/Ubuntu installs:

sudo perl -p -i -e 's|^(subjectAltName=)|#$1|;' /etc/openvpn/easy-rsa/openssl-1.0.0.cnf

Generate some .ovpn configs

To start with, let’s generate 10 new configs, to gift to friends/colleagues etc:

sudo ./genkeys.sh 1 10

You should now be able to do the following and see your new *.ovpn files:

sudo ls keys/clients

Each config is unique, and numbered sequentially for ease of management (like revocation) later. So it follows that if you wish to generate another 5 keys at a later date, you would run the script like so:

sudo ./genkeys.sh 11 16

A single config at number 17 would be like so:

sudo ./genkeys.sh 17 17

Install stunnel on the server

Stunnel is our ‘sheath’ of encryption, in which we hide our OpenVPN traffic. Install it like so:

sudo apt-get update
sudo apt-get install stunnel4

Now create an stunnel config:

sudo nano /etc/stunnel/stunnel.conf

And put the following lines into this file:

cert = /etc/stunnel/stunnel.pem
pid = /var/run/stunnel.pid
; It's recommended to switch process ownership if started as root. 
; But make sure you have these:
;setuid = nobody 
;setgid = nogroup
output = /var/log/stunnel
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

[openvpn]
client = no
accept=443
connect=34567
;Use the below with a CA to harden against MiTM attacks
;verify=3 

Create the stunnel certificate

This will create a cert valid for 730 days, or 2 years. Change the -days argument below if you want more or less:

cd /etc/stunnel/
sudo openssl genrsa -out key.pem 2048
sudo openssl req -new -x509 -key key.pem -out cert.pem -days 730 
sudo cat key.pem cert.pem >> /etc/stunnel/stunnel.pem

A key size of 2048 is what I’ve found to be the best compromise for performance and security. 4096 would simply tap out when bandwidth wasn’t available. Again, the job of our SSL tunnel is just to mask the already-encrypted VPN traffic in the layer under it.

NOTE: If you have a Certificate Authority file, use it! This will harden the setup against MiTM attacks. You can either use the certificate separately with a CAfile=/path/to/CA.crt in your config, or just concatenate it into the stunnel.pem. Also note that if it is a separate file, the client will also need it.

You are now done setting up stunnel on the server.

Securely copy keys and configs to your laptop to distribute

Securely copy client configs to your laptop

NOTE: it’s not sane to have VPN configs on your laptop without disk-encryption and a strong password for your everyday user.

Using a UNIX derived/based system (GNU/Linux or OS X), we can copy the configs back over an encrypted tunnel. First however, we need to copy them to a place on the server we can access from our laptop remotely (the user you use for ssh shouldn’t/won’t have permissions to read the openvpn directory.

On the server we’ll copy our clients directory

sudo tar cvzf ~/sheath-clients.tar.gz /etc/openvpn/easy-rsa/keys/clients /etc/stunnel/stunnel.pem

Change the permissions so that our user can access it over the tunnel:

sudo chown $USER:$USER ~/stealth-clients.tar.gz

Now copy the configs to your laptop using the reverse scp method. On your laptop type:

scp you@your.server:sheath-clients.tar.gz ~/

As an extra precaution (from rogue programs running as a normal user reading this file on your system) you might want to ensure that root owns the file and that none else can read it

sudo chown root:root sheath-clients.tar.gz
sudo chmod go-r sheath-clients.tar.gz

Once copied, it’s advisable to remove the archive on your server:

sudo rm -f sheath-clients.tar.gz

Accessing the configs on your laptop

Unpack the archive, creating the ‘clients’ directory in turn, as follows:

sudo tar xvzf sheath-clients.tar.gz

Note that the clients directory created will be owned by the root user and you won’t be able to read it as a normal user.

Distributing the configs and stunnel.pem

Every client will need the stunnel.pem in their stunnel config directory (usually /etc/stunnel/) and a unique sheath*.ovpn on their laptops.

How you give these to other people is up to you. As they’re owned by the root user, you’ll need to work as root to read them. If you want to mail them to people, be sure to use PGP. You could use an install of Signal on your laptop to send one to your friend’s phone as an attachment, or have them connect to the same WiFi network as you and scp it to their machine. If you use a USB stick, make sure it’s clean before you put it into your laptop.

Setup a firewall on the server using iptables

Wherever you prefer on the server (for instance a ‘scripts’ directory), create a
new file:

nano sheath-tables.sh

In it, place the following lines.

# TODO change /22 (1024 clients) to /25 (128 clients)

#!/bin/sh

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

# IP FORWARDING
iptables -A FORWARD -i ifext -o tun0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -s 10.11.12.0/22 -o ifext -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.11.12.0/22 -o ifext -j MASQUERADE

# GLOBAL DoS ANTI 
iptables -A INPUT -p icmp -m limit --limit 2/second --limit-burst 2 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -m limit --limit 80/minute --limit-burst 100 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m limit --limit 80/minute --limit-burst 100 -j ACCEPT

iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 22 -j DROP
iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 80 -j DROP
# If you want SMTP over the VPN, uncomment the below
#iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 25 -j ACCEPT
iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 53 -j ACCEPT

iptables -A INPUT -p tcp --destination-port 53 -j DROP 
iptables -A INPUT -p tcp --destination-port 25 -j DROP 

We’ll need to be able to execute this as a script later, so we need to make it executable:

chmod +x sheath-tables.sh

Enable Packet Forwarding

This will ensure that traffic is allowed to be forwarded to-fro the VPN:

echo 1 > /proc/sys/net/ipv4/ip_forward

To make this setting persist across reboots, we need to edit a file:

nano /etc/sysctl.conf

Now find the line that reads:

#net.ipv4.ip_forward=1

And uncomment it, such that it looks like this:

net.ipv4.ip_forward=1

You’ve finished setting up the server.

Client setup

Stunnel needs to be installed on the client. It creates the SSL tunnel between the client and the server through which we route our OpenVPN traffic, effectively hiding it from deep packet inspection, looking like standard SSL traffic on port 443 (HTTPS)

Install stunnel

Debian based system

sudo apt-get install stunnel4

OS X (with homebrew)

brew install stunnel

OS X (with MacPorts)

port install stunnel

Copy the stunnel.pem file to the correct directory

However you do it, on each and every client stunnel.pem needs to go into /etc/stunnel/. Also make absolutely sure it is not readable by any other than the owner of the process that runs stunnel. On my GNU/Linux systems, that process is owned by the user stunnel4, created on install.

Create an stunnel config

sudo nano /etc/stunnel/stunnel.conf

Into this file copy in the following, being sure to edit the line to include your server IP.

cert = /etc/stunnel/stunnel.pem
client = yes
; Wise to do this if you start stunnel as root. Be sure the user and group actually exist!
;setuid = nobody 
;setgid = nogroup
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

[openvpn]
client = yes
accept = 127.0.0.1:1194
connect = <IP OF YOUR SERVER GOES HERE>:443
; Only use the below verification option if your pem includes a CA certificate
;verify=3

Starting it up (testrun)

The best approach here, when testing, is to have two terminals open on your laptop, with one logged into your server (over ssh).

Start the firewall on the server

Assuming your sheath-tables.sh file is in your home directory on the server, you would:

sudo ./sheath-tables.sh

Check that the rules are all in place by typing:

sudo iptables -L

Start stunnel on the server

Stunnel needs to be started on the server before anything else. This will open port 443 on the server and wait for a connection from OpenVPN on the server (you read it right), connecting to localhost, and a connection from the remote client (your laptop)

sudo stunnel4 /etc/stunnel/stunnel.conf

Test it is running with

ps ax | grep stunnel

Start stunnel on the client (your laptop for now)

It’s usually no different. In most cases the same should be fine:

sudo stunnel4 /etc/stunnel/stunnel.conf

Or (on some systems):

sudo stunnel /etc/stunnel/stunnel.conf

Test it is running with

ps ax | grep stunnel

If that doesn’t work, be sure you can actually reach port 443 at all. From your laptop:

telnet <YOUR SERVER IP> 443

Start OpenVPN on the server

sudo openvpn --config /etc/openvpn/server.conf

You should see in the output that OpenVPN comes up and is waiting for connections

Start OpenVPN on the client

If you’re an OS X user, and you use OpenVPN with something like TunnelBlick, you should just be able to load in an OpenVPN config. As a test, try loading sheath1.ovpn.

If you use OpenVPN in the commandline (recommended for debugging - and in general tbh), run it like so:

sudo openvpn --config /path/to/sheath1.ovpn

You should see it come up and connect to the server. If using the command line, you should see something a little like this in the last lines of the output:

Fri Jul  6 15:51:27 2018 /sbin/ip route add 10.11.12.1/22 via 10.11.12.10
Fri Jul  6 15:51:27 2018 GID set to nogroup
Fri Jul  6 15:51:27 2018 UID set to nobody
Fri Jul  6 15:51:27 2018 Initialization Sequence Completed

Test ping the server from the client

ping 10.11.12.1

Setup routing on the client

We need to ensure all our packets flow to our VPN server via the LAN gateway. If you’re on Linux, run the following command to determine your gateway IP:

/sbin/route -n

If you’re on OS X:

netstat -r

Now add a network to our routing table.

On Linux:

sudo route add -net <SERVER IP> netmask 255.255.255.255 gw <GATEWAY IP>

On OS X:

sudo route -n add -net <SERVER IP> netmask 255.255.255.255 gw <GATEWAY IP>

Set DNS server

Don’t leak DNS to your ISP! Set your DNS to a server you trust. A good bet is a DNS server run by your server host.

On Linux

sudo echo "nameserver <DNS SERVER>" > /etc/resolv.conf

On OS X

Just use the GUI or whatever method is the fashion of the day

Quick brower test

In your browser, visit https://wtfismyip.com. You should your IP is see the IP of your new VPN server. A traceroute should also demonstrate you are routing out through the sheathed VPN.

If so, congrats!

Test for DNS leaks

Go to https://dnsleaktest.com. Run the standard test and see if your DNS is is reporting to the correct server, and not your ISP.

TODO (coming soon)

Scripts for starting up stunnel, openvpn and all routing for Linux and OS X clients


#2