Introduction
While VPNs have been used by millions to route through hostile network territory (a government controlled ISP, or corporate environment), they are increasingly under threat from a border-firewall detection method known as Deep Packet Inspection, or DPI.
Perhaps no more popular example of this in action has been the so-called Great Firewall of China, which has very effectively managed to automate a detection and blocking mechanism for countless connections to VPNs outside of China, rendering the VPNs useless. On a more local level however (EU, in my case), supposedly net neutral providers are now routinely dropping VPN traffic.
For VPN users in Germany, a common complaint is the WiFi service on ICE trains that have semed to make using a normal VPN impossible. A great many hotels I have stayed in have blocked or throttled (to the point of being useless) exposed VPN traffic, alongside city-wide WiFi infrastructure, Eduroam networks at universities, and even major providers like Vodafone (in some cases) and Comcast (many cases).
Here I detail a method that I have found very reliable against DPI along the route. Not only does it ‘sheath’ VPN traffic in a layer of SSL encryption, it operates on port 443, and so appears as HTTPS traffic. It has been tested extensively worldwide in my own travels, in network contexts that I have personally known to fail before, and by a great many others using the Cyborg Unplug product (which employs the same method).
Who’s this for?
This guide is intended for those comfortable administering a UNIX or UNIX like server wishing to run a stealthy OpenVPN service scalable (with provided scripts) to hundreds of users. It will not defend against advanced end-point unmasking techniques like active probing, but it will punch through any firewall that doesn’t block SSL traffic, and doesn’t already know (somehow) that your server hosts a VPN.
Why OpenVPN?
While not the fastest and leanest (like Wireguard and some of the IPSec VPN implementations) it is the most popular and feature rich, with many free and open source clients for GNU/Linux and OS X systems.
Prerequisites
First things first, this guide assumes the following things:
-
You have full root access to a server running a Debian-based operating system, whether VPS or on-the-metal, it doesn’t matter.
-
Your server host has a DNS server and you know its IP. Alternatively, you’re happy trusting a public DNS server, and know its IP. This part is important: all your traffic might pass through the VPN but your DNS queries, each and every site you visit, are sent to your service provider’s DNS server unless you specify otherwise (DNS leaks).
-
You don’t mind typing in a few commands.
-
You know how to use an editor like nano, vim or (for some reason) emacs. This editor needs to be installed on your server.
-
Your server is running a Debian-based system (Ubuntu, Debian GNU/Linux, etc).
-
Your server has port 443 free for use. This, for instance, means it’s no use setting up a stealth VPN system on your web-server.
-
You’re willing to start again with your existing OpenVPN install, if it’s installed already.
-
You have a GNU/Linux, or OS X laptop.
Server setup
Install OpenVPN
SSH into your server on the standard port (22). For instance, if your username is you and your server is your.server:
ssh you@your.server
Now we need to install OpenVPN on the server.
sudo apt-get install openvpn easy-rsa
Create a special user for our OpenVPN process, with highly restricted privileges
sudo adduser --system --shell /usr/sbin/nologin --no-create-home ovpn
sudo groupadd vpn
sudo usermod -g vpn ovpn
Create a new config
Now we need to change into the (new) openvpn directory:
cd /etc/openvpn
Here we’ll create a new configuration file, specifically for your ‘sheathed’ tunnel.
I prefer Vim, but will use nano here as a humane baseline:
sudo nano server.conf
Copy in the following:
# OpenVPN will connect to the localhost
local 127.0.0.1
# Define a port above 1000 for OpenVPN to connect to
port 34567
tls-server
proto tcp
dev tun
# CERTS AND KEYS
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/sheath.crt
key /etc/openvpn/easy-rsa/keys/sheath.key # This file should be kept secret
dh /etc/openvpn/easy-rsa/keys/dh4096.pem
tls-auth /etc/openvpn/easy-rsa/keys/ta.key 0 # This file is secret
txqueuelen 1000
mtu-disc yes
# Slower, more bandwidth, but I had stability problems with LZO inside a sheath
comp-lzo no
push "comp-lzo no"
keepalive 10 120
persist-key
persist-tun
# You can allow for client-to-client interaction by uncommenting the below, but
# you'll need to then disable the related rules in the iptables/firewall script
#client-to-client
status /tmp/openvpn-status.log
verb 6
client-config-dir /etc/openvpn/ccd
server 10.11.12.0 255.255.255.0
# This ensures Internet-bound traffic to-from clients can pass through the VPN
push "redirect-gateway def1"
#TODO: EDIT DNS BELOW!!!!
push "dhcp-option DNS <DNS SERVER HERE>"
push "dhcp-option DNS <DNS SERVER HERE>"
# 1024 clients is represented as /22 in the firewall script (below).
# To change max-clients to 128 clients would require using a subnet mask of /25
# there. See (https://www.aelius.com/njh/subnet_sheet.html)
max-clients 1024
# Important, otherwise OpenVPN will run as root, opening up a significant
# vulnerability.
user ovpn
group vpn
Change the port and VPN IP range, as you prefer, in this config.
Generate certificates
NOTE: If you already have a working OpenVPN install you’ll need to be sure to backup your existing keys to another folder before continuing here. You’ll also need to then edit your existing configs to reflect the new location.
OpenVPN uses a familiar certificate system for client authentication, and provides helper scripts for building them.
Copy easy-rsa scripts
Copy in the scripts directory:
sudo cp -r /usr/share/easy-rsa/ /etc/openvpn
sudo mkdir /etc/openvpn/easy-rsa/keys
Customise certificate identity
Now we want to edit some variables that are resourced by the helper scripts:
sudo nano /etc/openvpn/easy-rsa/vars
The file will have some defaults as follows:
export KEY_COUNTRY="US"
export KEY_PROVINCE="TX"
export KEY_CITY="Dallas"
export KEY_ORG="My Company Name"
export KEY_EMAIL="sammy@example.com"
export KEY_OU="MYOrganizationalUnit"
Edit the above values to represent your setup.
Further down in the file you’ll see:
export KEY_NAME="EasyRSA"
Change it to read:
export KEY_NAME="sheath"
Save the file and exit.
The Diffie-Helman Parameters
Now it’s time to generate some parameters, known as the Diffie-Helman Parameters. IMO this is a great name for a crypto-folk band, but that’s another matter entirely.
4096 bits is sufficient here, at least until quantum computing breaks the world:
sudo openssl dhparam -out /etc/openvpn/easy-rsa/keys/dh4096.pem 4096
Prep for cert generation
Now we need to change to the easy-rsa directory
cd /etc/openvpn/easy-rsa
Import (source) the variables to our shell session for use by the scripts:
source vars
Clean out the directory of any existing keys. This should be unneccessary for a fresh install. Be warned that if you have an existing OpenVPN install, this will wipe out your keys! Backup first, if so.
sudo ./clean-all
Now we’ll build the main certificate authority (CA):
sudo ./build-ca
You’ll be pressing ENTER a few times here.
Generate certs/keys for the server
sudo ./build-key-server sheath
You’ll see something like the following, but just hit ENTER to accept defaults as a challenge password is awkward in practice, especially during reconnection, and doesn’t suit our setup:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Type ‘y’ when you see the following:
Sign the certificate? [y/n]
1 out of 1 certificate requests certified, commit? [y/n]
If it all worked, you should see the following:
Write out database with 1 new entries
Data Base Updated
Generate an HMAC key, for extra security
Resourcing a ta.key ensures that a special HMAC signature is required during the authentication process. Without this signature, authentication attempts will be dropped.
sudo openvpn --genkey --secret /etc/openvpn/easy-rsa/ta.key
Certs and keys for clients
Configs can be distributed in the form of the older *.config files, with keys resourced from the given directory specified in the config itself. What is far more convenient however, is an ‘inline’ distribution of keys, whereby clients can simply load one file containing configuration variables and keys/certs therein. Such files are commonly distributed ending with .ovpn.
I have written a simple shell script to generate client *.ovpn files, with the addition that it can be used to create any number of such configs derived from a range supplied as execution arguments. The port and host assignment may seem confusing at first, but that will be explained later.
To start with, copy the below script into a new file in your keys directory:
sudo nano /etc/openvpn/easy-rsa/genkeys.sh
–
#!/bin/bash
OUTDIR=keys/clients
REMOTE=localhost
NAME=sheath
PORT=1194
source ./vars
START=$1; END=$2
OPTIONS="
client
dev tun
proto tcp
remote $REMOTE $PORT
resolv-retry infinite
script-security 2
nobind
persist-key
persist-tun
user nobody
group nogroup
tls-client
tls-auth ta.key 1
comp-lzo
verb 3
"
ovpn(){
export KEY_CN=$NAME$1
OVPN=$NAME$1.ovpn
FN=$OUTDIR/$OVPN
./pkitool $NAME$1
echo "$OPTIONS" > $FN
echo "<ca>" >> $FN
cat keys/ca.crt >> $FN
echo "</ca>" >> $FN
echo "<cert>" >> $FN
cat keys/$NAME$1.crt >> $FN
echo "</cert>" >> $FN
echo "<key>" >> $FN
cat keys/$NAME$1.key >> $FN
echo "</key>" >> $FN
echo "<tls-auth>" >> $FN
cat keys/ta.key >> $FN
echo "</tls-auth>" >> $FN
}
if [ ! -d $OUTDIR ]; then
mkdir $OUTDIR
fi
for i in $(seq $START $END)
do
ovpn $i
echo "$FN generated"
done
Ensure you are in that directory:
cd /etc/openvpn/easy-rsa/keys
Now we need to make the script executable:
sudo chmod +x genkeys.sh
Create a directory to store our client configs
sudo mkdir /etc/openvpn/easy-rsa/keys/clients
Fix an issue with local OpenVPN provided openssl.cnf file in Debian/Ubuntu installs:
sudo perl -p -i -e 's|^(subjectAltName=)|#$1|;' /etc/openvpn/easy-rsa/openssl-1.0.0.cnf
Generate some .ovpn configs
To start with, let’s generate 10 new configs, to gift to friends/colleagues etc:
sudo ./genkeys.sh 1 10
You should now be able to do the following and see your new *.ovpn files:
sudo ls keys/clients
Each config is unique, and numbered sequentially for ease of management (like revocation) later. So it follows that if you wish to generate another 5 keys at a later date, you would run the script like so:
sudo ./genkeys.sh 11 16
A single config at number 17 would be like so:
sudo ./genkeys.sh 17 17
Install stunnel on the server
Stunnel is our ‘sheath’ of encryption, in which we hide our OpenVPN traffic. Install it like so:
sudo apt-get update
sudo apt-get install stunnel4
Now create an stunnel config:
sudo nano /etc/stunnel/stunnel.conf
And put the following lines into this file:
cert = /etc/stunnel/stunnel.pem
pid = /var/run/stunnel.pid
; It's recommended to switch process ownership if started as root.
; But make sure you have these:
;setuid = nobody
;setgid = nogroup
output = /var/log/stunnel
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
[openvpn]
client = no
accept=443
connect=34567
;Use the below with a CA to harden against MiTM attacks
;verify=3
Create the stunnel certificate
This will create a cert valid for 730 days, or 2 years. Change the -days argument below if you want more or less:
cd /etc/stunnel/
sudo openssl genrsa -out key.pem 2048
sudo openssl req -new -x509 -key key.pem -out cert.pem -days 730
sudo cat key.pem cert.pem >> /etc/stunnel/stunnel.pem
A key size of 2048 is what I’ve found to be the best compromise for performance and security. 4096 would simply tap out when bandwidth wasn’t available. Again, the job of our SSL tunnel is just to mask the already-encrypted VPN traffic in the layer under it.
NOTE: If you have a Certificate Authority file, use it! This will harden the setup against MiTM attacks. You can either use the certificate separately with a CAfile=/path/to/CA.crt in your config, or just concatenate it into the stunnel.pem. Also note that if it is a separate file, the client will also need it.
You are now done setting up stunnel on the server.
Securely copy keys and configs to your laptop to distribute
Securely copy client configs to your laptop
NOTE: it’s not sane to have VPN configs on your laptop without disk-encryption and a strong password for your everyday user.
Using a UNIX derived/based system (GNU/Linux or OS X), we can copy the configs back over an encrypted tunnel. First however, we need to copy them to a place on the server we can access from our laptop remotely (the user you use for ssh shouldn’t/won’t have permissions to read the openvpn directory.
On the server we’ll copy our clients directory
sudo tar cvzf ~/sheath-clients.tar.gz /etc/openvpn/easy-rsa/keys/clients /etc/stunnel/stunnel.pem
Change the permissions so that our user can access it over the tunnel:
sudo chown $USER:$USER ~/stealth-clients.tar.gz
Now copy the configs to your laptop using the reverse scp method. On your laptop type:
scp you@your.server:sheath-clients.tar.gz ~/
As an extra precaution (from rogue programs running as a normal user reading this file on your system) you might want to ensure that root owns the file and that none else can read it
sudo chown root:root sheath-clients.tar.gz
sudo chmod go-r sheath-clients.tar.gz
Once copied, it’s advisable to remove the archive on your server:
sudo rm -f sheath-clients.tar.gz
Accessing the configs on your laptop
Unpack the archive, creating the ‘clients’ directory in turn, as follows:
sudo tar xvzf sheath-clients.tar.gz
Note that the clients directory created will be owned by the root user and you won’t be able to read it as a normal user.
Distributing the configs and stunnel.pem
Every client will need the stunnel.pem in their stunnel config directory (usually /etc/stunnel/) and a unique sheath*.ovpn on their laptops.
How you give these to other people is up to you. As they’re owned by the root user, you’ll need to work as root to read them. If you want to mail them to people, be sure to use PGP. You could use an install of Signal on your laptop to send one to your friend’s phone as an attachment, or have them connect to the same WiFi network as you and scp it to their machine. If you use a USB stick, make sure it’s clean before you put it into your laptop.
Setup a firewall on the server using iptables
Wherever you prefer on the server (for instance a ‘scripts’ directory), create a
new file:
nano sheath-tables.sh
In it, place the following lines.
# TODO change /22 (1024 clients) to /25 (128 clients)
#!/bin/sh
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
# IP FORWARDING
iptables -A FORWARD -i ifext -o tun0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -s 10.11.12.0/22 -o ifext -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.11.12.0/22 -o ifext -j MASQUERADE
# GLOBAL DoS ANTI
iptables -A INPUT -p icmp -m limit --limit 2/second --limit-burst 2 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -m limit --limit 80/minute --limit-burst 100 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m limit --limit 80/minute --limit-burst 100 -j ACCEPT
iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 22 -j DROP
iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 80 -j DROP
# If you want SMTP over the VPN, uncomment the below
#iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 25 -j ACCEPT
iptables -A INPUT -s 10.11.12.0/22 -d 10.11.12.0/22 -p tcp --destination-port 53 -j ACCEPT
iptables -A INPUT -p tcp --destination-port 53 -j DROP
iptables -A INPUT -p tcp --destination-port 25 -j DROP
We’ll need to be able to execute this as a script later, so we need to make it executable:
chmod +x sheath-tables.sh
Enable Packet Forwarding
This will ensure that traffic is allowed to be forwarded to-fro the VPN:
echo 1 > /proc/sys/net/ipv4/ip_forward
To make this setting persist across reboots, we need to edit a file:
nano /etc/sysctl.conf
Now find the line that reads:
#net.ipv4.ip_forward=1
And uncomment it, such that it looks like this:
net.ipv4.ip_forward=1
You’ve finished setting up the server.
Client setup
Stunnel needs to be installed on the client. It creates the SSL tunnel between the client and the server through which we route our OpenVPN traffic, effectively hiding it from deep packet inspection, looking like standard SSL traffic on port 443 (HTTPS)
Install stunnel
Debian based system
sudo apt-get install stunnel4
OS X (with homebrew)
brew install stunnel
OS X (with MacPorts)
port install stunnel
Copy the stunnel.pem file to the correct directory
However you do it, on each and every client stunnel.pem needs to go into /etc/stunnel/. Also make absolutely sure it is not readable by any other than the owner of the process that runs stunnel. On my GNU/Linux systems, that process is owned by the user stunnel4, created on install.
Create an stunnel config
sudo nano /etc/stunnel/stunnel.conf
Into this file copy in the following, being sure to edit the line to include your server IP.
cert = /etc/stunnel/stunnel.pem
client = yes
; Wise to do this if you start stunnel as root. Be sure the user and group actually exist!
;setuid = nobody
;setgid = nogroup
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
[openvpn]
client = yes
accept = 127.0.0.1:1194
connect = <IP OF YOUR SERVER GOES HERE>:443
; Only use the below verification option if your pem includes a CA certificate
;verify=3
Starting it up (testrun)
The best approach here, when testing, is to have two terminals open on your laptop, with one logged into your server (over ssh).
Start the firewall on the server
Assuming your sheath-tables.sh file is in your home directory on the server, you would:
sudo ./sheath-tables.sh
Check that the rules are all in place by typing:
sudo iptables -L
Start stunnel on the server
Stunnel needs to be started on the server before anything else. This will open port 443 on the server and wait for a connection from OpenVPN on the server (you read it right), connecting to localhost, and a connection from the remote client (your laptop)
sudo stunnel4 /etc/stunnel/stunnel.conf
Test it is running with
ps ax | grep stunnel
Start stunnel on the client (your laptop for now)
It’s usually no different. In most cases the same should be fine:
sudo stunnel4 /etc/stunnel/stunnel.conf
Or (on some systems):
sudo stunnel /etc/stunnel/stunnel.conf
Test it is running with
ps ax | grep stunnel
If that doesn’t work, be sure you can actually reach port 443 at all. From your laptop:
telnet <YOUR SERVER IP> 443
Start OpenVPN on the server
sudo openvpn --config /etc/openvpn/server.conf
You should see in the output that OpenVPN comes up and is waiting for connections
Start OpenVPN on the client
If you’re an OS X user, and you use OpenVPN with something like TunnelBlick, you should just be able to load in an OpenVPN config. As a test, try loading sheath1.ovpn.
If you use OpenVPN in the commandline (recommended for debugging - and in general tbh), run it like so:
sudo openvpn --config /path/to/sheath1.ovpn
You should see it come up and connect to the server. If using the command line, you should see something a little like this in the last lines of the output:
Fri Jul 6 15:51:27 2018 /sbin/ip route add 10.11.12.1/22 via 10.11.12.10
Fri Jul 6 15:51:27 2018 GID set to nogroup
Fri Jul 6 15:51:27 2018 UID set to nobody
Fri Jul 6 15:51:27 2018 Initialization Sequence Completed
Test ping the server from the client
ping 10.11.12.1
Setup routing on the client
We need to ensure all our packets flow to our VPN server via the LAN gateway. If you’re on Linux, run the following command to determine your gateway IP:
/sbin/route -n
If you’re on OS X:
netstat -r
Now add a network to our routing table.
On Linux:
sudo route add -net <SERVER IP> netmask 255.255.255.255 gw <GATEWAY IP>
On OS X:
sudo route -n add -net <SERVER IP> netmask 255.255.255.255 gw <GATEWAY IP>
Set DNS server
Don’t leak DNS to your ISP! Set your DNS to a server you trust. A good bet is a DNS server run by your server host.
On Linux
sudo echo "nameserver <DNS SERVER>" > /etc/resolv.conf
On OS X
Just use the GUI or whatever method is the fashion of the day
Quick brower test
In your browser, visit https://wtfismyip.com. You should your IP is see the IP of your new VPN server. A traceroute should also demonstrate you are routing out through the sheathed VPN.
If so, congrats!
Test for DNS leaks
Go to https://dnsleaktest.com. Run the standard test and see if your DNS is is reporting to the correct server, and not your ISP.
TODO (coming soon)
Scripts for starting up stunnel, openvpn and all routing for Linux and OS X clients