or How to configure your own Postfix, Dovecot, MariaDB, Lighttpd, phpmyadmin, postfixadmin, dkimpy-milter, spamassassin, wireguard, ssh, bind9 server in Debian 12 with encrypted storage

Say Goodbye to Google, you're setting up the cadilac of e-mail systems. With it you can host many domains worth of email, give yourself and your family and friends the best vanity addresses in the world, swoop in to save the day for clubs and organizations who don't have the expertise or cash, and do it all on a small budget. You don't need all that much in the way of a server to accomplish it.

There are easier virtual domain e-mail solutions, like Mail-in-a-box, but these can hide too much of the engine. Their design decisions and nuts-and-bolts are often poorly documented, leaving it difficult to add pieces, and almost impossible to change components.

However doing it this way is setting up a lot of moving parts. Even following these instructions you can expect this to take a few days.

That said, at the end of this process you will have an email server you can add arbitrary domains to, one that is very secure from outside access and which is even somewhat secure from internal (VPS host) access.

You will need the following:

1. Static IP address with PTR (reverse DNS) capability.
   If you don't have a genuine static IP address from a provider that is willing to put in a reverse DNS PTR record, then don't attempt this. Period. Without it your email just won't be deliverable. Most VPS providers that give you a static IP will set up a PTR, though mine required me to send them my photo ID.
2. A shiny new domain name and the ability to control the DNS for it.
   The instructions below assume you are going to self-host your own DNS, but that requires you to have two unique IP addresses and a domain provider who will make the glue records for you. You don't have to self-host your DNS, but you will need a registrar that gives you good control of your DNS. You need one real domain that you own. Every other domain you host only needs to put an MX record on their DNS to say you are handling their email.
3. Actual server “hardware”.
   I personally use a VPS with 2 cores, 80GiB of space, and 4GiB of RAM. Many of my design decisions are specifically tailored to reducing memory and CPU footprint. The first VPS I ran back on Debian 7 had half a core (one core shared between two VPSs) and 1GiB of RAM. This is still a feasible setup.
4. A workstation.
   This probably goes without saying, but you'll need a computer to work from to build this. On it you'll need:
   1. WireGuard “client”
   2. SSH - a good modern one with scp capability and sntrup761x25519 support. That means OpenSSH (9+), TinySSH (20210601+), or for Windows, PuTTY (0.78+) / WinSCP (6.2+)
5. Watchdog computer
   It isn't an absolute requirement, but for the internal security portion (where we encrypt the server's mail and database storage) we need a watchdog computer. By that what is meant is an always-on always-on computer, one that is hopefully inside your local LAN and which you can trust implicitly. It needs to be able to perform periodic checks (ie: cron). A spare OpenWRT router/device is an eminently great, energy efficient, and cheap solution.

• Postfix MTA
• Dovecot IMAP/POP3/MDA
• MariaDB SQL Database
• dkimpy-milter DKIM signing and verification milter
• postfix-policyd-spf-python SPF verification tool
• SpamAssassin milter with spamd for UCE control
• Lighttpd web server
• phpMyAdmin web-based MariaDB administration tool
• postfixadmin Virtual Doman eMail control panel
• RoundCube Webmail client
• WireGuard VPN Tunnel

Web Server: The decision to go with Lighttpd has been revisted several times in the iterations leading up to this one. Lighttpd is in the significant minority of installations today. Nginex is slightly more performant on high-resource systems, but Lighttpd is still by far the best way to squeeze every megabyte out of your server. If you are running on a VPS and are paying for every megabyte, then Lighttpd is the way to go. That said, it hasn't always been the easiest choice, and might mean a learning curve for you in the future when you add pieces. Many instructions ust assume you're going to be using Apache or Nginex.

DKIM: OpenDKIM is the DKIM provider most people think of. But it hasn't been updated since Debian 7 in 2015. A new player in town is dkimpy-milter, a switch made with the previous, Debian 11, iteration and it has performed excellently. Even if it is written in Python.

Packaging: A major design consideration has been to avoid the use of any third party package/dependency systems (Composer, PyPI, Go-anything, etc). Some projects, like RoundCube, have officially adopted Composer as their plugin distribution mechanism. This has no place on a production server, which is what we're making here. All modules, libraries, and plugins are either supplied by Debian packages or are vetted and manually downloaded.

We are starting here from the point where you have a brand new VPS you are starting up for the first time. Even if you've already been using your server for some time, you may want to read these steps.

NOTE: Command-lines below assume the use of joe as my a text editor. It's a bit more old school than nano, so feel free to adjust to your preferences.

You've got a shiny new VPS or server. Your first consideration is to secure it. András Stribik said it best when he said “my goal with this .. is to make NSA analysts sad”. This is absolutely our goal.

If your VPS provider is like mine, you'll get a remote console using something like VNC-over-https. First order of business, make sure you've got nothing listening on it, and if there is, if it's something that hasn't yet been vetted, then shut it down:

$ sudo netstat -tuelnp

If ssh is running, turn it off until it's configured properly:

$ sudo service ssh stop

Next we install WireGuard and secure SSH. WireGuard gives you the ability to use pre-shared keys, which takes the guesswork out of whether or not any KEX is secure. But since you don't have physical access to the server, you are going to be generating the pre-shared keymat on the server then transferring it off. This means you need to establish a channel offering a level of security equal or higher than the keymat you are transferring. It doesn't make much sense to generate a 256-bit security level pre-shared key on your server, and then transfer it off over ssh that is itself secured with only a 128-bit equivalent KEX.

Once this is done, it's recommended you always communicate with your sever by ssh OVER WireGuard.

This procedure assumes you don't have physical access to your server. If you do, then you can sneaker-net transfer the keymat and you can likely figure out how to do that:

1. Create a temporary configuration in wireguard on your workstation. This will generate a local private and public key. All you need at this stage is the public key. Note it.
   For Windows bring up WireGuard's window from the task tray, click the ↓ symbol beside “Add tunnel” and select “Add empty tunnel…”
   For Linux $ wg genkey | tee TEMPORARY.key | wg pubkey > TEMPORARY.pub
2. Transfer (wget on the server) the wireguard interface configuration (/etc/network/interfaces.d/wg0)), the ifup helper script (/etc/wireguard/wg0.up), and setup script (wgnetgen) in the wireguard section below to the server. There is nothing sensitive in those files yet.
   wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=0' -O wg0
wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=1' -O wg0.up
wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=2' -O wgnetgen
3. Over your server's remote console, edit your WireGuard wgnetgen script to add the public key from step one, and edit the other parameters as required. Run the script to generate the wireguard keymat and peer (client) configuration files, but don't look at them. Don't cat or edit any of it except INITIAL.conf. That is the template for your workstation. Combine the settings in INITIAL.conf with the private key you made in step one and use it to help you finish your initial workstation client configuration.
4. Bring up wireguard on the server, reboot the server and connect to its wireguard using the temporary initial config on your workstation.
5. Enable ssh on the server, and connect to it from your workstation using the server's wireguard IP address.
6. Transfer (using scp) the permanent wireguard configuration for your workstation to the workstation - this config has your workstation's (semi-)permanent pre-shared key.
7. Disconnect from ssh, disconnect from wireguard, and on your workstation configure wireguard to use the new .conf.
8. Connect your workstation to the server with wireguard using the new config. Connect with ssh over wireguard.

Your server and this connection to it is as secure as you can make it remotely. Now transfer all the remaining devices configurations off to your workstation.

1. Now edit /etc/wireguard/wg0 and remove the temporary config entry from the bottom.

From now on always connect with ssh over wireguard. From anywhere, no exceptions.

We avoid using wg-quick in this setup. Wireguard is an interface, so we will employ the normal Debian method of bringing up an interface. Namely, adding a definition to /etc/network/interfaces.d/wg0

The example below assumes a network on 10.30.1.0/24. (Some people think it's a security risk to re-use sample private IP addresses, and this is technically true for some definitions of “risk”. Feel free to use the example, or edit to suit. I personally tend to use addresses in a 10.X.Y.0/24 address space for server VPNs and in 192.168.X.0/24 range for home, but use whatever system makes sense for you.)

$ sudo apt-get install wireguard-tools iptables-nft
$ sudo joe /etc/network/interfaces.d/wg0

/etc/network/interfaces.d/wg0

You can see the marker tag (When I did this the first time, I just edited all my config files bare-bones. It wasn't until I did my first upgrade a year and a half later that I realized how hard it was to remember exactly what I'd done, and how hard it is to identify changes I had made. So now I mark every configuration file I make with a signature. VA1DER, which is me, the hostname of the computer it was made for, and what it's for. I do the same thing when I add sections to existing config files. Put something in that's easy to grep for. Also, never touch an existing config file without first cp -a config.conf config.conf.orig.) I use for all my local config files.

The above interface definition references a post-up script for making sure everything is forwarding properly. This is stored as /etc/wireguard/wg0.up:

$ sudo joe /etc/wireguard/wg0.up
$ sudo chmod 755 /etc/wireguard/wg0.up

/etc/wireguard/wg0.up

Next generate the keymat. For this just sudo su - and operate as root. Make an admin folder and a wg0 subfolder to hold your keys and device configuration files.

$ sudo su -
# cd ~
# mkdir -p admin/wg0
# cd admin/wg0

Here's a script for you to generate keys and config files:

/root/admin/wg0/wgnetgen

The wgnetgen script needs to be edited with your configuration. For it you need:

1. A network address
2. The list of hosts/devices that you are generating for. You might think of making extras for future devices.
3. The full hostname of your server. If it doesn't yet have one reachable over the internet, substitute its ip address.
4. Port number, same as in wg0.up
5. A DNS server. This is only really needed if you decide to use this wireguard link as a full-traffic VPN for any devices. Use your server's upstream DNS, or any public one you like.
6. A temporary public wireguard key for your workstation. For Windows you can click “Add Empty Tunnel” and it will generate one. In Linux:

   $ wg genkey | tee TEMP.key | wg pubkey > TEMP.pub

# joe wgnetgen

# ./wgnetgen

At this point, don't actually look at any of the keymat. Don't edit or look at wg0.conf. If you do look at it (to make sure the script is operating as it should and/or to check your values) then re-run the script again after to generate new keymat and then use the regenerated wg0.conf. The only file you should look at is INITIAL.conf, and use that to help you configure your workstation's wireguard.

# cp ./wg0.conf /etc/wireguard
# cat INITIAL.conf

Before you reboot, check to make sure that /etc/network/interfaces is set to read the files in /etc/network/interfaces.d/. The first line should be source-directory /etc/network/interfaces.d/ or source /etc/network/interfaces.d/*

Once you reboot, your wireguard adapter should be up. You can verify with wg and ifconfig. $ sudo apt-get install net-tools

$ sudo wg
$ sudo ifconfig

You can also see the forwarding rules and make sure that /etc/wireguard/wg0.up was run properly:

$ sudo iptables -L -n --line-numbers -v

The above can help you diagnose issues if you have troubles connecting. Once you successfully connect with your temporary wireguard config from your workstation, you can then turn on SSH in the server, ssh to it, and get your permanent workstation config file (with its preshared key) off it and configure your workstation's wireguard to use it. Once that is done and you're sshing over that, you can get the rest of your device keys and/or config files off and configure the rest of your devices. Remember to use secure means to transfer those configurations to your devices. A sneaker-net is your friend here.

Goals:

1. Secure a channel sufficient to transmit both its own 256-bit session key, and our 256-bit WireGuard pre-shared key with as little degradation to the security guarantee of those keys as possible.
2. Protect those pre-shared keys, and any other communication over that channel, from store-now-decrypt-later schemes for when quantum computing becomes available.

András Stribik's excellent work Secure Secure Shell should be required reading. It's old, and events have overtaken it in many areas, but it has excellent explanations of why to configure as we will be.

Since OpenSSH adopted the sntrup761x25519-sha512@openssh.com algorithm, which is two very different KEXs combined into one (This KEX combines the post-quantum-ready Simplified NTRU Prime lattice key exchange with the traditional elliptic curve ed25519 and their shared secrets hash-combined together. Its value isn't even so much in its safety from quantum computing, which is nevertheless a plus and one of our goals. Its real value is that it combines two radically different public key algorithms into one KEX in way that makes it as minimally secure as either and maximally secure as both. Dr. Daniel Benstein's involvement in it is also a major factor in lending it credibility. This is the best it gets for SSH until they finally start supporting pre-shared keys. But even if they did that, this is still the best we can get on a remote server we don't physically have access to. I don't recommend allowing any other KEX on your server.) there is no compelling reason to use anything else, and our configuration becomes much simpler. Interestingly, 99.9% of the botnets that are always trying to probe ssh servers and find weaknesses can't even connect with it right now.

If the ssh service is running, then stop it. Then edit your config:

$ sudo service ssh stop
$ sudo joe /etc/ssh/sshd_config.d/local.conf

/etc/ssh/sshd_config.d/local.conf

The above config file:

1. Sets the KEX to sntrup761x25519 only, as discussed
2. Sets the symmetric cipher set to chacha20-poly1305 ONLY, which is both faster and more secure than AES
3. Allows only members of the ssh group to log in
4. Allows X and arbitrary TCP forwarding so you can use your secure channel to tunnel through
5. Prohibits password logins

We need to choose a host-key type. The use of sntrup761x25519 is the best we can get for KEX, but without a host-key of similar strength the server is still vulnerable to a MitM attack. Here is where I recommend good old-fashioned RSA, and to go right to the limit with a key size of 16384 bits.

Go to your /etc/ssh directory and generate your host key. If you decided to go with ed25519 then one should already be made for you. If you are going with the recommended 16384 bit RSA, then:

$ sudo sh-keygen -t rsa -b 16384 -P '' -f /etc/ssh/ssh_host_rsa16384_key 

Go to your /etc/ssh directory and delete the host keys you won't use. There is no reason any more to have multiple host keys. That just confuses the issue of host key fingerprints when you connect.

Start your ssh service, and check the journal to make sure it started ok:

$ sudo service ssh start
$ sudo service ssh status

For your client, make sure you have a recent-ish copy of OpenSSH (version 9+), TinySSH (20210601+), or for Windows, PuTTY (0.78+) and WinSCP (6.2+). Whenever I use ssh from the command line to talk to my server, I set up a shell script or ~/.ssh/config entry that explicitly connects only with the configured algos. For example:

$ ssh -4 -YC -o "KexAlgorithms=sntrup761x25519-sha512@openssh.com" -c "chacha20-poly1305@openssh.com" -o "HostKeyAlgorithms=rsa-sha2-512" login@10.30.1.1

You should now be ready to connect, hopefully over a WireGuard connection for better security. Remember, your initial WireGuard connection won't have a pre-shared key. So the security of this ssh configuration is doing the heavy lifting for securing the channel to transmit that key. Make it count.

Running a VPS means that your server's data is available to anyone with access to the hypervisor. Which equates to the lowest-common-denominator tech at your VPS hosting company. The easiest vector for Joe Noseypants ( Joe could personally nosy, or Joe could be looking because someone you provide email services to sends a letter that offends someone else and Joe is executing a warrant. Your local VPS host may or may not try to fight an overreaching warrant for a small customer. Whatever the case is, Joe isn't the one you want to have making that decision - that's for you.) is to simply mount your VPS's drive “container” file and go grepping around in it.

You can counter this by locating sensitive data inside an encrypted container. Whole-disk encryption isn't feasible for our use case. …because, while it would protect your entire OS, it would then require a password entered through the console at every boot. That's not something you want for a high-availability server. Four nines availability isn't the goal, but having to manually punch in a password at the remote console at every boot is too much. Plus, not all VPS companies offer a remote console that is active during the boot phase.

The solution presented here is a btrfs filesystem inside of a VeraCrypt-encrypted container:

For obvious reasons the encryption keys are not kept on the server. This means you need a watchdog computer (by computer really what is meant is any “always-on” computing device - this can be almost anything, even a little OpenWRT “router” device sitting inside a home LAN. These are basically pocket-servers anyway, eminently suitable and affordable) in a trusted location holding the key. Syncthing is used for signaling the watchdog when the server has been rebooted, and ssh is used on the watchdog to perform the remote mounts.

Visit Veracrypt's download page and get the Debian 12 package for veracrypt console. As of this writing that was version 1.26.7.

$ wget https://launchpad.net/veracrypt/trunk/1.26.7/+download/veracrypt-console-1.26.7-Debian-12-amd64.deb
$ sudo apt-get install ./veracrypt-console-1.26.7-Debian-12-amd64.deb

Find out if your VPS includes AES acceleration instructions:

$ cat /proc/cpuinfo

Look for “aes” in the flags.

Create your VeraCrypt container. You can make it any size you like. Experience has shown 10GiB to be a good size - but it depends heavily on the expected number of accounts and, of course, available space.

$ sudo veracrypt -c --pim=1 --size=10G --filesystem=none --volume-type=normal /media/aegis.img

VeraCrypt will ask:

• Encryption Algorithm:
  If your VPS provides AES-NI instructions then algo #13, Serpent(AES), is recommended. Otherwise #2, Serpent
• Hash Algorithm:
  Algo #3, Blake2s, is recommended. SHA-512 and SHA-256 are both solid choices too, but not as fast
• Password:
  Use a proper password manager (KeePassXC or something similar) to generate your password. Create one that has at least 256 bits of entropy, so on the order of 44 characters.
• Entropy:
  You'll be asked to enter random keyboard entries. Piping your password in here isn't a terrible idea, as that ensures the random data has at least as much entropy as your password.

We'll be using btrfs for this filesystem, primarily for its compression support:

$ sudo apt-get install btrfs-progs btrfs-compsize

Mount the Veracrypt container as a block device and format it:

$ sudo mkdir /media/storage
$ sudo veracrypt --pim=1 --protect-hidden=no --filesystem=none /media/aegis.img /media/storage
$ sudo mkfs.btrfs -d single -m single -M -n 4096 -L aegis -f /dev/mapper/veracrypt1
$ sudo mount /dev/mapper/veracrypt1 /media/storage

We'll be using both compressed (email) and uncompressed (MariaDB databases) storage:

$ sudo mkdir /storage/compressed
$ sudo mkdir /storage/uncompressed
$ sudo btrfs property set /storage/compressed compression zstd

Syncthing is employed to signal to the watchdog computer when the server has rebooted and needs assistance mounting the above encrypted storage. Syncthing is something that's useful to have on the server anyway, as a means of quickly and easily transferring configuration files on and off. It is moderately secure, but without extra measures (Syncthing's security can be improved by forcing it to use WireGuard, but this will require the watchdog computer to be added to the WireGuard network. That isn't a bad thing at all, though, so feel free. It can also be forced into a hub-and-spoke star topology where directory encryption is used as a sort of poor-man's pre-shared key, but that requires one node to be untrusted and not able to easily access the files being shared. Either or both these extra measures are left as an exercise for the reader.) it should not be trusted with keymat or anything really sensitive.

Syncthing is now in normal Debian repositories, so you can just install it…

$ sudo apt-get install syncthing

Debian packages make it runable as a service. Enable it for the user you want to have run it (likely your main normal user - not root)

$ sudo systemctl enable syncthing@<user>.service
$ sudo service syncthing@<user> start
$ sudo service syncthing@<user> status

By default it will listen on the loopback adapter for the GUI configuration. You can leave this, and use ssh to tunnel to it. An easier way is to change the GUI listen address to use WireGuard.

$ joe /home/<user>/.config/syncthing/config.xml

Look for the line <gui enabled="true" tls="false" debugging="false"> and change the next line from <address>127.0.0.1:8384</address> to <address>10.30.1.1:8384</address> (or whatever you set your server's WireGuard address to). Then restart the service:

$ sudo service syncthing@<user> restart

Now you can navigate a browser to: http://10.30.1.1:8384/ and you should see the Syncthing GUI.

1. You'll be asked whether you want to enable anonymous usage reporting. This is a production server, so it's recommended to turn that off. You'll also be asked to set the GUI authentication password, which you should do.
2. In Actions→Settings→General you can set your server name, and verify the usage reporting you selected.
3. On the Connections tab you'll see checkboxes for “Enable NAT traversal”, “Global Discovery”, “Local Discovery” and “Enable Relaying”. Turn them all off.
   
4. Also on the connections tab you'll see the listen address. Recommend quic4://0.0.0.0:22000. Click “save”.
5. In Actions→Advanced→Options you'll see “Enable Crash Reporting”. This is also a good thing to turn off.
6. The default folder settings are reasonable and you can retain it. If you do then your shared folder will be /home/<user>/Sync

Setting up Syncthing on the watchdog device depends on the type of device. If you have a Debian or Raspbian based device, then setup will be similar to the server. If you use OpenWRT, or a device that supports entware, then the following method and supporting scripts can be used almost directly.

In OpenWRT you can install Syncthing and some of the other utilities the following scripts will need with:

# opkg install syncthing coreutils-nohup logger procps-ng-ps joe joe-extras

If you are using some other type of device, then you can get Syncthing binaries for almost any architecture directly from Syncthing's download page.

Smaller Linux devices like OpenWRT often don't have a startup mechanism for Syncthing, so you may need one:

# mkdir /root/bin
# joe /root/bin/ststart

/root/bin/ststart

You will need to ensure the directory structure used in the script exists (the above ststart script uses /root/admin/etc/syncthing/). As noted in the script, be mindful of the type of storage your device has and, if possible, consider using USB or sdcard storage locations. (On all my OpenWRT devices I make my whole /root user folder a softlink to a permanently mounted sdcard or usb device to eliminate wear on irrepaceable internal device storage.)

Since your whole server restart mechanism fails if the watchdog's Syncthing fails, a little cron job script to periodically check it is a good idea:

# joe /root/bin/stcheck

/root/bin/stcheck

…and add it to the crontab

# crontab -e

crontab

The above runs the check every 15 minutes.

The first time you run syncthing it will generate its config file. Be default syncthing only listens to GUI connections on localhost. If you're using a smaller device, you won't have a web browser on it to connect. So you will at least need to open it up to a) your device's local lan address, or b) your device's wireguard address (if you attach it to the server's wireguard. Don't open it up to the world (0.0.0.0), even if it's sitting safely inside your firewall.

# joe /root/admin/etc/syncthing/config.xml

Look for the line <gui enabled="true" tls="false" debugging="false"> and change the next line from <address>127.0.0.1:8384</address> to whatever your decide above.

I've been at this since Debian 7, and so many of these guides are old. But many of them are still valuable references and I include them here as most of these provided at least some guidance to me at one point or another:

• Debian 4 (Etch): Full Mail Server Solution w/ Virtual Domains & Users
• Debian 7 (Wheezy): Debian Wheezy Mail Server
• Debian 7 (Wheeze): Installing Lighttpd With PHP5 (PHP-FPM) And MySQL Support
• Debian 9 (Stretch): Set up a Mail Server with PostfixAdmin on Debian 9
• Debian 11 (Bullseye): PostfixAdmin – Create Virtual Mailboxes on Debian, also part1 and part2

• How To set up WireGuard VPN on Debian 10 Buster
  Most all recent howtos use wg-quick, which I don't like at all. This one is older, but speaks to the better way of doing it as an interface:
• Secure Secure Shell
  Secure Secure Shell should be required reading for anyone setting up an SSH server. It's old, and a lot of it is no longer relevant, but it's invaluable in understanding the important parts. Seriously, read it.
• Securing Lighttpd with SSL/TLS Encryption: an In-Depth Walkthrough
  Detailed information on securing Lighttpd

• Replacing OpenDKIM with dkimpy-milter
  DKIM verification and signing with both RSA and ed25519 keys
• How to Use PolicyD-SPF With Postfix
  SPF verification (not a Milter per se but works mostly like one)

• How to Install Synchthing on Debian desktop/server
  The right way to get Syncthing running in Debian with auto-start/restart.