VA1DER Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Virtual Domain Mail Server for Debian 12 (Bookworm) ======
//or **How to configure your own Postfix, Dovecot, MariaDB, Lighttpd, phpmyadmin, postfixadmin, dkimpy-milter, spamassassin, wireguard, ssh, bind9 server in Debian 12 with encrypted storage
**//

Say Goodbye to Google, you're setting up the Cadilac of e-mail systems.  With it you can host many domains worth of email, give yourself and your family and friends the best vanity addresses in the world, swoop in to save the day for clubs and organizations who don't have the expertise or cash, and do it all on a small budget.  You don't need all that much in the way of a server to accomplish it.

Doing it this way is setting up a lot of moving parts.  So first think about whether you perhaps want a more turn-key solution like [[https://mailinabox.email/|Mail-in-a-box]].

That said, at the end of this process you will have an email server you knowv inside and out, one you can add arbitrary domains to, one that is very secure from outside access and which is even somewhat secure from internal (VPS host) access.

----

====== Requirements ======
You will need the following:
  - **Static IP address with PTR (reverse DNS) capability.**++☛|**(**A reverse DNS PTR record is required for deliverability.  Outgoing emails from your server will end up in the spam folders or jsut be dropped outright forever by every major email player without a reverse DNS name that matches your forward DNS name.  If you don't have a genuine static IP address from a provider that is willing to put in a reverse DNS PTR record, then don't attempt this.  Period.  Most VPS providers that give you a static IP will set up a PTR, though mine required me to send them my photo ID.**)**++
  - **A shiny new domain name and the ability to control the DNS for it.**++☛|**(**The instructions below assume you are going to self-host your own DNS, but that requires you to have two unique IP addresses and a domain provider who will make the glue records for you.  You don't have to self-host your DNS, but you will need a registrar that gives you good control of your DNS.  You need one real domain that you own.  Every other domain you host only needs to put an MX record on their DNS to say you are handling their email.  [[https://www.pairdomains.com/|Pair Domains]] is one registrar I can recommend as giving exceptional (and free) DNS control right with your domain.**)**++
  - **Server.**++☛|**(**A physical or Virtual Private Server.  Going with a VPS is the common solution.  Experience showed a VPS with ½ core (one core shared between two VPSs), 1GiB RAM and 10GiB storage is a feasible minimal server for a small one or two domain setup.  Something more robust would be 2 cores, 4GiB RAM, 40GiB storage.  Many of the design decisions here are specifically tailored to reducing memory and CPU footprint.**)**++
  - **A workstation with:**
    - WireGuard "client"
    - SSH  Either OpenSSH (9+) or, for Windows, PuTTY (0.78+) / WinSCP (6.2+)
  - **Watchdog computer/device**++☛|\\**(**It isn't an //absolute// requirement, but for the internal security portion (where we encrypt the server's mail and database storage) we need a watchdog computer.  By that what is meant is an always-on always-on computer, one that is hopefully inside your local LAN and which you can trust implicitly.  It needs to be able to perform periodic checks (ie: cron).  A spare OpenWRT router/device is an eminently great, energy efficient, and cheap solution.**)**++

----

====== Design ======
===== Components =====

  * [[https://www.postfix.org/documentation.html|Postfix]] MTA
  * [[https://doc.dovecot.org/|Dovecot]] IMAP/POP3/MDA
  * [[https://mariadb.com/kb/en/documentation/|MariaDB]] SQL Database
  * [[https://pypi.org/project/dkimpy-milter/|dkimpy-milter]] DKIM signing and verification milter
  * [[https://think.unblog.ch/en/how-to-use-sender-policy-framework-on-debian-server/|postfix-policyd-spf-python]] SPF verification tool
  * [[https://spamassassin.apache.org/doc.html|SpamAssassin]] milter with spamd for UCE control
  * [[https://redmine.lighttpd.net/projects/lighttpd/wiki|Lighttpd]] web server
  * [[https://www.phpmyadmin.net/docs/|phpMyAdmin]] web-based MariaDB administration tool
  * [[https://github.com/postfixadmin/postfixadmin?tab=readme-ov-file|postfixadmin]] Virtual Doman eMail control panel
  * [[https://github.com/roundcube/roundcubemail/wiki|RoundCube]] Webmail client
  * [[https://www.wireguard.com/|WireGuard]] VPN Tunnel

===== Considerations =====
Some of the design considerations which may need a little explanation:

**Web Server:** This design uses Lighttpd.  Lighttpd is one of Debian's three officially supported web servers, but today is in the significant minority of installations.  The decision to continue with Lighttpd has been revisited several times in the iterations leading up to this one.  While it is true that Nginex is slightly more performant on high-resource systems, Lighttpd is still by far the best way to squeeze every megabyte out of your server. If you are running on a VPS and are paying for every megabyte, then Lighttpd is the way to go.

**DKIM:**  OpenDKIM is the DKIM provider most people think of.  But it [[https://sourceforge.net/projects/opendkim/files/|hasn't been updated]] //since// Debian 7 in 2015.  A new player in town is dkimpy-milter, a switch made with the previous, Debian 11, iteration of this procedure, and it has performed excellently.  Even if it is written in Python. ;-)

**External Packaging:** A major design consideration has been to avoid the use of //any// third party packaging/dependency systems (Composer, PyPI, Go-anything, etc).  Some projects, like RoundCube, have officially adopted Composer as their plugin distribution mechanism.  This has no place on a production server, which is what we're making here.  All modules, libraries, and plugins are either supplied by Debian packages or are vetted and manually downloaded.


===== Architecture =====
{{pdfjs 820px >:wiki:debian12mailarch.pdf}}

----

====== Procedure ======
We are starting from the point where you have a brand new VPS you are starting up for the first time.  Even if you've already been using your server for some time, you may want to read these initial steps.

NOTE: Command-lines below assume the use of ''%%joe%%'' as a text editor.  It's a bit more old school than ''%%nano%%'', so feel free to adjust to your preferences.

===== Phase 1 - Batten the Hatches (External Security) =====
You've got a shiny new VPS or server.  Your first consideration is to secure it.  The goal here, in the words of András Stribik, is to make NSA analysts sad.

Security goals:
  - 256-bit level security throughout (ie: <m>2^256</m> operations to crack). 
  - All keymat transfers (including session keys) protected by crypto as strong as the security level of the keymat.  256-bit keymat should never be transported without being protected by crypto that matches its security level
  - Post-Quantum safe
  - Redundant security - SSH over WireGuard

If your VPS provider is like mine, you'll start with a remote console using something like VNC-over-https.  First order of business, make sure you've got nothing listening on your server, and if there is, and you haven't yet ensured it's secure, then shut it down.  Take a look at what's listening:
  $ sudo netstat -tuelnp

If ssh is running, turn it off until it's configured properly:
  $ sudo service ssh stop

Next we install WireGuard and secure SSH.  WireGuard gives you the ability to use pre-shared keys, which takes the guesswork out of whether or not any KEX is secure.  But since you don't have physical access to the server, you are going to be generating the pre-shared keymat on the server then transferring it off.  This means you need to establish a channel offering a level of security //equal or higher// than the keymat you are transferring.  It doesn't make much sense to generate a 256-bit security level pre-shared key on your server, and then transfer it off over ssh that is itself secured with only a 128-bit equivalent KEX.

Once this is done, it's recommended you always communicate with your sever by SSH OVER WireGuard.

==== SSH over WireGuard Remote Server Procedure ====
This procedure will let you use an untrusted and insecure connection to create a secure one in a way that is likely safe from even state-level actors unless they employ rubber-hose cryptography.  It assumes you don't have physical access to your server.  If you do, then you can sneaker-net transfer the keymat and you can likely figure out how to do that:
  - Create a temporary configuration in wireguard on your workstation.  This will generate a local private and public key.  All you need at this stage is the public key.  __Note it__.\\ For ++Windows| bring up WireGuard's window from the task tray, click the ↓ symbol beside "Add tunnel" and select "Add empty tunnel..."++\\ For ++Linux| ''%%$ wg genkey | tee TEMPORARY.key | wg pubkey > TEMPORARY.pub%%''++
  - Transfer (''%%wget%%'' on the server) the wireguard interface configuration (''%%/etc/network/interfaces.d/wg0%%'')), the ifup helper script (''%%/etc/wireguard/wg0.up%%''), and setup script (''%%wgnetgen%%'') in the [[wiki:debianmailserver#Wireguard|wireguard]] section below to the server.  There is nothing sensitive in those files yet.\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=0' -O wg0%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=1' -O wg0.up%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=2' -O wgnetgen%%''
  - Over your server's remote console, edit your WireGuard ''%%wgnetgen%%'' script to add the public key from step one, and edit the other parameters as required.  Run the script to generate the wireguard keymat and peer (client) configuration files, but don't look at them.  Don't ''%%cat%%'' or edit any of it except ''%%INITIAL.conf%%''.  That is the template for your workstation.  Combine the settings in ''%%INITIAL.conf%%'' with the private key you made in step one and use it to help you finish your initial workstation client configuration.  
  - Bring up wireguard on the server, reboot the server and connect to its wireguard using the temporary initial config on your workstation.
  - **Properly** configure SSH on the server [[wiki:debianmailserver#SSH|as per the guide below]], then enable it. Connect to it from your workstation using the server's wireguard IP address.
  - Transfer (using ''%%scp%%'') the permanent wireguard configuration for your workstation to the workstation - this config has your workstation's (semi-)permanent pre-shared key.
  - Disconnect from ssh, disconnect from wireguard, and on your workstation configure wireguard to use the new .conf.
  - Connect your workstation to the server with wireguard using the new config.  Connect with ssh over wireguard. Your server and this connection to it is as secure as you can make it remotely.  Now transfer all the remaining devices configurations off to your workstation.
  - Now edit ''/etc/wireguard/wg0'' and remove the temporary config entry from the bottom.

From now on **always** connect with ssh over wireguard.  From anywhere, no exceptions.

==== WireGuard ====
We avoid using wg-quick in this setup. Wireguard is an interface, so we will employ the normal Debian method of adding an interface.  Namely, adding a definition to /etc/network/interfaces.d/wg0

  $ sudo apt-get install wireguard-tools iptables-nft
  $ wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=0' -O wg0
  $ sudo cp wg0 /etc/network/interfaces.d/wg0
  $ sudo joe /etc/network/interfaces.d/wg0
++++ /etc/network/interfaces.d/wg0 |
<code - /etc/network/interfaces.d/wg0># VA1DER - qro.va1der.ca WireGuard interface config
#
# Interface definition file for wireguard interface wg0

# Tell the system to bring wg0 up at boot or ifup -a
auto wg0
# wg interfaces are static IPv4 interfaces
iface wg0 inet static

	# static IP address of server on a /24
	address 10.30.1.1/24

	# before ifup, create the device with this ip link command and set
	# the wg config file
	pre-up ip link add $IFACE type wireguard
	pre-up wg setconf $IFACE /etc/wireguard/$IFACE.conf

        # after ifup make sure we're forwarding the wg interface
	post-up /etc/wireguard/$IFACE.up

	# after ifdown, destroy the wg0 interface
	post-down ip link del $IFACE
# end.

</code>
++++
Make sure you edit the address to reflect what you want for your network. The examples here assume a network on ++10.30.1.0/24.| **(**//Some people think it's a security risk to re-use sample private IP addresses, and this is technically true for some definitions of "risk".  Feel free to use the example, or edit to suit.  I personally tend to use addresses in a 10.X.Y.0/24 address space for server VPNs and in 192.168.X.0/24 range for home networks, but use whatever system makes sense for you.//**)**++ Make sure any change you make above is reflected below.

Also, of course, edit the header signature to ++reflect your system | //(When I did this the first time, all of my config files we bare-bones.  No signatures, no comments.  It wasn't until I performed my first upgrade a year and a half later that I realized how hard it was to remember exactly what I'd done, and how hard it is to identify changes I had made.  Sometimes even recognizing my own configurations apart from system ones was difficult.  So now I mark every configuration file I make with a signature.  VA1DER, which is me, the hostname of the computer it was made for, and what it's for.  I do the same thing when I add sections to existing config files.  Put something in that's easy to ''%%grep%%'' for.  Also, never touch an existing config file without first ''%%cp -a config.conf config.conf.orig%%''.//)++

The above interface definition references a post-up script for making sure everything is forwarding properly.  This is stored as ''%%/etc/wireguard/wg0.up%%'':
  $ wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=1' -O wg0.up
  $ sudo cp wg0.up /etc/wireguard/wg0.up
  $ sudo joe /etc/wireguard/wg0.up
  $ sudo chmod 755 /etc/wireguard/wg0.up
++++ /etc/wireguard/wg0.up |
<code bash /etc/wireguard/wg0.up>
#!/bin/sh
# VA1DER - qro.va1der.ca local script to enable forwarding over wireguard
#
IPT="/sbin/iptables"

IN_IFACE="eth0"                   # Server's internet-facing interface name
WG_IFACE="wg0"                    # WireGuard interface name
SUB_NET="10.8.30.0/24"            # WireGuard network in CIDR
WG_PORT="51820"                   # WireGuard UDP port number

# Delete old rules, just in case
$IPT -t nat -D POSTROUTING 1 -s $SUB_NET -o $IN_IFACE -j MASQUERADE 2> /dev/null
$IPT -D INPUT -i $WG_IFACE -j ACCEPT 2> /dev/null
$IPT -D FORWARD -i $IN_IFACE -o $WG_IFACE -j ACCEPT 2> /dev/null
$IPT -D FORWARD -i $WG_IFACE -o $IN_IFACE -j ACCEPT 2> /dev/null
$IPT -D INPUT -i $IN_IFACE -p udp --dport $WG_PORT -j ACCEPT 2>/dev/null

# Now add the new ones
$IPT -t nat -I POSTROUTING 1 -s $SUB_NET -o $IN_IFACE -j MASQUERADE
$IPT -I INPUT 1 -i $WG_IFACE -j ACCEPT
$IPT -I FORWARD 1 -i $IN_IFACE -o $WG_IFACE -j ACCEPT
$IPT -I FORWARD 1 -i $WG_IFACE -o $IN_IFACE -j ACCEPT
$IPT -I INPUT 1 -i $IN_IFACE -p udp --dport $WG_PORT -j ACCEPT
# end.

</code>

Make sure you set IN_IFACE to the name of your main internet interface you see in ''%%ifconfig%%'' or ''%%ip addr show%%''.  The other params need to be consistent with your ''%%/etc/network/interfaces.d/wg0%%'' interface file.  Using port 51820 is recommended - this keeps your wireguard traffic on the same port lots of people use so it doesn't stand out.
++++

Next you'll generate the WireGuard keymat.  Here's a script for you to run on the server to generate keys and the config files for the server and clients:
++++ /root/admin/wg0/wgnetgen |
<code bash /root/admin/wg0/wgnetgen>
#!/bin/sh
#
# VA1DER - qsy.va1der.ca Generate wireguard keys and configs
#

# Edit these values for your network

# Interface name, wg0 for your first one
INTERFACE="wg0"
# Prefix part of network, any private IP area
NETPREFIX="10.30.1."
# Put the names of all the devices you'll be generating keys for - this
# doesn't have to be hostnames - any name will do.  The first one is for
# this server
HOSTS="guardian glitch hexadecimal megabyte frisket hack slash device8 device9"
# The actual full internet hostname for this server
SERVER="guardian.mydomain.ca"
# Port number for this instance - generally best to stick to 51820
PORT="51820"
# A DNS server your peers will use if you decide to route all traffic on them
DNS="9.9.9.9"
# A temporary public key for a workstation for the initial connection
PUBTEMP="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
#############################################################################


A=01
for HOST in $HOSTS; do

  # Generate the keys and save them
  touch "$A"_$HOST.key "$A"_$HOST.psk
  chmod g-rwx,o-rwx "$A"_$HOST.key "$A"_$HOST.psk
  wg genkey | tee "$A"_$HOST.key | wg pubkey > "$A"_$HOST.pub 2> /dev/null
  wg genpsk > "$A"_$HOST.psk 2> /dev/null
  chmod g-rwx,o-rwx "$A"_$HOST.key "$A"_$HOST.psk

  # Output our configs
  if [ $A -eq 1 ]; then # The first host is the "server"
    # Make the [Interface] section for the server
    touch $INTERFACE.conf
    chmod g-rwx,o-rwx $INTERFACE.conf
    printf "# $SERVER wireguard configuration for $INTERFACE\n\n# Our private key and port\n[Interface]\nPrivateKey = %s\nListenPort = $PORT\n\n# Sections for our peers (really our clients)\n\n" $(cat "$A"_$HOST.key) > $INTERFACE.conf
    rm "$A"_$HOST.psk   # We don't actually need or want a preshared key for the server
    SERVERHOST=$HOST    # Save for later when we're doing the peer configs
  else                  # Every other host is a "client"
    # Make a [Peer] section for the client in the VPN config
    printf "# $HOST\n[Peer]\nPublicKey = %s\nPresharedKey = %s\nAllowedIPs = $NETPREFIX$(expr $A + 0)/32\nPersistentKeepalive = 25\n\n" $(cat "$A"_$HOST.pub) $(cat "$A"_$HOST.psk) >> $INTERFACE.conf
    # Also generate a config file for the peer itself
    printf "# WireGuard configuration for $HOST\n\n[Interface]\nPrivateKey = %s\nAddress = $NETPREFIX$(expr $A + 0)/24\nDNS = $DNS\n\n[Peer]\nPublicKey = %s\nPresharedKey = %s\nAllowedIPs = $NETPREFIX""0/24\nEndPoint = $SERVER:$PORT\nPersistentKeepalive = 25\n"\
            $(cat "$A"_$HOST.key) $(cat 01_$SERVERHOST.pub) $(cat "$A"_$HOST.psk) > "$A"_$HOST.conf
  fi
  
  A=$(printf "%02d" $(expr $A + 1));
done

# Add a tempoary peer to the configuration file for use with the initial connection
printf "# TEMPORARY peer for the initial connection\n[Peer]\nPublicKey = $PUBTEMP\nAllowedIPs = $NETPREFIX""200/32\nPersistentKeepalive = 25\n" >> $INTERFACE.conf

# Create a temporary peer configuration template.  This doesn't have the private key, that never leaves the workstation.
printf "# Wireguard TEMPORARY configuration for one workstation. Use this as a\n# template to set up wireguard on the workstation for an initial connection.\n# Use the private key generated on the workstation.\n\n" > INITIAL.conf
printf "[Interface]\nPrivateKey = <workstationkey>\nAddress = $NETPREFIX""200/32\nDNS = $DNS\n\n[Peer]\nPublicKey = %s\nAllowedIPs = $NETPREFIX""0/24\nEndPoint = $SERVER:$PORT\nPersistentKeepalive = 25\n"\
       $(cat 01_$SERVERHOST.pub) >> INITIAL.conf
#end.

</code>
++++

For this part just ''%%sudo su -%%'' and operate as root.  Make an admin folder and a wg0 subfolder to hold your keys and device configuration files.

  $ sudo su -
  # cd ~
  # mkdir -p admin/wg0
  # cd admin/wg0
 
The ''%%wgnetgen%%'' script needs to be edited with your configuration.  You will need:
  - The network part of the private IP you chose in the interface config.
  - The list of hosts/devices that you are generating client configurations for.  You might want one for your watchdog device, and maybe a few extras for future expansion.
  - The full (FQDN) hostname of your server.  If it doesn't yet have one reachable over the internet, substitute its static ip address.
  - Port number, same as in ''%%wg0.up%%''
  - A DNS server.  This is only really needed if you decide to use this WireGuard link as a full-traffic VPN for any devices.  Use your server's upstream DNS, or any public one you like.
  - The public key from the temporary WireGuard tunnel you made in [[wiki:debianmailserver#SSH over WireGuard Remote Server Procedure|step 1 above]].  If you haven't made it yet then:\\ For ++Windows| bring up WireGuard's window from the task tray, click the ↓ symbol beside "Add tunnel" and select "Add empty tunnel…". Give it a temporary name, copy the public key off, and save it++\\ For ++Linux|''%%$ wg genkey | tee TEMPORARY.key | wg pubkey > TEMPORARY.pub && cat TEMPORARY.pub%%''\\ The above will generate the private and public keys and print the public one.  You'll have to use the private key to set up your temporary tunnel in Network Manager or whatever your distribution's method is.++

  # wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=2' -O wgnetgen
  # joe wgnetgen
  # chmod +x wgnetgen
  # ./wgnetgen
At this point, don't actually look at any of the keymat.  Don't edit or even look at ''%%wg0.conf%%''.  If you do look at it (to make sure the script is operating as it should and/or to check your values) then re-run the script again after to generate new keymat and then use the regenerated ''%%wg0.conf%%''.  You want to make your connection as secure as it can be before moving any keymat over it.  The only file you should look at is INITIAL.conf.
  # cp ./wg0.conf /etc/wireguard
  # cat INITIAL.conf
Use the settings in INITIAL.conf to help you configure your workstation's initial temporary WireGuard tunnel to the server.

Before you reboot, check to make sure that ''%%/etc/network/interfaces%%'' is set to read the files in ''%%/etc/network/interfaces.d/%%''.  The first line should be ''%%source-directory /etc/network/interfaces.d/%%'' or ''%%source /etc/network/interfaces.d/*%%''

Once you reboot, your wireguard adapter should be up. You can verify with ''%%wg%%'' and ++ifconfig. | ''%%$ sudo apt-get install net-tools%%''++
  $ sudo wg
  $ sudo ifconfig
You can also see the forwarding rules and make sure that ''%%/etc/wireguard/wg0.up%%'' was run properly:
  $ sudo iptables -L -n --line-numbers -v
  
Try and connect your workstation's initial temporary tunnel.  The above can help you diagnose issues if you have troubles connecting.  Once you successfully connect with your temporary wireguard config from your workstation, you can then set up SSH on the server (see below).

Once SSH is properly configured and running, then ''%%ssh%%'' from your workstation to the server using the server's WireGuard address. Use this connection to ''%%scp%%'' your workstation's permanent WireGuard config file (with its preshared key) off it and configure your workstation's WireGuard to use it.  Once that is done and you're connected over ''%%ssh%%'' that is over your permanent WireGuard config, then you can get the rest of your device keys and/or config files off and configure the rest of your devices.  Remember to use secure means to transfer those configurations to your devices.  A sneaker-net is your friend here. 

==== SSH ====

**Goals:** 
  - Secure a channel sufficient to transmit both its own 256-bit session key, and our 256-bit WireGuard pre-shared key with as little degradation to the security guarantee of those keys as possible.
  - Protect those pre-shared keys, and any other communication over that channel, from store-now-decrypt-later schemes for when quantum computing becomes available.

András Stribik's excellent work [[https://blog.stribik.technology/2015/01/04/secure-secure-shell.html|Secure Secure Shell]] should be required reading.  It's old, and events have overtaken it in many areas, but it has excellent explanations of //why// to configure as we will be.

Since OpenSSH adopted the sntrup761x25519-sha512@openssh.com algorithm, ++which is two very different KEXs combined into one|  **(**//This KEX combines the post-quantum-ready [[https://ntruprime.cr.yp.to/|Simplified NTRU Prime]] lattice key exchange with the traditional elliptic curve [[https://curves.xargs.org/|ed25519]] and their shared secrets hash-combined together.  Its value isn't even so much in its safety from quantum computing, which is nevertheless a plus and one of our goals.  Its real value is that it combines two radically different public key algorithms into one KEX in way that makes it as minimally secure as either and maximally secure as both. Dr. Daniel Benstein's involvement in it is also a major factor in lending it credibility.  This is the best it gets for SSH until they finally start supporting pre-shared keys.  But even if they did that, this is still the best we can get on a remote server we don't physically have access to.  I don't recommend allowing any other KEX on your server.//**)**  ++ there is no compelling reason to use anything else, and our configuration becomes much simpler.  Interestingly, 99.9% of the botnets that are always trying to probe ssh servers and find weaknesses can't even connect with it right now.

If the ssh service is running, then stop it.  Then edit your config:
  $ sudo service ssh stop
  $ sudo joe /etc/ssh/sshd_config.d/local.conf
++++ /etc/ssh/sshd_config.d/local.conf |
<code - /etc/ssh/sshd_config.d/local.conf>
# VA1DER - qro.va1der.ca's local config for sshd /etc/ssh/local.conf
#
# Some recommendations from "Secure Secure Shell", retrieved from
# https://stribika.github.io/2015/01/04/secure-secure-shell.html

# Default is only 6, but with only pubkey auth allowed, are even that many needed? 
MaxAuthTries 2

# Protocol 2 has long been the default, but let's make it explicit just for safety...
Protocol 2

# Make sure that only the users attached to the ssh group can ssh in
AllowGroups ssh

# KexAlgorighthms is much simplified since sntrup761x25519, which combines both lattice and (a trustworthy) elliptic curve
KexAlgorithms sntrup761x25519-sha512@openssh.com
# Ciphers is also much simplified with chacha20 which is both faster and more secure than AES and comes bundled with its own MAC
Ciphers chacha20-poly1305@openssh.com
# No need to specify a MAC since our only cipher comes with its own baked in, but just in case more ciphers are added, lets limit ourselves to proper ones:
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
# Pick one host key type and run with it.  No need for multiple types any more.
# ed25519 is a solid choice, small key sized and fast connections, but only 128-bit-level security
#HostKeyAlgorithms ssh-ed25519
# RSA 16384-bit is still the highest security you can use, but connections are a bit slower to start
HostKeyAlgorithms rsa-sha2-512

# Choose a host key based on the algo above
#HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa16384_key

# No passwords, just pubkey
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no

AllowTcpForwarding yes
X11Forwarding yes

# Let's keep sessions alive.  This allows you to step away from the
# computer for a bit without the connection dying.
TCPKeepAlive yes
# Server will sent a client a keepalive every 5 minutes an disconnect it if
# that fails 12 times (an hour)
ClientAliveInterval 300
ClientAliveCountMax 12
</code>
++++
The above config file:
  - Sets the KEX to sntrup761x25519 only, as discussed
  - Sets the symmetric cipher set to chacha20-poly1305 ONLY, which is both faster and more secure than AES
  - Allows only members of the ssh group to log in
  - Allows X and arbitrary TCP forwarding so you can use your secure channel to tunnel through
  - Prohibits password logins

We need to choose a host-key type.  The use of sntrup761x25519 is the best we can get for KEX, but without a host-key of similar strength the server is still vulnerable to a MitM attack.  Here is where I recommend good old-fashioned RSA, and to go right to the limit with a key size of 16384 bits.

Go to your /etc/ssh directory and generate your host key.  If you decided to go with ed25519 then one should already be made for you.  If you are going with the recommended 16384 bit RSA, then:
  $ sudo sh-keygen -t rsa -b 16384 -P '' -f /etc/ssh/ssh_host_rsa16384_key 

Go to your /etc/ssh directory and delete the host keys you won't use.  There is no reason any more to have multiple host keys.  That just confuses the issue of host key fingerprints when you connect.

Start your ssh service, and check the journal to make sure it started ok:
  $ sudo service ssh start
  $ sudo service ssh status
  
For your client, make sure you have a recent-ish copy of OpenSSH (version 9+), TinySSH (20210601+), or for Windows, PuTTY (0.78+) and WinSCP (6.2+).  Whenever I use ''%%ssh%%'' from the command line to talk to my server, I set up a shell script or ~/.ssh/config entry that explicitly connects only with the configured algos.  For example:

  $ ssh -4 -YC -o "KexAlgorithms=sntrup761x25519-sha512@openssh.com" -c "chacha20-poly1305@openssh.com" -o "HostKeyAlgorithms=rsa-sha2-512" login@10.30.1.1

You should now be ready to connect, hopefully over a WireGuard connection for better security.  Remember, your initial WireGuard connection won't have a pre-shared key.  So the security of this ssh configuration is doing the heavy lifting for securing the channel to transmit that key.  Make it count.

----

===== Phase 2 - Internal Security (OPTIONAL) =====

Running a VPS means that your server's data is available to anyone with access to the hypervisor.  Which equates to the lowest-common-denominator tech at your VPS hosting company. The easiest vector for ++Joe Noseypants|**(**// Joe could personally nosy, or Joe could be looking because someone you provide email services to sends a letter that offends someone else and Joe is executing a warrant.  Your local VPS host may or may not try to fight an overreaching warrant for a small customer.  Whatever the case is, Joe isn't the one you want to have making that decision - that's for you.//**)**++ is to simply mount your VPS's drive "container" file and go grepping around in it.

You can counter this by locating sensitive data inside an encrypted container.  Whole-disk encryption isn't feasible ++for our use case.|...because, while it would protect your entire OS, it would then require a password entered through the console at every boot. That's not something you want for a high-availability server.  Four nines availability isn't the goal, but having to manually punch in a password at the remote console at every boot is too much.  Plus, not all VPS companies offer a remote console that is active during the boot phase.++

The solution presented here is a btrfs filesystem inside of a VeraCrypt-encrypted container:
{{pdfjs 400px,250px >:wiki:debian12mailarch_crypto.pdf}}

For obvious reasons the encryption keys are not kept on the server.  This means you need a watchdog ++computer| **(**//by computer really what is meant is any "always-on" computing device - this can be almost anything, even a little OpenWRT "router" device sitting inside a home LAN.  These are basically pocket-servers anyway, eminently suitable and affordable//**)**++ in a trusted location holding the key.  Syncthing is used for signaling the watchdog when the server has been rebooted, and ssh is used on the watchdog to perform the remote mounts.

==== VeraCrypt ====
Visit Veracrypt's [[https://www.veracrypt.eu/en/Downloads.html|download page]] and get the Debian 12 package for veracrypt console.  As of this writing that was [[https://launchpad.net/veracrypt/trunk/1.26.7/+download/veracrypt-console-1.26.7-Debian-12-amd64.deb|version 1.26.7]].
  $ wget https://launchpad.net/veracrypt/trunk/1.26.7/+download/veracrypt-console-1.26.7-Debian-12-amd64.deb
  $ sudo apt-get install ./veracrypt-console-1.26.7-Debian-12-amd64.deb
  
Find out if your VPS includes AES acceleration instructions:
  $ cat /proc/cpuinfo
Look for "aes" in the flags.

Create your VeraCrypt container.  You can make it any size you like.  Experience has shown 10GiB to be a good size - but it depends heavily on the expected number of accounts and, of course, available space.
  $ sudo veracrypt -c --pim=1 --size=10G --filesystem=none --volume-type=normal /media/aegis.img
VeraCrypt will ask:
  * {{ :wiki:debian12mail_veracryptformat.png?direct&400|}}**Encryption Algorithm:**\\ The suggestion algorithm is #2, Serpent.  Though if your VPS provides AES-NI instructions then algorithm #13, Serpent(AES) will add AES to that at almost no cost. 
  * **Hash Algorithm:**\\ Algo #3, ++Blake2s|**(**//Blake2s is a relatively new algorithm that adapts the chacha20 symmetric encryption cipher into a hash.//**)**++, is recommended.  SHA-512 and SHA-256 are both solid choices too, but not as fast.
  * **Password:**\\ Use a proper password manager (KeePassXC or something similar) to generate your password.  Create one that has at least 256 bits of entropy, so on the order of 44 characters in length.  A password of sufficient entropy eliminates the need for a [[https://veracrypt.eu/en/Personal%20Iterations%20Multiplier%20%28PIM%29.html|PIM]].
  * **Entropy:**\\ You'll be asked to enter random keyboard entries.  Pasting your password in here isn't a terrible idea, as that ensures the random data has at least as much entropy as your password.

We'll be using btrfs for this filesystem, primarily for its compression support:
  $ sudo apt-get install btrfs-progs btrfs-compsize
  
Mount the Veracrypt container as a block device and format it:
  $ sudo mkdir /media/storage
  $ sudo veracrypt --pim=1 --protect-hidden=no --filesystem=none /media/aegis.img /media/storage
  $ sudo mkfs.btrfs -d single -m single -M -n 4096 -L aegis -f /dev/mapper/veracrypt1
  $ sudo mount /dev/mapper/veracrypt1 /media/storage

We'll be using both compressed (email) and uncompressed (MariaDB databases) storage:
  $ sudo mkdir /storage/compressed
  $ sudo mkdir /storage/uncompressed
  $ sudo btrfs property set /storage/compressed compression zstd

==== Syncthing ====
Syncthing is employed to signal to the watchdog computer when the server has rebooted and needs assistance mounting the above encrypted storage. Syncthing is also something that's useful to have on the server anyway, as a means of quickly and easily transferring configuration files on and off.  It is moderately secure, but without ++extra measures| **(**//Syncthing's security can be improved by forcing it to use WireGuard, but this will require the watchdog computer to be added to the WireGuard network.  That isn't a bad thing at all, though, so feel free.  It can also be forced into a hub-and-spoke star topology where directory encryption is used as a sort of poor-man's pre-shared key, but that requires one node to be untrusted and not able to easily access the files being shared. 
 Either or both these extra measures are left as an exercise for the reader.//**)**++ it should **not** be trusted with keymat or anything really sensitive.

Syncthing is now in normal Debian repositories, so you can just install it...
  $ sudo apt-get install syncthing
Debian packages make it runable as a service.  Enable it for the user you want to have run it (likely your main normal user - not root)
  $ sudo systemctl enable syncthing@<user>.service
  $ sudo service syncthing@<user> start
  $ sudo service syncthing@<user> status
By default it will listen on the loopback adapter for the GUI configuration.  You can leave this, and use ssh to tunnel to it.  An easier way is to change the GUI listen address to use WireGuard.
  $ joe /home/<user>/.config/syncthing/config.xml
Look for the line ''%%<gui enabled="true" tls="false" debugging="false">%%'' and change the next line from ''%%<address>127.0.0.1:8384</address>%%'' to ''%%<address>10.30.1.1:8384</address>%%'' (or whatever you set your server's WireGuard address to).  Then restart the service:
  $ sudo service syncthing@<user> restart
Now you can navigate a browser to: ''%%http://10.30.1.1:8384/%%'' (or whatever you set the server's GUI listen address to) and you should see the Syncthing GUI.
  - You'll be asked whether you want to enable anonymous usage reporting.  This is a production server, so it's recommended to turn that off.  You'll also be asked to set the GUI authentication password, which you should do.
  - In Actions→Settings→General you can set your server name, and verify the usage reporting you selected.
  - On the Connections tab you'll see checkboxes for "Enable NAT traversal", "Global Discovery", "Local Discovery" and "Enable Relaying".  Turn them all off.\\ {{wiki:debian12mail_syncthingconnections.png?600|Syncthing Server Connections Settings}}
  - Also on the connections tab you'll see the listen address.  Recommend ''%%quic4://0.0.0.0:22000%%''.  Click "save".
  - In Actions→Advanced→Options you'll see "Enable Crash Reporting".  This is also a good thing to turn off.
  - The default folder settings are reasonable and you can retain it.  If you do then your shared folder will be ''%%/home/<user>/Sync%%''

==== Syncthing (on Watchdog)====

The method for setting up Syncthing on your watchdog device depends on the type of device.  If you have a Debian or Raspbian based device, then setup will be similar to the server.  If you use OpenWRT, or a device that supports [[https://entware.net/|entware]], then the following method and supporting scripts can be used almost directly.

In OpenWRT you can install Syncthing and some of the other utilities the following scripts will need with:
  # opkg install syncthing coreutils-nohup logger procps-ng-ps joe joe-extras
If you are using some other type of device, then you can get Syncthing binaries for almost any architecture directly from Syncthing's [[https://syncthing.net/downloads/|download page]].

Smaller Linux devices like OpenWRT often don't have a startup mechanism for Syncthing, so you may need one:
  # mkdir /root/bin
  # joe /root/bin/ststart
++++ /root/bin/ststart |
<code bash /root/bin/ststart>
#!/bin/sh
# VA1DER - apprentice.va1der.net startup script for Syncthing
# Make sure you open up the port for syncthing in the firewall settings

# Recommend OpenWrt's full-featured nuhup package for this: 'opkg install coreutils-nohup'

# Base dir for where the Syncthing config folder and logs will go - be mindful
# of wear on your device's internal flash and perhaps locate this on a USB or
# sdcard drive as your device supports
ETCDIR=/root/admin/etc

# Use the daemonize tool to run Syncthing if you have access to it
#/usr/local/sbin/daemonize -p /var/run/syncthing.pid -u root -l /var/lock/syncthing.lock /usr/bin/syncthing serve --logfile=$ETCDIR/syncthing/syncthing.log --log-max-size=10485760 --log-max-old-files=3 --data=$ETCDIR/syncthing --config=$ETCDIR/syncthing --no-upgrade
# Otherwise nohup is a good alternative...
nohup /usr/bin/syncthing serve --logfile=$ETCDIR/syncthing/syncthing.log --log-max-size=10485760 --log-max-old-files=3 --data=$ETCDIR/syncthing --config=$ETCDIR/syncthing --no-upgrade 0<&- &>/dev/null &
# end.
</code>
++++
Ensure the directory structure used in the script exists (the above ''%%ststart%%'' script uses ''%%/root/admin/etc/syncthing/%%'').  Be mindful of the type of storage your device has and, if possible, consider using ++USB or sdcard storage locations. | **(**//On all my OpenWRT devices I make my whole /root user folder a softlink to a permanently mounted sdcard or USB device to eliminate wear on irreplaceable internal device storage.//**)**++

Since your whole server restart mechanism fails if the watchdog's Syncthing fails, a little cron job script to periodically check it is a good idea:
  # joe /root/bin/stcheck
++++ /root/bin/stcheck |
<code bash /root/bin/stcheck>
#!/bin/sh
# VA1DER - apprentice.va1der.net check to make sure Syncthing is still running

# This needs OpenWRT packages 'procps-ng-ps' and 'logger' to function

# Count the number of entries in the process list taken by syncthing
STNUM=$(/bin/ps auxwww | grep -v grep | grep syncthing | wc -l)

# If there are less than two entries it means something's wrong, restart
if [ 0$STNUM -lt 2 ]; then
  logger -t stcheck -p cron.info "syncthing not detected - restarting"
  killall -q syncthing
  sleep 2
  /root/bin/ststart   
fi
# end.
</code>
++++
...and add it to the crontab
  # crontab -e
++++ crontab |
<code - crontab>
# --------------------- Minute (0 - 59)
# |   ----------------- Hour (0 - 23)
# |   |   ------------- Day (1 - 31)
# |   |   |   --------- Month (1 - 12)
# |   |   |   |   ----- Day of week (0 - 6, Sunday = 0)
# |   |   |   |   |     Command
# -   -   -   -   -     |
 */15 *   *   *   *     /root/bin/belfry/stcheck
</code>
++++
The above runs the check every 15 minutes.


The first time you run Syncthing it will generate its config file. Be default Syncthing only listens to GUI connections on localhost.  If you're using a smaller device, you won't have a web browser on it to connect with.  You'll need to allow GUI connections from a) your device's local LAN address, or b) your device's wireguard address (if you attach it to the server's wireguard.  Don't open it up to the world (0.0.0.0), even if it's sitting safely inside your firewall.
  # joe /root/admin/etc/syncthing/config.xml
Look for the line ''%%<gui enabled="true" tls="false" debugging="false">%%'' and change the next line from ''%%<address>127.0.0.1:8384</address>%%'' to whatever your decide above.
  # killall syncthing
  # /root/bin/ststart

Now you can navigate a browser to ''%%http://<watchdogui>:8384/%%'' (whatever you set the watchdog's GUI listen address to) and you should see the Syncthing GUI:
  - {{ :wiki:debian12mail_syncthingpriv.png?direct&600|Privileged user warning}}The first time you connect to the GUI, it will give you the same notifications you had on the server asking for permission to phone home and to change the GUI password.  It will also complain about being run as root. There will be op open ports on this installation and OpenWrt has limited support for non-root anyway, so you can ignore this warning.
  - In Actions→Settings→General you can set your watchdog device's name, and verify usage reporting is as you selected it.
  - On the Connections tab you'll see checkboxes for “Enable NAT traversal”, “Global Discovery”, “Local Discovery” and “Enable Relaying”. Turn them all off.
  - Also on the connections tab, clear the **Sync Protocol Listen Addressses** textbox - delete the word ''%%default%%''
  - In Actions→Advanced→Options you'll see “Enable Crash Reporting”. This is also a good thing to turn off.
  - The default shared folder will likely be in ''%%/root/Sync%%''.  This is reasonable, but you should again be mindful of wear on your device's non-volatile storage - ''%%/root/Sync%%'' should be located somewhere on a USB or sdcard storage.  You can't move a Syncthing folder location after it's been created - you have to remove it and re-ad it.  Or, stop Syncthing, move the folder, and make the old location a soft-link to it.

You can connect your watchdog's Syncthing to the server's.
  - Navigate another browser tab to the server's Syncthing and select Actions→Show ID, click "Copy"
  - Go back to the watchdog's Syncthing, select Add Remote Device
  - Paste in the device ID from above.  You can fill in the server's Syncthing name or, if you leave it blank, it will pick up the name from the server.
  - Go to the Advanced tab.  You need to enter the server's listen address here.  If the watchdog device is on the server's WireGuard network, then you can use the server's WireGuard address.  This will add extra security for Syncthing but isn't required.  You can use the server's public hostname/IP address, and Syncthing will still be ++moderately secure |**(**//**"moderately secure"** = secure enough for our server reboot signaling and casual use moving scripts and files, but **not** secure enough to, say, transfer keymat//**)**++  For the address delete ''%%dynamic%%'' and enter: ''%%quic4://<serveraddress>:22000%%''
  - {{ :wiki:debian12mail_syncthingnewdevice.png?direct&600|New Device Request}}Go to the server's Syncthing GUI browser tab.  You should see a new device connection request there.  Check to make sure the device ID matches the watchdog's Syncthing ID and then select **Add Device**.
  - Click on the **Sharing** tab, make sure introducer and auto-accept are turned off, click to check the ''%%Sync%%'' folder, and click **Save**.
  - Return to the watchdog devices Syncthing GUI browser tab.  You should get a notification that the server wants to share the ''%%Sync%%'' folder.  Accept.

And now your watchdog and server have linked folders ''%%/home/<user>/Sync%%'' on the server should now mirror ''%%/root/Sync%%'' on the watchdog device.

==== Reboot Signalling ====

Now that we've gone through the pain of linking folders between the server and watchdog, signaling the watchdog when the server reboots is pretty painless.  On the server:

  # mkdir Sync/etc
  # echo 0 > Sync/etc/<SERVERNAME>RebootFlag
  # sudo crontab -e

Add this to the crontab:
  @reboot              sleep 30 && echo 1 > /home/<username>/Sync/etc/<SERVERNAME>RebootFlag

Now every time the server reboots, it will write a 1 to the file, which Syncthing will transfer over to the watchdog.

==== Encrypted Container Remote Mount ====

The whole purpose of the watchdog is to enable it to mount the server's encrypted container when the server reboots.  We're going to do this with SSH.  The password for the encrypted container and also the sudo password for a sudo-capable user (it might as well be your main user) are both going to be sent over SSH.  Put them in files on the watchdog:
  # touch /root/admin/etc/<SERVERNAME>_container /root/admin/etc/<SERVERNAME>_control
  # chmod 600 /root/admin/etc/<SERVERNAME>_container /root/admin/etc/<SERVERNAME>_control
  # joe /root/admin/etc/<SERVERNAME>_container
  # joe /root/admin/etc/<SERVERNAME>_control
Make sure each file consists of the exact password and one newline.  Nothing else.

The server name used for subsequent examples will be "QRO".  Make sure you change accordingly.


====== References ======
I've been at this since Debian 7, and so many of these guides are old.  But many of them are still valuable references and I include them here as most of these provided at least some guidance to me at one point or another:

==== Complete Howtos: ====
  * **Debian 4 (Etch):** [[https://www.howtoforge.com/postfix_mysql_dovecot_dspam_clamav_postgrey_rbl_debian_etch|Full Mail Server Solution w/ Virtual Domains & Users]]
  * **Debian 7 (Wheezy):** [[https://www.xenlens.com/debian-wheezy-mail-server-postfix-dovecot-sasl-mysql-postfixadmin-roundcube-spamassassin-clamav-greylist-nginx-php5/|Debian Wheezy Mail Server]]
  * **Debian 7 (Wheeze):** [[https://www.howtoforge.com/installing-lighttpd-with-php5-php-fpm-and-mysql-support-on-debian-wheezy|Installing Lighttpd With PHP5 (PHP-FPM) And MySQL Support]]
  * **Debian 9 (Stretch):** [[https://www.rosehosting.com/blog/set-up-a-mail-server-with-postfixadmin-on-debian-9|Set up a Mail Server with PostfixAdmin on Debian 9]]
  * **Debian 11 (Bullseye):** [[https://www.linuxbabe.com/mail-server/postfixadmin-create-virtual-mailboxes-debian|PostfixAdmin – Create Virtual Mailboxes on Debian]], also [[https://www.linuxbabe.com/mail-server/build-email-server-from-scratch-debian-postfix-smtp|part1]] and [[https://www.linuxbabe.com/mail-server/build-email-server-from-scratch-debian-postfix-smtp|part2]]

==== Security: ====

  * [[https://std.rocks/vpn_wireguard_buster.html|How To set up WireGuard VPN on Debian 10 Buster]]\\ Most all recent howtos use wg-quick, which I don't like at all.  This one is older, but speaks to the better way of doing it as an interface:
  * [[https://stribika.github.io/2015/01/04/secure-secure-shell.html|Secure Secure Shell]]\\ Secure Secure Shell should be required reading for anyone setting up an SSH server.  It's old, and a lot of it is no longer relevant, but it's invaluable in understanding the important parts.  Seriously, read it.
  * [[https://thelinuxcode.com/setup-ssl-lighttpd/|Securing Lighttpd with SSL/TLS Encryption: an In-Depth Walkthrough]]\\ Detailed information on securing Lighttpd

==== Milters: ====
  * [[https://colincogle.name/blog/replacing-opendkim/|Replacing OpenDKIM with dkimpy-milter]]\\ DKIM verification and signing with both RSA and ed25519 keys
  * [[https://think.unblog.ch/en/how-to-use-sender-policy-framework-on-debian-server/|How to Use PolicyD-SPF With Postfix]]\\ SPF verification (not a Milter per se but works mostly like one)

==== Other ====
  * [[https://www.linuxbabe.com/debian/install-syncthing|How to Install Synchthing on Debian desktop/server]]\\ The right way to get Syncthing running in Debian with auto-start/restart.
  *