| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| wiki:debianmailserver [2024/08/28 17:56] – [WireGuard] va1der | wiki:debianmailserver [2024/09/28 01:29] (current) – va1der |
|---|
| **// | **// |
| |
| Say Goodbye to Google, you're setting up the cadilac of e-mail systems. With it you can host many domains worth of email, give yourself and your family and friends the best vanity addresses in the world, swoop in to save the day for clubs and organizations who don't have the expertise or cash, and do it all on a small budget. You don't need all that much in the way of a server to accomplish it. | Say Goodbye to Google, you're setting up the Cadilac of e-mail systems. With it you can host many domains worth of email, give yourself and your family and friends the best vanity addresses in the world, swoop in to save the day for clubs and organizations who don't have the expertise or cash, and do it all on a small budget. You don't need all that much in the way of a server to accomplish it. |
| |
| There are easier virtual domain e-mail solutions, like [[https://mailinabox.email/|Mail-in-a-box]], but these can hide too much of the engine. Their design decisions and nuts-and-bolts are often poorly documented, leaving it difficult to add pieces, and almost impossible to change components. | Doing it this way is setting up a lot of moving parts. So first think about whether you perhaps want a more turn-key solution like [[https://mailinabox.email/|Mail-in-a-box]]. |
| |
| However doing it this way is setting up a lot of moving parts. Even following these instructions you can expect this to take a few days. | That said, at the end of this process you will have an email server you knowv inside and out, one you can add arbitrary domains to, one that is very secure from outside access and which is even somewhat secure from internal (VPS host) access. |
| | |
| That said, at the end of this process you will have an email server you can add arbitrary domains to, one that is very secure from outside access and which is even somewhat secure from internal (VPS host) access. | |
| |
| ---- | ---- |
| ====== Requirements ====== | ====== Requirements ====== |
| You will need the following: | You will need the following: |
| - **Static IP address with PTR (reverse DNS) capability.**\\ If you don't have a genuine static IP address from a provider that is willing to put in a reverse DNS PTR record, then don't attempt this. Period. Without it your email just won't be deliverable. Most VPS providers that give you a static IP will set up a PTR, though mine required me to send them my photo ID. | - **Static IP address with PTR (reverse DNS) capability.**++☛|**(**A reverse DNS PTR record is required for deliverability. Outgoing emails from your server will end up in the spam folders or jsut be dropped outright forever by every major email player without a reverse DNS name that matches your forward DNS name. If you don't have a genuine static IP address from a provider that is willing to put in a reverse DNS PTR record, then don't attempt this. Period. Most VPS providers that give you a static IP will set up a PTR, though mine required me to send them my photo ID.**)**++ |
| - **A shiny new domain name and the ability to control the DNS for it.**\\ The instructions below assume you are going to self-host your own DNS, but that requires you to have two unique IP addresses and a domain provider who will make the glue records for you. You don't have to self-host your DNS, but you will need a registrar that gives you good control of your DNS. You need one real domain that you own. Every other domain you host only needs to put an MX record on their DNS to say you are handling their email. | - **A shiny new domain name and the ability to control the DNS for it.**++☛|**(**The instructions below assume you are going to self-host your own DNS, but that requires you to have two unique IP addresses and a domain provider who will make the glue records for you. You don't have to self-host your DNS, but you will need a registrar that gives you good control of your DNS. You need one real domain that you own. Every other domain you host only needs to put an MX record on their DNS to say you are handling their email. [[https://www.pairdomains.com/|Pair Domains]] is one registrar I can recommend as giving exceptional (and free) DNS control right with your domain.**)**++ |
| - **Actual server "hardware".**\\ I personally use a VPS with 2 cores, 80GiB of space, and 4GiB of RAM. Many of my design decisions are specifically tailored to reducing memory and CPU footprint. The first VPS I ran back on Debian 7 had half a core (one core shared between two VPSs) and 1GiB of RAM. This is still a feasible setup. | - **Server.**++☛|**(**A physical or Virtual Private Server. Going with a VPS is the common solution. Experience showed a VPS with ½ core (one core shared between two VPSs), 1GiB RAM and 10GiB storage is a feasible minimal server for a small one or two domain setup. Something more robust would be 2 cores, 4GiB RAM, 40GiB storage. Many of the design decisions here are specifically tailored to reducing memory and CPU footprint.**)**++ |
| - **A workstation.**\\ This probably goes without saying, but you'll need a computer to work from to build this. On it you'll need: | - **A workstation with:** |
| - WireGuard "client" | - WireGuard "client" |
| - SSH - a good modern one with scp capability and sntrup761x25519 support. That means OpenSSH (9+), TinySSH (20210601+), or for Windows, PuTTY (0.78+) / WinSCP (6.2+) | - SSH Either OpenSSH (9+) or, for Windows, PuTTY (0.78+) / WinSCP (6.2+) |
| - **Watchdog computer**\\ It isn't an //absolute// requirement, but for the internal security portion (where we encrypt the server's mail and database storage) we need a watchdog computer. By that what is meant is an always-on always-on computer, one that is hopefully inside your local LAN and which you can trust implicitly. It needs to be able to perform periodic checks (ie: cron). A spare OpenWRT router/device is an eminently great, energy efficient, and cheap solution. | - **Watchdog computer/device**++☛|\\**(**It isn't an //absolute// requirement, but for the internal security portion (where we encrypt the server's mail and database storage) we need a watchdog computer. By that what is meant is an always-on always-on computer, one that is hopefully inside your local LAN and which you can trust implicitly. It needs to be able to perform periodic checks (ie: cron). A spare OpenWRT router/device is an eminently great, energy efficient, and cheap solution.**)**++ |
| |
| ---- | ---- |
| |
| ===== Considerations ===== | ===== Considerations ===== |
| | Some of the design considerations which may need a little explanation: |
| |
| **Web Server:** The decision to go with Lighttpd has been revisted several times in the iterations leading up to this one. Lighttpd is in the significant minority of installations today. Nginex is slightly more performant on high-resource systems, but Lighttpd is still by far the best way to squeeze every megabyte out of your server. If you are running on a VPS and are paying for every megabyte, then Lighttpd is the way to go. That said, it hasn't always been the easiest choice, and might mean a learning curve for you in the future when you add pieces. Many instructions ust assume you're going to be using Apache or Nginex. | **Web Server:** This design uses Lighttpd. Lighttpd is one of Debian's three officially supported web servers, but today is in the significant minority of installations. The decision to continue with Lighttpd has been revisited several times in the iterations leading up to this one. While it is true that Nginex is slightly more performant on high-resource systems, Lighttpd is still by far the best way to squeeze every megabyte out of your server. If you are running on a VPS and are paying for every megabyte, then Lighttpd is the way to go. |
| |
| **DKIM:** OpenDKIM is the DKIM provider most people think of. But it [[https://sourceforge.net/projects/opendkim/files/|hasn't been updated]] //since// Debian 7 in 2015. A new player in town is dkimpy-milter, a switch made with the previous, Debian 11, iteration and it has performed excellently. Even if it is written in Python. ;-) | **DKIM:** OpenDKIM is the DKIM provider most people think of. But it [[https://sourceforge.net/projects/opendkim/files/|hasn't been updated]] //since// Debian 7 in 2015. A new player in town is dkimpy-milter, a switch made with the previous, Debian 11, iteration of this procedure, and it has performed excellently. Even if it is written in Python. ;-) |
| |
| **Packaging:** A major design consideration has been to avoid the use of any third party package/dependency systems (Composer, PyPI, Go-anything, etc). Some projects, like RoundCube, have officially adopted Composer as their plugin distribution mechanism. This has no place on a production server, which is what we're making here. All modules, libraries, and plugins are either supplied by Debian packages or are vetted and manually downloaded. | **External Packaging:** A major design consideration has been to avoid the use of //any// third party packaging/dependency systems (Composer, PyPI, Go-anything, etc). Some projects, like RoundCube, have officially adopted Composer as their plugin distribution mechanism. This has no place on a production server, which is what we're making here. All modules, libraries, and plugins are either supplied by Debian packages or are vetted and manually downloaded. |
| |
| |
| |
| ====== Procedure ====== | ====== Procedure ====== |
| We are starting here from the point where you have a brand new VPS you are starting up for the first time. Even if you've already been using your server for some time, you may want to read these steps. | We are starting from the point where you have a brand new VPS you are starting up for the first time. Even if you've already been using your server for some time, you may want to read these initial steps. |
| |
| NOTE: Command-lines below assume the use of ''%%joe%%'' as my a text editor. It's a bit more old school than ''%%nano%%'', so feel free to adjust to your preferences. | NOTE: Command-lines below assume the use of ''%%joe%%'' as a text editor. It's a bit more old school than ''%%nano%%'', so feel free to adjust to your preferences. |
| |
| ===== Phase 1 - Batten the Hatches (External Security) ===== | ===== Phase 1 - Batten the Hatches (External Security) ===== |
| You've got a shiny new VPS or server. Your first consideration is to secure it. András Stribik said it best when he said "my goal with this .. is to make NSA analysts sad". This is absolutely our goal. | You've got a shiny new VPS or server. Your first consideration is to secure it. The goal here, in the words of András Stribik, is to make NSA analysts sad. |
| | |
| | Security goals: |
| | - 256-bit level security throughout (ie: <m>2^256</m> operations to crack). |
| | - All keymat transfers (including session keys) protected by crypto as strong as the security level of the keymat. 256-bit keymat should never be transported without being protected by crypto that matches its security level |
| | - Post-Quantum safe |
| | - Redundant security - SSH over WireGuard |
| |
| If your VPS provider is like mine, you'll start with a remote console using something like VNC-over-https. First order of business, make sure you've got nothing listening on your server, and if there is, and you haven't yet ensured it's secure, then shut it down. Take a look at what's listening: | If your VPS provider is like mine, you'll start with a remote console using something like VNC-over-https. First order of business, make sure you've got nothing listening on your server, and if there is, and you haven't yet ensured it's secure, then shut it down. Take a look at what's listening: |
| Next we install WireGuard and secure SSH. WireGuard gives you the ability to use pre-shared keys, which takes the guesswork out of whether or not any KEX is secure. But since you don't have physical access to the server, you are going to be generating the pre-shared keymat on the server then transferring it off. This means you need to establish a channel offering a level of security //equal or higher// than the keymat you are transferring. It doesn't make much sense to generate a 256-bit security level pre-shared key on your server, and then transfer it off over ssh that is itself secured with only a 128-bit equivalent KEX. | Next we install WireGuard and secure SSH. WireGuard gives you the ability to use pre-shared keys, which takes the guesswork out of whether or not any KEX is secure. But since you don't have physical access to the server, you are going to be generating the pre-shared keymat on the server then transferring it off. This means you need to establish a channel offering a level of security //equal or higher// than the keymat you are transferring. It doesn't make much sense to generate a 256-bit security level pre-shared key on your server, and then transfer it off over ssh that is itself secured with only a 128-bit equivalent KEX. |
| |
| Once this is done, it's recommended you always communicate with your sever by ssh OVER WireGuard. | Once this is done, it's recommended you always communicate with your sever by SSH OVER WireGuard. |
| |
| ==== SSH over WireGuard Remote Server Procedure ==== | ==== SSH over WireGuard Remote Server Procedure ==== |
| This procedure assumes you don't have physical access to your server. If you do, then you can sneaker-net transfer the keymat and you can likely figure out how to do that: | This procedure will let you use an untrusted and insecure connection to create a secure one in a way that is likely safe from even state-level actors unless they employ rubber-hose cryptography. It assumes you don't have physical access to your server. If you do, then you can sneaker-net transfer the keymat and you can likely figure out how to do that: |
| - Create a temporary configuration in wireguard on your workstation. This will generate a local private and public key. All you need at this stage is the public key. __Note it__.\\ For ++Windows| bring up WireGuard's window from the task tray, click the ↓ symbol beside "Add tunnel" and select "Add empty tunnel..."++\\ For ++Linux| ''%%$ wg genkey | tee TEMPORARY.key | wg pubkey > TEMPORARY.pub%%''++ | - Create a temporary configuration in wireguard on your workstation. This will generate a local private and public key. All you need at this stage is the public key. __Note it__.\\ For ++Windows| bring up WireGuard's window from the task tray, click the ↓ symbol beside "Add tunnel" and select "Add empty tunnel..."++\\ For ++Linux| ''%%$ wg genkey | tee TEMPORARY.key | wg pubkey > TEMPORARY.pub%%''++ |
| - Transfer (''%%wget%%'' on the server) the wireguard interface configuration (''%%/etc/network/interfaces.d/wg0%%'')), the ifup helper script (''%%/etc/wireguard/wg0.up%%''), and setup script (''%%wgnetgen%%'') in the [[wiki:debianmailserver#Wireguard|wireguard]] section below to the server. There is nothing sensitive in those files yet.\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=0' -O wg0%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=1' -O wg0.up%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=2' -O wgnetgen%%'' | - Transfer (''%%wget%%'' on the server) the wireguard interface configuration (''%%/etc/network/interfaces.d/wg0%%'')), the ifup helper script (''%%/etc/wireguard/wg0.up%%''), and setup script (''%%wgnetgen%%'') in the [[wiki:debianmailserver#Wireguard|wireguard]] section below to the server. There is nothing sensitive in those files yet.\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=0' -O wg0%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=1' -O wg0.up%%''\\ ''%%wget 'https://wiki.va1der.net/doku.php/wiki:debianmailserver?do=export_code&codeblock=2' -O wgnetgen%%'' |
| - Over your server's remote console, edit your WireGuard ''%%wgnetgen%%'' script to add the public key from step one, and edit the other parameters as required. Run the script to generate the wireguard keymat and peer (client) configuration files, but don't look at them. Don't ''%%cat%%'' or edit any of it except ''%%INITIAL.conf%%''. That is the template for your workstation. Combine the settings in ''%%INITIAL.conf%%'' with the private key you made in step one and use it to help you finish your initial workstation client configuration. | - Over your server's remote console, edit your WireGuard ''%%wgnetgen%%'' script to add the public key from step one, and edit the other parameters as required. Run the script to generate the wireguard keymat and peer (client) configuration files, but don't look at them. Don't ''%%cat%%'' or edit any of it except ''%%INITIAL.conf%%''. That is the template for your workstation. Combine the settings in ''%%INITIAL.conf%%'' with the private key you made in step one and use it to help you finish your initial workstation client configuration. |
| - Bring up wireguard on the server, reboot the server and connect to its wireguard using the temporary initial config on your workstation. | - Bring up wireguard on the server, reboot the server and connect to its wireguard using the temporary initial config on your workstation. |
| - Enable ssh on the server, and connect to it from your workstation using the server's wireguard IP address. | - **Properly** configure SSH on the server [[wiki:debianmailserver#SSH|as per the guide below]], then enable it. Connect to it from your workstation using the server's wireguard IP address. |
| - Transfer (using ''%%scp%%'') the permanent wireguard configuration for your workstation to the workstation - this config has your workstation's (semi-)permanent pre-shared key. | - Transfer (using ''%%scp%%'') the permanent wireguard configuration for your workstation to the workstation - this config has your workstation's (semi-)permanent pre-shared key. |
| - Disconnect from ssh, disconnect from wireguard, and on your workstation configure wireguard to use the new .conf. | - Disconnect from ssh, disconnect from wireguard, and on your workstation configure wireguard to use the new .conf. |
| - Connect your workstation to the server with wireguard using the new config. Connect with ssh over wireguard. | - Connect your workstation to the server with wireguard using the new config. Connect with ssh over wireguard. Your server and this connection to it is as secure as you can make it remotely. Now transfer all the remaining devices configurations off to your workstation. |
| Your server and this connection to it is as secure as you can make it remotely. Now transfer all the remaining devices configurations off to your workstation. | |
| - Now edit ''/etc/wireguard/wg0'' and remove the temporary config entry from the bottom. | - Now edit ''/etc/wireguard/wg0'' and remove the temporary config entry from the bottom. |
| |
| |
| ==== Syncthing ==== | ==== Syncthing ==== |
| Syncthing is employed to signal to the watchdog computer when the server has rebooted and needs assistance mounting the above encrypted storage. Syncthing is something that's useful to have on the server anyway, as a means of quickly and easily transferring configuration files on and off. It is moderately secure, but without ++extra measures| **(**//Syncthing's security can be improved by forcing it to use WireGuard, but this will require the watchdog computer to be added to the WireGuard network. That isn't a bad thing at all, though, so feel free. It can also be forced into a hub-and-spoke star topology where directory encryption is used as a sort of poor-man's pre-shared key, but that requires one node to be untrusted and not able to easily access the files being shared. | Syncthing is employed to signal to the watchdog computer when the server has rebooted and needs assistance mounting the above encrypted storage. Syncthing is also something that's useful to have on the server anyway, as a means of quickly and easily transferring configuration files on and off. It is moderately secure, but without ++extra measures| **(**//Syncthing's security can be improved by forcing it to use WireGuard, but this will require the watchdog computer to be added to the WireGuard network. That isn't a bad thing at all, though, so feel free. It can also be forced into a hub-and-spoke star topology where directory encryption is used as a sort of poor-man's pre-shared key, but that requires one node to be untrusted and not able to easily access the files being shared. |
| Either or both these extra measures are left as an exercise for the reader.//**)**++ it should **not** be trusted with keymat or anything really sensitive. | Either or both these extra measures are left as an exercise for the reader.//**)**++ it should **not** be trusted with keymat or anything really sensitive. |
| |
| ==== Syncthing (on Watchdog)==== | ==== Syncthing (on Watchdog)==== |
| |
| Setting up Syncthing on the watchdog device depends on the type of device. If you have a Debian or Raspbian based device, then setup will be similar to the server. If you use OpenWRT, or a device that supports [[https://entware.net/|entware]], then the following method and supporting scripts can be used almost directly. | The method for setting up Syncthing on your watchdog device depends on the type of device. If you have a Debian or Raspbian based device, then setup will be similar to the server. If you use OpenWRT, or a device that supports [[https://entware.net/|entware]], then the following method and supporting scripts can be used almost directly. |
| |
| In OpenWRT you can install Syncthing and some of the other utilities the following scripts will need with: | In OpenWRT you can install Syncthing and some of the other utilities the following scripts will need with: |
| # Make sure you open up the port for syncthing in the firewall settings | # Make sure you open up the port for syncthing in the firewall settings |
| |
| # Recommend full OpenWRT nuhup package for this: 'opkg install coreutils-nohup' | # Recommend OpenWrt's full-featured nuhup package for this: 'opkg install coreutils-nohup' |
| |
| # Base dir for where the Syncthing config folder and logs will go - be mindful | # Base dir for where the Syncthing config folder and logs will go - be mindful |
| </code> | </code> |
| ++++ | ++++ |
| Eensure the directory structure used in the script exists (the above ''%%ststart%%'' script uses ''%%/root/admin/etc/syncthing/%%''). Be mindful of the type of storage your device has and, if possible, consider using ++USB or sdcard storage locations. | **(**//On all my OpenWRT devices I make my whole /root user folder a softlink to a permanently mounted sdcard or usb device to eliminate wear on irrepaceable internal device storage.//**)**++ | Ensure the directory structure used in the script exists (the above ''%%ststart%%'' script uses ''%%/root/admin/etc/syncthing/%%''). Be mindful of the type of storage your device has and, if possible, consider using ++USB or sdcard storage locations. | **(**//On all my OpenWRT devices I make my whole /root user folder a softlink to a permanently mounted sdcard or USB device to eliminate wear on irreplaceable internal device storage.//**)**++ |
| |
| Since your whole server restart mechanism fails if the watchdog's Syncthing fails, a little cron job script to periodically check it is a good idea: | Since your whole server restart mechanism fails if the watchdog's Syncthing fails, a little cron job script to periodically check it is a good idea: |
| |
| |
| The first time you run syncthing it will generate its config file. Be default syncthing only listens to GUI connections on localhost. If you're using a smaller device, you won't have a web browser on it to connect with. So allow GUI connections from a) your device's local lan address, or b) your device's wireguard address (if you attach it to the server's wireguard. Don't open it up to the world (0.0.0.0), even if it's sitting safely inside your firewall. | The first time you run Syncthing it will generate its config file. Be default Syncthing only listens to GUI connections on localhost. If you're using a smaller device, you won't have a web browser on it to connect with. You'll need to allow GUI connections from a) your device's local LAN address, or b) your device's wireguard address (if you attach it to the server's wireguard. Don't open it up to the world (0.0.0.0), even if it's sitting safely inside your firewall. |
| # joe /root/admin/etc/syncthing/config.xml | # joe /root/admin/etc/syncthing/config.xml |
| Look for the line ''%%<gui enabled="true" tls="false" debugging="false">%%'' and change the next line from ''%%<address>127.0.0.1:8384</address>%%'' to whatever your decide above. | Look for the line ''%%<gui enabled="true" tls="false" debugging="false">%%'' and change the next line from ''%%<address>127.0.0.1:8384</address>%%'' to whatever your decide above. |
| # /root/bin/ststart | # /root/bin/ststart |
| |
| As in the server, now you can navigate a browser to ''%%http://<watchdogui>:8384/%%'' (whatever you set the watchdog's GUI listen address to) and you should see the Syncthing GUI: | Now you can navigate a browser to ''%%http://<watchdogui>:8384/%%'' (whatever you set the watchdog's GUI listen address to) and you should see the Syncthing GUI: |
| - {{ :wiki:debian12mail_syncthingpriv.png?direct&600|Privileged user warning}}The first time you connect to the GUI, it will give you the same notifications you had on the server asking for permission to phone home and to change the GUI password. It will also complain about being run as root. There will be op open ports on this installation and OpenWrt has limited support for non-root anyway, so you can ignore this warning. | - {{ :wiki:debian12mail_syncthingpriv.png?direct&600|Privileged user warning}}The first time you connect to the GUI, it will give you the same notifications you had on the server asking for permission to phone home and to change the GUI password. It will also complain about being run as root. There will be op open ports on this installation and OpenWrt has limited support for non-root anyway, so you can ignore this warning. |
| - In Actions→Settings→General you can set your watchdog device's name, and verify usage reporting is as you selected it. | - In Actions→Settings→General you can set your watchdog device's name, and verify usage reporting is as you selected it. |
| |
| And now your watchdog and server have linked folders ''%%/home/<user>/Sync%%'' on the server should now mirror ''%%/root/Sync%%'' on the watchdog device. | And now your watchdog and server have linked folders ''%%/home/<user>/Sync%%'' on the server should now mirror ''%%/root/Sync%%'' on the watchdog device. |
| | |
| | ==== Reboot Signalling ==== |
| | |
| | Now that we've gone through the pain of linking folders between the server and watchdog, signaling the watchdog when the server reboots is pretty painless. On the server: |
| | |
| | # mkdir Sync/etc |
| | # echo 0 > Sync/etc/<SERVERNAME>RebootFlag |
| | # sudo crontab -e |
| | |
| | Add this to the crontab: |
| | @reboot sleep 30 && echo 1 > /home/<username>/Sync/etc/<SERVERNAME>RebootFlag |
| | |
| | Now every time the server reboots, it will write a 1 to the file, which Syncthing will transfer over to the watchdog. |
| | |
| | ==== Encrypted Container Remote Mount ==== |
| | |
| | The whole purpose of the watchdog is to enable it to mount the server's encrypted container when the server reboots. We're going to do this with SSH. The password for the encrypted container and also the sudo password for a sudo-capable user (it might as well be your main user) are both going to be sent over SSH. Put them in files on the watchdog: |
| | # touch /root/admin/etc/<SERVERNAME>_container /root/admin/etc/<SERVERNAME>_control |
| | # chmod 600 /root/admin/etc/<SERVERNAME>_container /root/admin/etc/<SERVERNAME>_control |
| | # joe /root/admin/etc/<SERVERNAME>_container |
| | # joe /root/admin/etc/<SERVERNAME>_control |
| | Make sure each file consists of the exact password and one newline. Nothing else. |
| | |
| | The server name used for subsequent examples will be "QRO". Make sure you change accordingly. |
| | |
| |
| ====== References ====== | ====== References ====== |
