Mikrotik - automation

After a [multi-step migration process](https://blog.themillhousegroup.com/search/label/mikrotik), we've finally got the Mikrotik RB2011 router doing routery-things on our local network, freeing up the Raspberry Pi that was previously somewhat shackled to this task. Now we can actually get to the main reason for adding the RB2011 in the first place - __automation__. Why would you want to do this? ### Certbot automation Every now and again, your Let's Encrypt certificate needs to be renewed. With [shorter certificate lifespans](https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days) on the horizon, [automation of certificate renewal becomes absolutely non-optional](https://www.digicert.com/blog/why-certificate-automation-is-an-absolute-must). Excellently, the Let's Encrypt `certbot` tool [supports `--pre-hook` and `--post-hook` options](https://eff-certbot.readthedocs.io/en/stable/using.html#certbot-command-line-options), where you can perform tasks to make the `certbot` process work properly in your particular setting - for example, shutting down a server on port 80 that would prevent `certbot` from being able to listen on that port during the renewal process, and then restarting it again once `certbot` is done. ### Extra security WireGuard is great, but leaving the WireGuard port open all the time, to the whole internet, seems a little too inviting. By automating a firewall rule we can allow IP address *"a.b.c.d"* ONLY when WE want access, and default to keeping that port inaccessible. ### an SO-Friendly Kid Control As mentioned in the [previous post](https://blog.themillhousegroup.com/2025/06/mikrotik-routerboard-kid-control.html), the RouterOS Kid Control feature is excellent, but it's not exactly Significant-Other-friendly. Stand by for a future post where I explore this aspect more thoroughly... #### Let's Do It. Let's Encrypt For our first bit of automation, we're going to sort out the __Let's Encrypt__ `certbot` so that it can safely be run from `cron`, by routing port 80 to the `certbot` machine temporarily (in the `--pre-hook`) and then reverting that route after the certificate was renewed (in the `--post-hook`). Of course web traffic forwarding is always a little tricky when the device doing the forwarding *also* has a web interface on port 80, so I'll *double*-port-forward this via port 8888! Here are the network *pre*-conditions for this: ```mermaid architecture-beta service internet(affinity:cloud) group home(affinity:house)[Home] service ispr(affcircle:c-firewall-green) [ISP Router] in home group mik(affcircle:c-router)[RB2011] in home service gbe(affcircle:c-switch-red)[Blocks] in mik service pi(affcircle:c-server-blue)[Pi] in home internet:B -[Port 80]-> T:ispr ispr:B -[Port 8888]-> T:gbe gbe:B -- T:pi ``` Our ISP router just blindly forwards port 80 as port 8888 to the Mikrotik, which duly drops everything. The Pi is *physically* connected to the RB2011 but never gets those packets. *After* we've run our script, we want: ```mermaid architecture-beta service internet(affinity:cloud) group home(affinity:house)[Home] service ispr(affcircle:c-firewall-green) [ISP Router] in home group mik(affcircle:c-router)[RB2011] in home service gbe(affcircle:c-switch-green)[Allows] in mik service pi(affcircle:c-server-green)[Pi] in home internet:B <-[Port 80]-> T:ispr ispr:B <-[Port 8888]-> T:gbe gbe:B <-[Port 80]-> T:pi ``` Port 8888 traffic is forwarded across the router, switched *back* to port 80, and sent to the Pi, with a suitable return path configured via NAT. All we really want here is to have (similarly to our [Kid-Control solution](https://blog.themillhousegroup.com/2025/06/mikrotik-routerboard-kid-control.html)) a router-traversing NAT path, but rather than the main rule being Source-NAT (because we're NATting network packets heading for the outside-world), this time the main rule is a `dstnat` that allows packets from the outside world to find their way to the Pi. For RouterOS, this translates to the following additions to the firewall: - A `forward` filter rule so that incoming packets don't hit the floor - A `dstnat` NAT rule (aka port-forwarding) so that packets for the router go to the Pi; AND - Because the Mikrotik is NOT the ISP router (and so the Pi knows to send packets back to the RouterBoard and NOT direct to the ISP router, which would drop them), we also need the dreaded ["hairpin" SNAT rule](https://help.mikrotik.com/docs/spaces/ROS/pages/3211299/NAT#NAT-HairpinNAT) *as well*! #### Hairpin - a deep dive I struggled for a long time after implementing only the first 2 rules in the list above, which seemed to half-work, so I just thought I was missing some vital detail in those rules. Little did I realise a _whole extra rule_ was needed. But here's what happens without the hairpin - follow the arrows!: ```mermaid architecture-beta service internet(affinity:cloud) group home(affinity:house)[Home] service ispr(affcircle:c-firewall-green) [ISP Router] in home group mik(affcircle:c-router)[RB2011] in home service gbe(affcircle:c-switch-green) in mik service pi(affcircle:c-server-green)[Pi] in home internet:B --> T:ispr ispr:B --> T:gbe gbe:R --> L:pi pi:T --> R:ispr ``` The pi, seeing traffic coming from (what appears to be) the ISP router due to the DST-NAT being employed by the RB2011, does the logical thing and addresses its responses to it. But the ISP router _isn't expecting anything from the Pi_ - it was told to port-forward to the Mikrotik - so it drops the responses. When you add the hairpin rule, it causes the packets to stop using that "roundabout" path and tricks the destination server into taking the exact same return path (hence the name): ```mermaid architecture-beta service internet(affinity:cloud) group home(affinity:house)[Home] service ispr(affcircle:c-firewall-green) [ISP Router] in home group mik(affcircle:c-router)[RB2011] in home service gbe(affcircle:c-switch-green) in mik service pi(affcircle:c-server-green)[Pi] in home internet:B --> T:ispr ispr:B <--> T:gbe gbe:R <--> L:pi ``` And so we end up with: ### Allow dst-NATted port 80 traffic to the Pi ``` /ip/firewall/filter/add chain=forward action=accept connection-nat-state=dstnat protocol=tcp dst-address=10.240.0.200 dst-port=80 comment="allow port 80 dstnatted to pi to cross router" ``` ### dst-NAT incoming port 8888 to Pi port 80 ``` /ip/firewall/nat/add chain=dstnat action=dst-nat to-addresses=10.240.0.200 to-ports=80 protocol=tcp dst-address=10.240.0.11 dst-port=8888 comment="port-forward 8888 from ISP to pi:80" ``` ### "Hairpin" the port-forwarded Pi traffic back via the RB ``` /ip/firewall/nat/add chain=srcnat action=masquerade protocol=tcp dst-address=10.240.0.200 comment="hairpin for the pi port fwd" ``` #### Testing For testing I find the `nc` command-line tool is a lot quicker to give the "I got through" feedback than a browser, which usually will have lots of retry mechanisms making things less black-and-white. On the server (`pi`), after stopping anything already on port 80 (e.g. `sudo service lighttpd stop`): ``` % sudo nc -vlk 80 Listening on [0.0.0.0] (family 2, port 80) ``` On a random client inside the `Home` network: ``` % nc -v mikrotik 8888 Connection to mikrotik port 8888 [tcp/ddi-tcp-1] succeeded! ``` #### API Control And once all that is in place, we can control it atomically by simply disabling the `forward` filter rule until we need it. The RouterOS API is not particularly-well documented and nor is it particularly consistent in its operations, but luckily some excellent Internet citizen has prepared [a Postman collection](https://tikoci.github.io/restraml/routeros-openapi3.json) that perfectly captures all of the weird-and-wonderful quirks. Fortunately, the API to enable an existing firewall filter rule is dead simple: ``` POST /ip/firewall/filter/enable Payload: { "numbers": "a list of the firewall filter rule ids to enable" } ``` So for our particular situation, where the firewall filter list shows: ``` /ip/firewall> /ip/firewall/filter/print Flags: X - disabled, I - invalid; D - dynamic 0 D ;;; special dummy rule to show fasttrack counters chain=forward action=passthrough 1 ;;; allow 10.248/13 to be forwarded(see also the NAT tab) chain=forward action=accept connection-nat-state="" src-address=10.248.0.0/13 log=no log-prefix="allow-13-subnet-to-forward" 2 X ;;; allow port 80 dstnatted to pi to cross router chain=forward action=accept connection-nat-state=dstnat protocol=tcp dst-address=10.240.0.200 dst-port=80 limit=10,20:packet log=no log-prefix="any-dst-nat" ... ``` ... so rule number `2` (the disabled one) is the target. As a `curl` command, it'll be: ``` curl -X POST -s -k -u apiuser:apipw \ http://mikrotik/rest/ip/firewall/filter/enable \ --data '{ "numbers" : "2" }' \ -H "content-type: application/json" ``` and to disable is exactly the same with the word `disable` in place of `enable`. We're finally ready to automate! # Putting it all together First let's see the state of the certificate we want to renew: ``` % sudo certbot certificate - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Found the following certs: Certificate Name: my.example.com Domains: my.example.com Expiry Date: 2025-08-28 04:02:03+00:00 (VALID: 60 days) Certificate Path: /etc/letsencrypt/live/my.example.com/fullchain.pem Private Key Path: /etc/letsencrypt/live/my.example.com/privkey.pem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ``` Let's put some suitable commands into a Bash script that will be executed as a `--pre-hook`: `/home/pi/lets-encrypt-pre-renewal-tasks.sh`: ``` #!/bin/bash echo "Stopping Lighttpd" /usr/sbin/service lighttpd stop echo "Stopped Lighttpd" echo "Enabling port 80 forwarding to Pi" curl -X POST -s -k -u apiuser:apipw \ http://mikrotik/rest/ip/firewall/filter/enable \ --data '{ "numbers" : "2" }' \ -H "content-type: application/json" echo "Port 80 forward enabled" ``` ... and similarly, but reversed, in `/home/pi/lets-encrypt-post-renewal-tasks.sh`. Let's try it in `--dry-run` mode and forcing it to renew even though it's early: ``` % sudo certbot renew \ --pre-hook="/home/pi/lets-encrypt-pre-renewal-tasks.sh" \ --post-hook="/home/pi/lets-encrypt-post-renewal-tasks.sh" \ --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/my.example.com.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Plugins selected: Authenticator standalone, Installer None Running pre-hook command: /home/pi/lets-encrypt-pre-renewal-tasks.sh Output from lets-encrypt-pre-renewal-tasks.sh: Stopping Lighttpd Stopped Lighttpd Enabling port 80 forwarding to Pi []Port 80 forward enabled Renewing an existing certificate Performing the following challenges: http-01 challenge for my.example.com Waiting for verification... Cleaning up challenges - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - new certificate deployed without reload, fullchain is /etc/letsencrypt/live/my.example.com/fullchain.pem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all renewals succeeded. The following certs have been renewed: /etc/letsencrypt/live/my.example.com/fullchain.pem (success) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Running post-hook command: /home/pi/lets-encrypt-post-renewal-tasks.sh Output from lets-encrypt-post-renewal-tasks.sh: Disabling port 80 forwarding to Pi []Port 80 forward disabled Starting Lighttpd Started Lighttpd ``` Magic! Now we can just add this to our Pi's `crontab` to run monthly, and let certbot itself decide if it actually needs to take any action: ``` # Attempt to renew the cert at 7:11am on the 13th and 26th of each month: 11 7 13,26 * * certbot renew \ --pre-hook="/home/pi/lets-encrypt-pre-renewal-tasks.sh" \ --post-hook="/home/pi/lets-encrypt-post-renewal-tasks.sh" \ >> /var/log/cert-renew.log 2>&1 ```