What we have

We have a Kubernetes cluster with several web services deployed for internal use.

What we want to achieve

We want to expose our internal web services to the Internet, but restrict access by requiring authorization. Access should be granted only to users authenticated through our Identity Provider (such as Google, GitHub, Keycloak, etc.).

Assumptions

For simplicity, let’s assume that both ingress-nginx and cert-manager are already deployed in the cluster.

I will use Pocket ID as Identity Provider in this tutorial. Configuration slightly differs for different providers. Check the official documentation for your provider.

For the examples in this guide, I’ll use my alikhil.dev domain:

– pocket-id.k8s.alikhil.dev - will be used for Pocket ID

• k8s.alikhil.dev - will be used for oauth2-proxy. I recommend to have higher domain for oauth2-proxy service for easier cookie setup.

• *.k8s.alikhil.dev - reserved for services deployed for internal usage

Preparation

DNS

I have added two DNS records:

1. A record for k8s.alikhil.dev pointing to ingress-nginx LoadBalancer IP address in the cluster (kubectl get svc -n ingress-nginx | grep LoadBalancer)
2. CNAME record for *.k8s.alikhil.dev pointing to k8s.alikhil.dev

Install Pocket ID

helm repo add anza-labs https://anza-labs.github.io/charts
helm upgrade --install pocket-id anza-labs/pocket-id -f ./values/pocket-id.yaml

persistence:
  data:
    enabled: true

ingress:
  # -- Specifies whether ingress should be enabled.
  enabled: true
  # -- Ingress class name.
  className: "nginx"
  # -- Annotations to add to the ingress.
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.allow-http: "true"
    kubernetes.io/tls-acme: "true"
  # -- Ingress host configuration.
  host: pocket-id.k8s.alikhil.dev
  paths:
    - path: /
      pathType: ImplementationSpecific
  # -- List of TLS configurations for the ingress.
  tls:
   - secretName: pocket-id-tls
     hosts:
       - pocket-id.k8s.alikhil.dev

Configure PocketID

Go to https://pocket-id.k8s.alikhil.dev/signup/setup and set initial configuration for Pocket ID.

Then add your passkey.

Create developers group and add yourself to the list of members.

After that, go to OIDC clients page and create one for oauth2-proxy. Set proper callback url.

Save generated Client ID and Client Secret for later use.

Installing oauth2-proxy

I am using raw k8s secrets in this tutorial, but I highly recommend storing secrets in Vault or similar services and use External Secretes Operator to deliver them to kubernetes.

kubectl create secret generic oauth2-proxy-secrets --from-literal=client-id=$CLIENT_ID --from-literal=client-secret=$CLIENT_SECRET --from-literal=cookie-secret=$(openssl rand -base64 32 | head -c 32 | base64)

helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy -f values/oauth2-proxy.yaml

# Oauth client configuration specifics
config:
  existingSecret: oauth2-proxy-secrets

  cookieName: "general-oauth2"
  # Default configuration, to be overridden
  configFile: |-
    email_domains = [ "*" ]
    upstreams = [ "file:///dev/null" ]
    skip_provider_button = true
    allowed_groups = [ "developers", "admins" ]
    cookie_secure = false
    cookie_domains = [".k8s.alikhil.dev", "k8s.alikhil.dev"]
    whitelist_domains = [ "*.k8s.alikhil.dev", "k8s.alikhil.dev" ]
    cookie_samesite = "lax"
    cookie_csrf_per_request = true
    cookie_csrf_expire = "15m"
    pass_access_token = true
    pass_authorization_header = true
    provider = "oidc"
    provider_display_name = "PocketID"
    reverse_proxy = true
    scope = "openid profile email groups"
    session_store_type = "redis"
    set_xauthrequest = true
    set_authorization_header = true
    silence_ping_logging = true
    skip_auth_preflight = true
    ssl_insecure_skip_verify = true
    ssl_upstream_insecure_skip_verify = true
    insecure_oidc_allow_unverified_email = true
    oidc_issuer_url = "https://pocket-id.k8s.alikhil.dev"
    redirect_url = "https://k8s.alikhil.dev/oauth2/callback"
    # to reduce log amount
    request_logging = false


ingress:
  enabled: true
  className: nginx
  path: /
  # Only used if API capabilities (networking.k8s.io/v1) allow it
  pathType: ImplementationSpecific
  # Used to create an Ingress record.
  hosts:
    - k8s.alikhil.dev
  labels: {}
  annotations:
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/enable-cors: 'true'
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  tls:
    # Secrets must be manually created in the namespace.
    - secretName: oauth2-proxy-tls
      hosts:
        - k8s.alikhil.dev

# Configure the session storage type, between cookie and redis
sessionStorage:
  # Can be one of the supported session storage cookie|redis
  type: redis
  redis:
    existingSecret: ""
    password: ""
    passwordKey: "redis-password"
    clientType: "standalone"

# Enables and configure the automatic deployment of the redis subchart
redis:
  # provision an instance of the redis sub-chart
  enabled: true
  architecture: standalone
  auth:
    enabled: false
  master:
    persistence:
      enabled: false

    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 1
      memory: 1Gi

Testing

Install whoami

To check oauth2-proxy we need a dummy service. I will use whoami helm chart for this.

helm repo add cowboysysop https://cowboysysop.github.io/charts/
helm install whoami cowboysysop/whoami

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.allow-http: "true"
    kubernetes.io/tls-acme: "true"
    # put oauth2-proxy domain here
    nginx.ingress.kubernetes.io/auth-signin: "https://k8s.alikhil.dev/oauth2/start?rd=https://$host$request_uri$is_args$args"
    # service-name.namespace-name
    nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.oauth-example.svc.cluster.local:80/oauth2/auth
  hosts:
  - host: whoami.k8s.alikhil.dev
    paths:
    - /
  tls:
  - hosts:
    - whoami.k8s.alikhil.dev
    secretName: whoami-cert

Perform test

Go to whoami url and check if oauth2-proxy redirects you to Pocket ID like in the demo:

Troubleshooting

Takeaways

Later, when you need to protect any service in Kubernetes with oauth2-proxy, you simply need to add two annotations to your Ingress resource:

annotations:
  nginx.ingress.kubernetes.io/auth-signin: "https://k8s.alikhil.dev/oauth2/start?rd=https://$host$request_uri$is_args$args"
  nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.oauth-example.svc.cluster.local:80/oauth2/auth

Tested with: MikroTik hEX S (RB760iGS) · RouterOS 7.x · Ubuntu 22.04 LTS · WireGuard hub-and-spoke topology

Requirements

What I have

• A Mikrotik Hex S router with a dynamic public IP address.

• Services in a cloud VM (Ubuntu 22) with a static public IP address.

• Services in a VM on my home network.

• Clients - laptops and phones - that need to access the services in my home and cloud network.

What I want

• Clients outside of my home network should be able to access services both on my home and cloud network.

• Only traffic to my home and cloud network should be routed through the VPN.

• Clients inside my home network should be able to access services on my cloud network without additional configuration.

• No external centralized service should be used.

• No open ports on my home router.

Implementation

Nowadays, there are plenty of VPN solutions like zero-tier and tailscale. However, I think they are too bloated for my humble needs and WireGuard is more than enough for that.

Because of last requirement, it’s obvious that traffic to home network should be routed though my cloud server.
So, I will use WireGuard to create a tunnel between mikrotik router and cloud server.
This way, I can access my home network from anywhere without exposing any ports on my home router.

I. Initial configuration of WireGuard on cloud VM

For the sake of reproducibility and simplicity I will use vanilla wireguard and configure it on OS level, not in docker container.

1. Install wireguard on Ubuntu 22.04

sudo apt update
sudo apt install wireguard -y

# Edit the configuration file
sudo nano /etc/sysctl.conf
# Find and uncomment the line:
net.ipv4.ip_forward=1
# Save and exit

# Then apply the changes:
sudo sysctl -p

2. Create public and private key

sudo su -
cd /etc/wireguard
# Generate private and public keys
wg genkey | tee privatekey | wg pubkey > publickey

# Check the keys
cat privatekey
cat publickey

3. Create the configuration file

I will use 172.16.10.0/24 subnet for Wireguard network.

sudo nano /etc/wireguard/wg0.conf
# Add the following lines:
[Interface]
Address = 172.16.10.1/24
ListenPort = 27277 # You can choose any port you want
Privatekey = <your_private_key> # from previous step

# These are placeholder lines for next steps, keep them commented for now
# # Mikrotik peer
# [Peer]
# AllowedIPs = 172.16.10.0/24,192.168.0.0/16
# PublicKey = mikrotik public key

# # client peer
# [Peer]
# AllowedIPs = 172.16.10.2/32
# PublicKey = first client device public key

# CRTL+X: save and exit

# Change the permissions of the configuration file
chmod 600 wg0.conf

4. Start the WireGuard interface

# Enable service to start on boot
systemctl enable wg-quick@wg0.service
systemctl daemon-reload

# Start the service
systemctl start wg-quick@wg0
# Check wg status
wg

It should show the interface is up like this

More commands to check the status of the service:

# Check the status of the service
systemctl status wg-quick@wg0
# Restart the service if needed
systemctl restart wg-quick@wg0

II. MikroTik hEX S WireGuard setup

I will use Mikrotik command line interface (CLI) to configure the router. You can use Winbox or WebFig if you prefer.

1. Create WireGuard interface

# Create WireGuard interface and it will automatically generate private and public keys
/interface wireguard add listen-port=13231 mtu=1420 name=cloud-wg
# Add the IP address to the interface
/ip add address=172.16.10.3/24 interface=cloud-wg network=172.16.10.0

# Print the keys, you will need only the public key later
/interface/wireguard/print

2. Create the peer

Here we add cloud vm as a peer to the Mikrotik router’s wireguard. The public key of the cloud server is needed here.

# In allowed-address we add IP ranges which will be routed through this peer
# 172.21.0.0/16 - subnet of docker containers on cloud server
# 172.16.10.0/16 - subnet of wireguard network
/interface wireguard peers add allowed-address=172.16.10.0/24,172.21.0.0/16 \
    endpoint-address=cloud.vm.ip.address \
    endpoint-port=27277 \
    interface=cloud-wg \
    public-key="PUT CLOUD SERVER PUBLIC KEY HERE" \
    name=peer1 persistent-keepalive=25s

# Here is a tricky part
# Routing rule for subnet of wireguard network is added automatically on adding IP address to the interface
# But we need to add explicitly the route for docker containers subnet
# You may need to adjust the distance and routing-table values
/ip route disabled=no distance=1 dst-address=172.21.0.0/16 gateway=cloud-wg routing-table=main scope=30 suppress-hw-offload=no target-scope=10

III. Add WireGuard peer (VM)

sudo nano /etc/wireguard/wg0.conf
# Uncomment the lines for Mikrotik peer
# Mikrotik peer
[Peer]
AllowedIPs = 172.16.10.0/24,192.168.0.0/16
PublicKey = mikrotik public key

# Save and exit

# Restart the WireGuard service
sudo systemctl restart wg-quick@wg0
# Check if changes are applied
wg show

You should see the Mikrotik peer in the list of peers.

# ping wireguard IP of cloud server
ping 172.16.10.1

# resolve google.com via adguard container on cloud vm
put [resolve google.com server 172.21.0.114]

# ping wireguard IP of Mikrotik router
ping 172.168.10.3

# resolve google.com via adguard container on home adguard
dig google.com @192.168.11.2

# ping wireguard IP of cloud server
ping 172.16.10.1

# resolve google.com via adguard container on cloud adguard
dig google.com @172.121.0.114

IV. Add first client device as a peer

I recommend you to use your smartphone as first client device, because it can work from both home WiFi and mobile data. This way you can test the connection from both networks.

Also, install on your smartphone:

• WireGuard app (iOS, Android)
• Network debugging app (iOS, Android)

1. Create keys for the client

Install Wireguard app for your client OS.

Then, generate public and private keys on the device, for that create config from scratch in the app and then click on Generate keypair button.

Or you can generate keys on the cloud server and then copy them to the client device.

wg genkey | tee peer-privatekey | wg pubkey > peer-publickey

2. Add the client as a peer to WireGuard config on cloud server

On the cloud server, edit the WireGuard config file and add the client as a peer.

Each time increment previous peer address by 1.

sudo nano /etc/wireguard/wg0.conf
# Uncomment the lines for client peer
# client peer
[Peer]
AllowedIPs = 172.16.10.2/32
PublicKey = put public key

# CRTL+X: save and exit

# Restart the WireGuard service
sudo systemctl restart wg-quick@wg0

# Check if changes are applied
wg show

3. Continue configuring the client device

You already have public and private keys for the client device, other configuration parameters are:

Interface

• Address - It’s address of peer in wireguard subnet. Put the same address you set in AllowedIPs field in previous step.
• DNS servers - if you have adguard/pihole running on the cloud server, you can use it as a DNS server. Put it’s IP address here.
• MTU and ListenPort - you can leave them empty, they will be set automatically.

Peer

• Endpoint - cloud server IP address and port (27277)
• Public key - public key of the cloud server
• AllowedIPs - here we put all subnets that we want to access from this current client device

Here is an example of the configuration file for the client device:

[Interface]
PrivateKey = private-key-of-client-device
Address = 172.16.10.2/32
DNS = 172.21.0.114 # IP of adguard container on cloud server

[Peer]
PublicKey = public-key-of-cloud-server
AllowedIPs = 192.168.0.0/16, 172.16.10.0/24, 172.21.0.0/16 # subnets of home, cloud and wireguard network
Endpoint = ip.of.cloud.servner:27277

That’s it! Disconnect your device from the home Wi-Fi, switch to mobile data and connect to the VPN.

Then try to:

• ping the cloud server and Mikrotik router IP addresses in wireguard subnet.
• check ports of services in docker containers on cloud and home server VM.

ping cloud	ping home	check adguard port

V. Troubleshooting

VI. Bonus. DNS configuration

Since I have 2 adguard instances and I use them as DNS servers everywhere, I will add DNS records for accessing my services:

• *.cloud.domain.com - pointing to traefik docker container on cloud server
• *.home.domain.com - pointing to traefik docker container on home server

Thus, I can access my services using domain names instead of IP addresses.

Final thoughts

I hope this tutorial was helpful for you. I will keep it updated if I find any issues or improvements.

If you have any questions or suggestions, feel free to leave a comment below.

Credits to @laroberto for their guide on LAN access with WireGuard. I followed it to set up the initial configuration and then adapted it to my needs.

Recommended reading

• Storing and Using Secrets in MikroTik RouterOS — how to avoid hardcoding passwords and tokens in your RouterOS scripts.

When I started writing my first scripts I faced a problem: how to store and use secrets in my scripts. I have found a solution and I want to share it with you.

The problem

Let’s say I want to write a script that will send me telegram notifications. To do so I need to store my telegram bot token and chat id. Since I keep my RouterOS configuration in a git repository, I don’t want to hardcode my secrets in the script.

:global sendTelegramMessage do={
    :local botToken "1234567890:ABCDEFGHIJKLMN"
    :local chatId "10000000"
    :local message "$1"

    # telegram notification
    /tool fetch url="https://api.telegram.org/bot$botToken/sendMessage\?chat_id=$chatId&text=$message" keep-result=no
}

The solution

RouterOS has low level feature /ppp secret which can be used to store secrets. However, it could be inconvenient and a bit messy to use it directly in scripts. Instead, I would like to have more high level API to store and use secrets.

And, I have one. There is post in mikrotik forum by user with nickname Amm0. Ammo has shared a script of global function which can be used to store and retrieve secrets like this:

$SECRET set mySecret password="mySecretPassword"
:put [$SECRET get mySecret]

Now, I modify my script to use this function to keep it clean and secure:

:global sendTelegramMessage do={
    :local botToken
    :set botToken "$[$SECRET get TELEGRAM_TOKEN]"
    :local chatId "$[$SECRET get CHAT_ID]"
    :local message "$1"

    # telegram notification
    /tool fetch url="https://api.telegram.org/bot$botToken/sendMessage\?chat_id=$chatId&text=$message" keep-result=no
}

### $SECRET
#   get <name>
#   set <name> password=<password>
#   remove <name
#   print
:global SECRET
:set $SECRET do={
    :global SECRET

    # helpers
    :local fixprofile do={
        :if ([/ppp profile find name="null"]) do={:put "nothing"} else={
            /ppp profile add bridge-learning=no change-tcp-mss=no local-address=0.0.0.0 name="null" only-one=yes remote-address=0.0.0.0 session-timeout=1s use-compression=no use-encryption=no use-mpls=no use-upnp=no
        }
    }
    :local lppp [:len [/ppp secret find where name=$2]]
    :local checkexist do={
        :if (lppp=0) do={
            :error "\$SECRET: cannot find $2 in secret store"
        }
    }

    # $SECRET
    :if ([:typeof $1]!="str") do={
        :put "\$SECRET"
        :put "   uses /ppp/secrets to store stuff like REST apikeys, or other sensative data"
        :put "\t\$SECRET print - prints stored secret passwords"
        :put "\t\$SECRET get <name> - gets a stored secret"
        :put "\t\$SECRET set <name> password=\"YOUR_SECRET\" - sets a secret password"
        :put "\t\$SECRET remove <name> - removes a secret"
    }

    # $SECRET print
    :if ($1~"^pr") do={
        /ppp secret print where comment~"\\\$SECRET"
        :return [:nothing]
    }

    # $SECRET get
    :if ($1~"get") do={
        $checkexist
       :return [/ppp secret get $2 password]
    }

    # $SECRET set
    :if ($1~"set|add") do={
        :if ([:typeof $password]="str") do={} else={:error "\$SECRET: password= required"}
        :if (lppp=0) do={
            /ppp secret add name=$2 password=$password
        } else={
            /ppp secret set $2 password=$password
        }
        $fixprofile
        /ppp secret set $2 comment="used by \$SECRET"
        /ppp secret set $2 profile="null"
        /ppp secret set $2 service="async"
        :return [$SECRET get $2]
    }

    # $SECRET remove
    :if ($1~"rm|rem|del") do={
        $checkexist
        :return [/ppp secret remove $2]
    }
    :error "\$SECRET: bad command"
}

Conclusion

The good thing about this approach is that secrets storing and retrieving mechanism encapsulated and can be easily changed in the future without changing the scripts. Also, it is easy to use and understand.

Keep your secrets safe and happy scripting!