There is a common misconception in the Kubernetes world that bothers me. The official Kubernetes documentation and most guides claim that “if you want zero downtime upgrades, just use rolling update mode on deployments”. I have learned the hard way that this simply it is not true - rolling updates alone are NOT enough for true zero-downtime deployments.

And it is not just about deployments. Your pods can be terminated for many other reasons: scaling events, node maintenance, preemption, resource constraints, and more. Without proper graceful shutdown handling, any of these events can lead to dropped requests and frustrated users.

In this post, I will share what I have learned about implementing proper graceful shutdown in Kubernetes. I will show you exactly what happens behind the scenes, provide working code examples, and back everything with real test results that clearly demonstrate the difference.

The Problem: Hidden Errors During Pod Termination

If you are running services on Kubernetes, you have probably noticed that even with rolling updates (where Kubernetes gradually replaces pods), you might still see errors during deployment. This is especially annoying when you are trying to maintain “zero-downtime” systems.

When Kubernetes needs to terminate a pod (for any reason), it follows this sequence:

1. Sends a SIGTERM signal to your container
2. Waits for a grace period (30 seconds by default)
3. If the container does not exit after the grace period, it gets brutal and sends a SIGKILL signal

The problem? Most applications do not properly handle that SIGTERM signal. They just die immediately, dropping any in-flight requests. In the real world, while most API requests complete in 100-300ms, there are often those long-running operations that take 5-15 seconds or more. Think about processing uploads, generating reports, or running complex database queries. When these longer operations get cut off, that’s when users really feel the pain.

When Does Kubernetes Terminate Pods?

Rolling updates are just one scenario where your pods might be terminated. Here are other common situations that can lead to pod terminations:

• Horizontal Pod Autoscaler Events: When HPA scales down during low-traffic periods, some pods get terminated.

• Resource Pressure: If your nodes are under resource pressure, the Kubernetes scheduler might decide to evict certain pods.

• Node Maintenance: During cluster upgrades, node draining causes many pods to be evicted.

• Spot/Preemptible Instances: If you are using cost-saving node types like spot instances, these can be reclaimed with minimal notice.

All these scenarios follow the same termination process, so implementing proper graceful shutdown handling protects you from errors in all of these cases - not just during upgrades.

Let’s Test It: Basic vs. Graceful Service

Instead of just talking about theory, I built a small lab to demonstrate the difference between proper and improper shutdown handling. I created two nearly identical Go services:

• Basic Service: A standard HTTP server with no special shutdown handling
• Graceful Service: The same service but with proper SIGTERM handling

Both services:

• Process requests that take about 4 seconds to complete (intentionally configured for easier demonstration)
• Run in the same Kubernetes cluster with identical configurations
• Serve the same endpoints

I specifically chose a 4-second processing time to make the problem obvious. While this might seem long compared to typical 100-300ms API calls, it perfectly simulates those problematic long-running operations that occur in real-world applications. The only difference between the services is how they respond to termination signals.

To test them, I wrote a simple k6 script that hammers both services with requests while triggering rolling restart of service’s deployment. Here is what happened:

Basic Service: The Error-Prone One

checks_total.......................: 695    11.450339/s
checks_succeeded...................: 97.98% 681 out of 695
checks_failed......................: 2.01%  14 out of 695

✗ status is 200
  ↳  97% — ✓ 681 / ✗ 14

http_req_failed....................: 2.01%  14 out of 696

Graceful Service: The Reliable One

checks_total.......................: 750     11.724824/s
checks_succeeded...................: 100.00% 750 out of 750
checks_failed......................: 0.00%   0 out of 750

✓ status is 200

http_req_failed........... ........: 0.00%  0 out of 751

The results speak for themselves. The basic service dropped 14 requests during the update (that is 2% of all traffic), while the graceful service handled everything perfectly without a single error.

You might think “2% it is not that bad” — but if you are doing several deployments per day and have thousands of users, that adds up to a lot of errors. Plus, in my experience, these errors tend to happen at the worst possible times.

So How Do We Fix It? The Graceful Shutdown Recipe

After digging into this problem and testing different solutions, I have put together a simple recipe for proper graceful shutdown. While my examples are in Go, the fundamental principles apply to any language or framework you are using.

Here are the key ingredients:

1. Listen for SIGTERM Signals

First, your app needs to catch that SIGTERM signal instead of ignoring it:

// Set up channel for shutdown signals
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

// Block until we receive a shutdown signal
<-stop
log.Println("Shutdown signal received")

This part is easy - you are just telling your app to wake up when Kubernetes asks it to shut down.

2. Track Your In-Flight Requests

You need to know when it is safe to shut down, so keep track of ongoing requests:

// Create a request counter
var inFlightRequests atomic.Int64

http.HandleFunc("/process", func(w http.ResponseWriter, r *http.Request) {
    // Increment counter when request starts
    inFlightRequests.Add(1)
    // do not forget to decrement when done!
    defer inFlightRequests.Add(-1)

    // Your normal request handling...
    time.Sleep(4 * time.Second)  // Simulating long-running work
})

This counter lets you check if there are still requests being processed before shutting down. it is especially important for those long-running operations that users have already waited several seconds for - the last thing they want is to see an error right before completion!

3. Separate Your Health Checks

Here is a commonly overlooked trick - you need different health check endpoints for liveness and readiness:

// Track shutdown state
var isShuttingDown atomic.Bool

// Readiness probe - returns 503 when shutting down
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
    if isShuttingDown.Load() {
        w.WriteHeader(http.StatusServiceUnavailable)
        fmt.Fprintf(w, "Shutting down, not ready")
        return
    }

    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "Ready for traffic")
})

// Liveness probe - always returns 200 (we are still alive!)
http.HandleFunc("/alive", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "I'm alive")
})

This separation is crucial. The readiness probe tells Kubernetes to stop sending new traffic, while the liveness probe says “do not kill me yet, I’m still working!”

4. The Shutdown Dance

Now for the most important part - the shutdown sequence:

// Step 1: Mark service as shutting down
isShuttingDown.Store(true)

// Step 2: Let Kubernetes notice the readiness probe failing
time.Sleep(5 * time.Second)

// Step 3: Wait for in-flight requests to finish
for inFlightRequests.Load() > 0 {
    time.Sleep(1 * time.Second)
}

// Step 4: Finally, shut down the server gracefully
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

if err := server.Shutdown(ctx); err != nil {
    log.Fatalf("Forced shutdown: %v", err)
}

I’ve found this sequence to be optimal. First, we mark ourselves as “not ready” but keep running. We pause to give Kubernetes time to notice and update its routing. Then we patiently wait until all in-flight requests finish before actually shutting down the server.

5. Configure Kubernetes Correctly

Do not forget to adjust your Kubernetes configuration:

# Use different probes for liveness and readiness
livenessProbe:
  httpGet:
    path: /alive  # Always returns OK
    port: 8080
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready  # Returns 503 during shutdown
    port: 8080
  periodSeconds: 3
  failureThreshold: 2

# Give pods enough time to shut down gracefully
terminationGracePeriodSeconds: 30 # Stops routing traffic after 2 failed checks (6 seconds)

This tells Kubernetes to wait up to 30 seconds for your app to finish processing requests before forcefully terminating it.

TL; DR; Quick Tips

If you are in a hurry, here are the key takeaways:

1. Catch SIGTERM Signals: Do not let your app be surprised when Kubernetes wants it to shut down.

2. Track In-Flight Requests: Know when it is safe to exit by counting active requests.

3. Split Your Health Checks: Use separate endpoints for liveness (am I running?) and readiness (can I take traffic?).

4. Fail Readiness First: As soon as shutdown begins, start returning “not ready” on your readiness endpoint.

5. Wait for Requests: Do not just shut down - wait for all active requests to complete first.

6. Use Built-In Shutdown: Most modern web frameworks have graceful shutdown options; use them!

7. Configure Terminaton Grace Period: Give your pods enough time to complete the shutdown sequence.

8. Test Under Load: You will not catch these issues in simple tests - you need realistic traffic patterns.

Wrap Up: Is It Worth the Extra Code?

You might be wondering if adding all this extra code is really worth it. After all, we’re only talking about a 2% error rate during pod termination events.

From my experience working with high-traffic services, I would say absolutely yes - for three reasons:

1. User Experience: Even small error rates look bad to users. Nobody wants to see “Something went wrong” messages, especially after waiting 10+ seconds for a long-running operation to complete.

2. Cascading Failures: Those errors can cascade through your system, especially if services depend on each other. Long-running requests often touch multiple critical systems.

3. Deployment Confidence: With proper graceful shutdown, you can deploy more frequently without worrying about causing problems.

The good news is that once you have implemented this pattern once, it is easy to reuse across your services. You can even create a small library or template for your organization.

In production environments where I have implemented these patterns, we have gone from seeing a spike of errors with every deployment to deploying multiple times per day with zero impact on users. that is a win in my book!

Further Reading

If you want to dive deeper into this topic, I recommend checking out the article Graceful shutdown and zero downtime deployments in Kubernetes from learnk8s.io. It provides additional technical details about graceful shutdown in Kubernetes, though it does not emphasize the critical role of readiness probes in properly implementing the pattern as we have discussed here.

For those interested in seeing the actual code I used in my testing lab, I’ve published it on GitHub with instructions for running the demo yourself.

Have you implemented graceful shutdown in your services? Did you encounter any other edge cases I didn’t cover? Let me know in the comments how this pattern has worked for you!

My devices

Computers

• Macbook Air M2 14" – work Laptop.
• Mac Mini M1 – my personal computer.
• ClockworkPi uConsole, based on RPI CM4 is a portable device for Linux tinkering.

Peripheral devices

• The UGREEN Vertical Wireless - can connect up to three devices: Two via Bluetooth and one via a 2.4GHz USB dongle.
• The Keychrone K15 Max allows me to connect up to three devices via Bluetooth.
• The Monitor Dell 27 4K UHD con USB-C (S2722QC) with two HDMI inputs and one USB-C port which allows me to connect it up to three devices.

As you can see, all peripheral devices could connect to all my computers. At least in theory.

Connecting devices

Historically my devices were numbered as follows:

1. uConsole
2. Macbook Air
3. Mac Mini

Keyboard

The keyboard connection scheme is straightforward: uConsole connects via Bluetooth to the first device, the MacBook Air, as the second device, and the Mac Mini as the third device.

Later, I can use the Fn + 1/2/3 hotkey to switch the keyboard between devices.

Mouse

The first connection of the Ugreen mouse is reserved for the 2.4 GHz USB dongle. I plugged it into my uConsole, and then my MacBook Air and Mac mini connected as the second and third devices, respectively.

To switch devices with the mouse, press the red button on the bottom.

Monitor

The MacBook Air is connected to the USB-C port, which charges the computer.

The Mac Mini connects via HDMI-1, and the uConsole connects via HDMI-2.

Easiest way to switch devices

The easiest way to switch devices on the monitor is to press the small, round button in the bottom right corner to open the settings menu.

Then, choose the desired port.

However, “easiest” does not mean “most comfortable”. Changing the monitor’s input source this way requires pressing these small buttons several times. You also need to keep track of which devices are connected to HDMI-1 and HDMI-2.

The comfortable way to switch devices

Instead of pressing buttons on the monitor and navigating its menu, I could press a hotkey on my keyboard to trigger the monitor to change input sources via its API (Display Data Channel/Command Interface Standard (DDC/CI).).

I have explained in detail how one could do it on a Mac in my another post - Monitor input source control on Mac.

I won’t go into detail here. I’ll just say that on my Mac Mini and MacBook Air, I have configured the following:

Mac Mini (#3)

• On CMD + F1 triggers switch to uConsole
• On CMD + F2 triggers switch to Macbook Air

Macbook Air (#2)

• On CMD + F1 triggers switch to uConsole
• On CMD + F3 triggers switch to Mac mini

For uConsole Linux machine (#1) I used ddcutil and configured the following hotkeys:

• Ctrl + F2 triggers switch to Macbook Air
• Ctrl + F3 triggers switch to Mac Mini

Switch sequence

Once all the devices are connected and the hotkeys are configured, the following sequence of actions is needed to switch to device #N (where N is 1, 2, or 3):

1. Switch the monitor input source by pressing CMD + F#N.
2. Switch the keyboard device by pressing Fn + #N.
3. Switch the mouse by pressing red button on the bottom of it until the indicator is under #N is blinking.

For example, let’s assume I am currently using my third device – Mac Mini, to switch to Macbook Air (#2) I do:

1. Press CMD + F2
2. Press Fn + 2
3. Press red button on the bottom of the mouse until indicator under 2 is on.

Here is a demo video:

Pros and Cons

Pros

• I don’t need a special KVM switch to switch between devices.
• I use my keyboard to switch (for two-thirds of the KVM).

Cons

• Switching the mouse takes longer since there is only one way iteration over the connected devices.
• Three actions are needed to switch all peripheral devices, as opposed to one action with a KVM.
• I still need to manually reconnect other peripheral devices, e.g. web camera.

Final thoughts

It has some limitations, but if you have a keyboard and mouse that support multiple connected devices, as well as a monitor, you won’t need a special KVM switch to switch between computers.

Thank you for reading this!

Requirements

What I have

• A Mikrotik Hex S router with a dynamic public IP address.

• Services in a cloud VM (Ubuntu 22) with a static public IP address.

• Services in a VM on my home network.

• Clients - laptops and phones - that need to access the services in my home and cloud network.

What I want

• Clients outside of my home network should be able to access services both on my home and cloud network.

• Only traffic to my home and cloud network should be routed through the VPN.

• Clients inside my home network should be able to access services on my cloud network without additional configuration.

• No external centralized service should be used.

• No open ports on my home router.

Implementation

Nowadays, there are plenty of VPN solutions like zero-tier and tailscale. However, I think they are too bloated for my humble needs and WireGuard is more than enough for that.

Because of last requirement, it’s obvious that traffic to home network should be routed though my cloud server.
So, I will use WireGuard to create a tunnel between mikrotik router and cloud server.
This way, I can access my home network from anywhere without exposing any ports on my home router.

I. Initial configuration of WireGuard on cloud VM

For the sake of reproducibility and simplicity I will use vanilla wireguard and configure it on OS level, not in docker container.

1. Install wireguard on Ubuntu 22.04

sudo apt update
sudo apt install wireguard -y

# Edit the configuration file
sudo nano /etc/sysctl.conf
# Find and uncomment the line:
net.ipv4.ip_forward=1
# Save and exit

# Then apply the changes:
sudo sysctl -p

2. Create public and private key

sudo su -
cd /etc/wireguard
# Generate private and public keys
wg genkey | tee privatekey | wg pubkey > publickey

# Check the keys
cat privatekey
cat publickey

3. Create the configuration file

I will use 172.16.10.0/24 subnet for Wireguard network.

sudo nano /etc/wireguard/wg0.conf
# Add the following lines:
[Interface]
Address = 172.16.10.1/24
ListenPort = 27277 # You can choose any port you want
Privatekey = <your_private_key> # from previous step

# These are placeholder lines for next steps, keep them commented for now
# # Mikrotik peer
# [Peer]
# AllowedIPs = 172.16.10.0/24,192.168.0.0/16
# PublicKey = mikrotik public key

# # client peer
# [Peer]
# AllowedIPs = 172.16.10.2/32
# PublicKey = first client device public key

# CRTL+X: save and exit

# Change the permissions of the configuration file
chmod 600 wg0.conf

4. Start the WireGuard interface

# Enable service to start on boot
systemctl enable wg-quick@wg0.service
systemctl daemon-reload

# Start the service
systemctl start wg-quick@wg0
# Check wg status
wg

It should show the interface is up like this

More commands to check the status of the service:

# Check the status of the service
systemctl status wg-quick@wg0
# Restart the service if needed
systemctl restart wg-quick@wg0

II. Mikrotik configuration

I will use Mikrotik command line interface (CLI) to configure the router. You can use Winbox or WebFig if you prefer.

1. Create WireGuard interface

# Create WireGuard interface and it will automatically generate private and public keys
/interface wireguard add listen-port=13231 mtu=1420 name=cloud-wg
# Add the IP address to the interface
/ip add address=172.16.10.3/24 interface=cloud-wg network=172.16.10.0

# Print the keys, you will need only the public key later
/interface/wireguard/print

2. Create the peer

Here we add cloud vm as a peer to the Mikrotik router’s wireguard. The public key of the cloud server is needed here.

# In allowed-address we add IP ranges which will be routed through this peer
# 172.21.0.0/16 - subnet of docker containers on cloud server
# 172.16.10.0/16 - subnet of wireguard network
/interface wireguard peers add allowed-address=172.16.10.0/24,172.21.0.0/16 \
    endpoint-address=cloud.vm.ip.address \
    endpoint-port=27277 \
    interface=cloud-wg \
    public-key="PUT CLOUD SERVER PUBLIC KEY HERE" \
    name=peer1 persistent-keepalive=25s

# Here is a tricky part
# Routing rule for subnet of wireguard network is added automatically on adding IP address to the interface
# But we need to add explicitly the route for docker containers subnet
# You may need to adjust the distance and routing-table values
/ip route disabled=no distance=1 dst-address=172.21.0.0/16 gateway=cloud-wg routing-table=main scope=30 suppress-hw-offload=no target-scope=10

3. Add WireGuard peer (VM)

sudo nano /etc/wireguard/wg0.conf
# Uncomment the lines for Mikrotik peer
# Mikrotik peer
[Peer]
AllowedIPs = 172.16.10.0/24,192.168.0.0/16
PublicKey = mikrotik public key

# Save and exit

# Restart the WireGuard service
sudo systemctl restart wg-quick@wg0
# Check if changes are applied
wg show

You should see the Mikrotik peer in the list of peers.

# ping wireguard IP of cloud server
ping 172.16.10.1

# resolve google.com via adguard container on cloud vm
put [resolve google.com server 172.21.0.114]

# ping wireguard IP of Mikrotik router
ping 172.168.10.3

# resolve google.com via adguard container on home adguard
dig google.com @192.168.11.2

# ping wireguard IP of cloud server
ping 172.16.10.1

# resolve google.com via adguard container on cloud adguard
dig google.com @172.121.0.114

IV. Add first client device as a peer

I recommend you to use your smartphone as first client device, because it can work from both home WiFi and mobile data. This way you can test the connection from both networks.

Also, install on your smartphone:

• WireGuard app (iOS, Android)
• Network debugging app (iOS, Android)

1. Create keys for the client

Install Wireguard app for your client OS.

Then, generate public and private keys on the device, for that create config from scratch in the app and then click on Generate keypair button.

Or you can generate keys on the cloud server and then copy them to the client device.

wg genkey | tee peer-privatekey | wg pubkey > peer-publickey

2. Add the client as a peer to WireGuard config on cloud server

On the cloud server, edit the WireGuard config file and add the client as a peer.

Each time increment previous peer address by 1.

sudo nano /etc/wireguard/wg0.conf
# Uncomment the lines for client peer
# client peer
[Peer]
AllowedIPs = 172.16.10.2/32
PublicKey = put public key

# CRTL+X: save and exit

# Restart the WireGuard service
sudo systemctl restart wg-quick@wg0

# Check if changes are applied
wg show

3. Continue configuring the client device

You already have public and private keys for the client device, other configuration parameters are:

Interface

• Address - It’s address of peer in wireguard subnet. Put the same address you set in AllowedIPs field in previous step.
• DNS servers - if you have adguard/pihole running on the cloud server, you can use it as a DNS server. Put it’s IP address here.
• MTU and ListenPort - you can leave them empty, they will be set automatically.

Peer

• Endpoint - cloud server IP address and port (27277)
• Public key - public key of the cloud server
• AllowedIPs - here we put all subnets that we want to access from this current client device

Here is an example of the configuration file for the client device:

[Interface]
PrivateKey = private-key-of-client-device
Address = 172.16.10.2/32
DNS = 172.21.0.114 # IP of adguard container on cloud server

[Peer]
PublicKey = public-key-of-cloud-server
AllowedIPs = 192.168.0.0/16, 172.16.10.0/24, 172.21.0.0/16 # subnets of home, cloud and wireguard network
Endpoint = ip.of.cloud.servner:27277

That’s it! Disconnect your device from the home Wi-Fi, switch to mobile data and connect to the VPN.

Then try to:

• ping the cloud server and Mikrotik router IP addresses in wireguard subnet.
• check ports of services in docker containers on cloud and home server VM.

ping cloud	ping home	check adguard port

V. Bonus. DNS configuration

Since I have 2 adguard instances and I use them as DNS servers everywhere, I will add DNS records for accessing my services:

• *.cloud.domain.com - pointing to traefik docker container on cloud server
• *.home.domain.com - pointing to traefik docker container on home server

Thus, I can access my services using domain names instead of IP addresses.

VI. Final thoughts

I hope this tutorial was helpful for you. I will keep it updated if I find any issues or improvements.

If you have any questions or suggestions, feel free to leave a comment below.

Credits to @laroberto for their guide on LAN access with WireGuard. I followed it to set up the initial configuration and then adapted it to my needs.

When I started writing my first scripts I faced a problem: how to store and use secrets in my scripts. I have found a solution and I want to share it with you.

The problem

Let’s say I want to write a script that will send me telegram notifications. To do so I need to store my telegram bot token and chat id. Since I keep my RouterOS configuration in a git repository, I don’t want to hardcode my secrets in the script.

:global sendTelegramMessage do={
    :local botToken "1234567890:ABCDEFGHIJKLMN"
    :local chatId "10000000"
    :local message "$1"

    # telegram notification
    /tool fetch url="https://api.telegram.org/bot$botToken/sendMessage\?chat_id=$chatId&text=$message" keep-result=no
}

The solution

RouterOS has low level feature /ppp secret which can be used to store secrets. However, it could be inconvenient and a bit messy to use it directly in scripts. Instead, I would like to have more high level API to store and use secrets.

And, I have one. There is post in mikrotik forum by user with nickname Amm0. Ammo has shared a script of global function which can be used to store and retrieve secrets like this:

$SECRET set mySecret password="mySecretPassword"
:put [$SECRET get mySecret]

Now, I modify my script to use this function to keep it clean and secure:

:global sendTelegramMessage do={
    :local botToken
    :set botToken "$[$SECRET get TELEGRAM_TOKEN]"
    :local chatId "$[$SECRET get CHAT_ID]"
    :local message "$1"

    # telegram notification
    /tool fetch url="https://api.telegram.org/bot$botToken/sendMessage\?chat_id=$chatId&text=$message" keep-result=no
}

### $SECRET
#   get <name>
#   set <name> password=<password>
#   remove <name
#   print
:global SECRET
:set $SECRET do={
    :global SECRET

    # helpers
    :local fixprofile do={
        :if ([/ppp profile find name="null"]) do={:put "nothing"} else={
            /ppp profile add bridge-learning=no change-tcp-mss=no local-address=0.0.0.0 name="null" only-one=yes remote-address=0.0.0.0 session-timeout=1s use-compression=no use-encryption=no use-mpls=no use-upnp=no
        }
    }
    :local lppp [:len [/ppp secret find where name=$2]]
    :local checkexist do={
        :if (lppp=0) do={
            :error "\$SECRET: cannot find $2 in secret store"
        }
    }

    # $SECRET
    :if ([:typeof $1]!="str") do={
        :put "\$SECRET"
        :put "   uses /ppp/secrets to store stuff like REST apikeys, or other sensative data"
        :put "\t\$SECRET print - prints stored secret passwords"
        :put "\t\$SECRET get <name> - gets a stored secret"
        :put "\t\$SECRET set <name> password=\"YOUR_SECRET\" - sets a secret password"
        :put "\t\$SECRET remove <name> - removes a secret"
    }

    # $SECRET print
    :if ($1~"^pr") do={
        /ppp secret print where comment~"\\\$SECRET"
        :return [:nothing]
    }

    # $SECRET get
    :if ($1~"get") do={
        $checkexist
       :return [/ppp secret get $2 password]
    }

    # $SECRET set
    :if ($1~"set|add") do={
        :if ([:typeof $password]="str") do={} else={:error "\$SECRET: password= required"}
        :if (lppp=0) do={
            /ppp secret add name=$2 password=$password
        } else={
            /ppp secret set $2 password=$password
        }
        $fixprofile
        /ppp secret set $2 comment="used by \$SECRET"
        /ppp secret set $2 profile="null"
        /ppp secret set $2 service="async"
        :return [$SECRET get $2]
    }

    # $SECRET remove
    :if ($1~"rm|rem|del") do={
        $checkexist
        :return [/ppp secret remove $2]
    }
    :error "\$SECRET: bad command"
}

Conclusion

The good thing about this approach is that secrets storing and retrieving mechanism encapsulated and can be easily changed in the future without changing the scripts. Also, it is easy to use and understand.

Keep your secrets safe and happy scripting!

In this post I will show you how to configure hotkeys for that.

Hardware

You will need a monitor with multiple input sources. For example, I have Dell S2722QC tt has 2 HDMI ports and 1 USB-C port where:

• Macbook Air connected to port HDMI-2
• Mac Mini connected to port USB-C-1

Software

There is app called BetterDisplay that has a lot of powerful features. But for our case we need only one feature - change display inputs using DDC.

Install it on both Macs. You will have 14 days trial period with all PRO features.

Enable Accessibility for BetterDisplay in System Settings -> Privacy & Security -> Accessibility.

Then try to switch input source by clicking on BetterDisplay icon in the menu bar -> DDC Input Source -> Select next port.

If it works, you can continue to the next step. Otherwise check if your monitor supports DDC protocol and ensure Accessibility is enabled for BetterDisplay.

Paid option

If you are ready to pay 19$/19€ x2 for both Macs you can buy BetterDisplay. And then configure hotkeys in the app settings Settings -> Keyboards -> Custom keyboard shortcuts -> DDC Input Source.

Click “Record Shortcut” and press the key combination you want to use, for example CMD + F1 and CMD + F2.

Free option

If you like me don’t want to pay for 40$ for single feature there is a hacky way to do it.

We need an app that can handle hotkeys and run shell commands. I use Raycast, so called “Spotlight on steroids” and it can handle custom hotkeys. Or you can use any other app you like.

Configuring shell command

Before configuring Raycast we need to know ddc value for each input source. To do so, go to Settings -> Displays -> “Your monitor name” -> DDC Input Sources, and save IDs from Value column for each input source:

In my case it’s 18 for HDMI-2 and 25 for USB-C-1.

Then create a directory ~/raycast-scripts and put there a script change-input-source.sh:

#!/bin/bash

# See full documentation here: https://github.com/raycast/script-commands
#
# Required parameters:
# @raycast.schemaVersion 1
# @raycast.title Switch Monitor Input Source
# @raycast.mode silent
#
# Optional parameters:
# @raycast.icon 🖥️
# @raycast.packageName Raycast Scripts

DEST_ID=18 <PUT YOUR DDC ID HERE>
DEST_NAME="Home Mac Mini"

# PUT YOUR ACTUAL HOSTNAME
if [ `hostname` == "Aliks-Mac-mini.local" ]; then
  DEST_ID=25 <PUT YOUR SECOND DDC ID HERE>
  DEST_NAME="Work Macbook Air"
fi

echo "Switching monitor input source to $DEST_NAME"
/Applications/BetterDisplay.app/Contents/MacOS/BetterDisplay set -ddc=$DEST_ID -vcp=inputSelect

After that try to run it in terminal:

chmod +x ~/raycast-scripts/change-input-source.sh
~/raycast-scripts/change-input-source.sh

If it works, you can continue to the next step.

Configuring Raycast

1. Open Raycast and go to Settings -> Extensions -> Search for Scripts
2. Click on Add Script Directory and select ~/raycast-scripts
3. Click on Record Shortcut for newly added script and press the key combination you want to use, for example CMD + F1 on first Mac and CMD + F2 on second.

And that’s it! Now you can switch input source using hotkeys.

In this post, I will quickly descibe how you can build your own drone.io docker image.

Drone is very popular container native CI/CD platform. Not long time ago, there was release of new 1.0 version of drone. Which brang a lot of cool features and new license. The license tells that we can use Enterprise version of drone for free without any limits by building our own docker image if we are individuals or startup (read the licence for more detail).

So, how to build it?

Instructions

First, clone the drone repo to your local machine.

git clone git@github.com:drone/drone.git

Second, checkout to version of drone you want to build. For example, I want to build v1.3.1:

git checkout v1.3.1

We will use single dockerfile to build the image. To do so, we need to add extra step to existing dockerfile which is in docker directory. Let’s say we want to build docker image for linux OS and amd64 architecture, then we will edit docker/Dockerfile.server.linux.amd64.

If you check the dockerfile you will see, that binaries are just copied into docker image during the build and they are built outside of the docker build. So, the step we will add to dockerfile is go build step.

To build the binary, we need to know what version of go is used for building binary in original docker image. We can find it in drone.yml build step. For version 1.3.1 of drone golang:1.12.9 docker image is used for building binaries.

Then, we use same image to build binary in our dockerfile:

docker/Dockerfile.server.linux.amd64

FROM golang:1.12.9 as builder

WORKDIR /go/src/github.com/drone/drone
COPY . .

ENV GOOS linux
ENV GOARCH amd64
ENV CGO_ENABLED 1
ENV REPO github.com/drone/drone
ENV GO111MODULE on
RUN go build -tags nolimit -ldflags "-extldflags \"-static\"" -o release/linux/${GOARCH}/drone-server ${REPO}/cmd/drone-server

FROM alpine:3.9 as alpine
RUN apk add -U --no-cache ca-certificates

FROM alpine:3.9
EXPOSE 80 443
VOLUME /data

ENV GODEBUG netdns=go
ENV XDG_CACHE_HOME /data
ENV DRONE_DATABASE_DRIVER sqlite3
ENV DRONE_DATABASE_DATASOURCE /data/database.sqlite
ENV DRONE_RUNNER_OS=linux
ENV DRONE_RUNNER_ARCH=amd64
ENV DRONE_SERVER_PORT=:80
ENV DRONE_SERVER_HOST=localhost
ENV DRONE_DATADOG_ENABLED=true
ENV DRONE_DATADOG_ENDPOINT=https://stats.drone.ci/api/v1/series

COPY --from=alpine /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

COPY --from=builder /go/src/github.com/drone/drone/release/linux/amd64/drone-server /bin/
ENTRYPOINT ["/bin/drone-server"]

Also we need to delete .dockerignore file from root of the repo.

rm .dockerignore

Then we build docker image like:

docker build -t alikhil/drone:1.3.1 -f docker/Dockerfile.server.linux.amd64 .

That’s all! Now you can use own newly built docker image instead of official one if your use case meet license conditions.

Today I want you to share with you tutorial on how to deploy your SPA application to Kubernetes. Tutorial is oriented for those don’t very familiar with docker and k8s but want their single page application run in k8s.

Dockerize the application

I expect that you have docker installed in your machine. If it isn’t you can install it by following official installation guide.

As SPA project I will use vue-realworld-example-app as SPA project. You can your own SPA project if you have one.

So, I have cloned it, installed dependencies and built:

git clone https://github.com/gothinkster/vue-realworld-example-app
yarn
yarn build

Next step is to decide how our application will be served. There are bunch of possible solutions but I decided to use nginx since it recommends itself as one of the best http servers.

To serve SPA we need to return all requested files if they exist or otherwise fallback to index.html. To do so I wrote the following nginx config:

nginx.conf

# ...
# other configs

server {
  listen 80;
  root /app;

  location / {
    alias /app/;
    try_files $uri /index.html;
  }

}

Full config file can be found in my fork of the repo

Then, we need to write Dockerfile for building image with our application. Here it is:

FROM nginx
WORKDIR /root/
COPY ./dist /app
COPY ./nginx.conf /etc/nginx/conf.d/default.conf

We assume that artifacts of build placed in the dist directory and so that during the docker build the content of dist directory copied into containers /app directory.

Now, we are ready to build it:

docker build -t alikhil/my-spa:0.1 .

And run it:

docker run -p 8080:80 alikhil/my-spa:0.1

Then if we open http://localhost:8080 we will see something similar to:

Cool! It works!

We will need to use our newly builded docker image to deploy to k8s. So, we need to make it available from the k8s cluster by pulling to some docker registry. I will push image to DockerHub:

docker push alikhil/my-spa:0.1

Deploy to k8s

To run the application in k8s we will use Deployment resource type. Here it is:

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-spa
  labels:
    app: my-spa
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-spa
    spec:
      containers:
      - image: alikhil/my-spa:0.1
        name: spa
        ports:
          - containerPort: 80
        resources:
          limits:
            cpu: 150m
            memory: 250Mi

Then we create deployment by running kubectl apply -f deployment.yaml and newly created pods can found:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-spa-84b6dcd48d-mhv9f   1/1     Running   0          18s

Then we need to expose our app to the world. It can be done by using service of type NodePort or via Ingress. We will do it with Ingress. For that we will need service:

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-spa
  labels:
    app: my-spa
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: my-spa

And ingress itself:

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-spa-ing
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - my-spa.example.com
    secretName: my-spa-cert-secret
  rules:
  - host: my-spa.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-spa
          servicePort: 80

kubectl apply -f ingress.yaml -f service.yaml

And here it is! Our SPA runs in the k8s!

In this post, I will go through configuring Bitly OAuth2 proxy in a kubernetes cluster.

UPD 5/08/2025

There is a fresh tutorial about oauth2-proxy

A few days ago I was configuring SSO for our internal dev-services in KE Technologies.

And I spent the whole to make it work properly, and at the end I decided that I will share my experience by writing this post, hoping that it will help others(and possibly me in the future) to go through this process.

What do we want?

We have internal services in our k8s cluster that we want to be accessible for developers. It can be kubernetes-dashboard or kibana or anything else.

Before that we used Basic Auth, it’s easy to setup in ingresses. But this approach has several disadvantages:

1. We need to share a single pair of login and password for all services among all developers
2. Developers will be asked to enter credentials each time when they access service first time

What we want is that developer will log in once and will have access to all other services without additional authentication.

So, a possible scenario could be:

1. Developers open https://kibana.example.com which is internal service
2. Browser redirects them to https://auth.example.com where they sign in
3. After successful authentication browser redirects them to https://kibana.example.com

Preparation

Update

UPD 1.0(07/30/18)

Using kube-lego for configuring Let’s Encrypt certificates is depricated now. Consider using cert-manager instead.

UPD 2.0(08/24/18)

Initialy, when I was writing this post I was using old version of nginx 0.9.0, because it did not work correctly on newer version. Now, I found the problem and it have been fixed in 0.18.0 release. But ingress exposing private services should be updated(more details):

    nginx.ingress.kubernetes.io/auth-signin: https://auth.example.com/oauth2/start?rd=https://$host$request_uri$is_args$args

    nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.oauth-proxy.svc.cluster.local:4180/oauth2/auth

Kubernetes

First of all, we need a Kubernetes cluster. I will use the newly created cluster in Google Cloud Platform with version 1.8.10-gke.0. If you have a cluster with configured ingress and https you can skip this step.

Then we need to install nginx ingress and kube lego. Let’s do it using helm:

Init helm

With RBAC:

# giving default accout admin role:
ACCOUNT=$(gcloud info --format='value(config.account)')

kubectl create clusterrolebinding owner-cluster-admin-binding \
    --clusterrole cluster-admin \
    --user $ACCOUNT


kubectl -n kube-system create sa tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller

without RBAC:

helm init

Install nginx-ingress

helm install stable/nginx-ingress --name nginx-ing --namespace nginx-ing \
    --set controller.image.repository=gcr.io/google_containers/nginx-ingress-controller \
    --set controller.image.tag="0.9.0-beta.15"
    --set rbac.create=true # if RBAC is enabled in the cluster
    # see all options here: https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/values.yaml

After it’s installed we can retrieve controller IP address:

kubectl --namespace nginx-ing get services -o wide -w nginx-ing-nginx-ingress-controller

and create DNS record to point our domain and subdomains to this IP address.

A       example.com     xxx.xxx.xx.xxx
CNAME   *.example.com   example.com

Install kube-lego

helm install --name kube-lego stable/kube-lego --namespace kube-lego \
    --set config.LEGO_SUPPORTED_INGRESS_CLASS=nginx \
    --set config.LEGO_SUPPORTED_INGRESS_PROVIDER=nginx \
    --set config.LEGO_DEFAULT_INGRESS_CLASS=nginx \
    --set config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory \
    --set rbac.create=true \
    --set image.tag=0.1.5 \
    --set config.LEGO_LOG_LEVEL=debug

Test it!

Let’s run simple HTTP server as service and expose it using nginx ingress:

kubectl run simple-http --image=strm/helloworld-http --port=80

kubectl expose deployment simple-http --name example-service --port=80 --target-port=80 --type=NodePort

example-ing.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
spec:
  rules:
    - host: service.example.com
      http:
        paths:
          - backend:
              serviceName: example-service
              servicePort: 80
            path: /
  tls:
    - hosts:
        - "service.example.com"
      secretName: ing-tls

kubectl apply -f example-ing.yaml

Wait for a few seconds and open https://service.example.com and you should see something similar to this:

Example

GitHub application

In this post, we will use GitHub accounts for authentication.

So, go to https://github.com/settings/applications/new and create new OAuth application

Fill Authorization callback URL field with https://auth.example.com/oauth2/callback where example.com is your domain name.

GitHub Application

After creating an application you will have Client ID and Client Secret which we will need in next step.

Deploy OAuth Proxy

There are a lot of docker images for OAuth proxy, but we can not use them because they do not support domain white-listing. The problem is that such functionality has not implemented yet.

Actualy there are several PRs that solve that problem but seems to be they frozen for an unknown amount of time.

So, the only thing I could do is to merge one of the PRs to current master and build own image.

You also can use my image, but if you worry about security just clone my fork and build image yourself.

Let’s create a namespace and set it as current:

kubectl create ns oauth-proxy

kns oauth-proxy # I am using kubectx tool -> https://github.com/ahmetb/kubectx

Deploy secret

secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: oauth-proxy-secret
  namespace: oauth-proxy
data:
  github-client-id: base64(YOUR_CLIENT_ID)
  github-client-secret: base64(YOUR_CLIENT_SECRET)
  cookie-secret: base64(random_string)

kubectl create -f secret.yml

Deploy deployment

oauth-proxy.deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: oauth-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: alikhil/oauth2_proxy:2.2.2
        imagePullPolicy: Always
        args:
        - --provider=github
        - --email-domain=*
        - --upstream=file:///dev/null
        - --http-address=0.0.0.0:4180
        - --whitelist-domain=.example.com
        - --cookie-domain=.example.com
        # - --cookie-expire duration: expire timeframe for cookie (default 168h0m0s)
        # - --cookie-name string: the name of the cookie that the oauth_proxy creates (default "_oauth2_proxy")
        # - --cookie-refresh duration: refresh the cookie after this duration; 0 to disable
        # - --cookie-secret string: the seed string for secure cookies (optionally base64 encoded)
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: github-client-id
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: github-client-secret
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: cookie-secret
        ports:
        - containerPort: 4180
          protocol: TCP

kubectl create -f oauth-proxy.deployment.yml

Deploy service

oauth-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: oauth-proxy
spec:
  type: NodePort
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: oauth2-proxy

kubectl create -f oauth-service.yml

Deploy ingress

oauth-ing.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: oauth-proxy
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"

spec:
  rules:
  - host: auth.example.com
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 4180
        path: /oauth2
  tls:
  - hosts:
    - auth.example.com
    secretName: oauth-proxy-tls

kubectl create -f oauth-ing.yml

Test it!

You can update ingress that we used while configuring nginx-ingress or create a new one:

example-ing.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    ingress.kubernetes.io/auth-url: https://auth.example.com/oauth2/auth
    ingress.kubernetes.io/auth-signin: https://auth.example.com/oauth2/start?rd=https://$host$request_uri$is_args$args
spec:
  rules:
    - host: service.example.com
      http:
        paths:
          - backend:
              serviceName: example-service
              servicePort: 80
            path: /
  tls:
    - hosts:
        - service.example.com
      secretName: ing-tls

kubectl apply -f example-ing.yml

Then visit service.example.com and you will be redirected to GitHub authorization page:

GitHub Authorization page

And once you authenticate, you will have access to all your services under ingress that point to auth.example.com until cookie expires.

And that’s it! Now you can put any of your internal services behind ingress with OAuth.

Resources

Here is a list of resources that helped me to go through this proccess first time:

• https://eng.fromatob.com/post/2017/02/lets-encrypt-oauth-2-and-kubernetes-ingress/
• https://www.midnightfreddie.com/oauth2-proxy.html
• https://thenewstack.io/single-sign-on-for-kubernetes-dashboard-experience/

In this tutorial, we will configure Go environment in VS Code and write our first program in Go.

Install Go

The first thing that you need to do it’s to install Go on your computer. To do so, download installer for your operating system from here and then run the installer.

Configure GOPATH

By language convention, Go developers store all their code in a single place called workspace. Go also puts dependency packages in the workspace. So, in order to Go perform correctly, we need to set GOPATH variable with the path to the workspace.

MacOS and Linux

Set the GOPATH envar with workspace

export GOPATH=$HOME/go

Also, we need to add GOPATH/bin to PATH in order to run compiler Go programs:

export PATH=$PATH:$GOPATH/bin

Configure VS Code

Install official Go extension.

Install delve debugger:

go get -u github.com/derekparker/delve/cmd/dlv

I recommend you to add the following lines to your VS Code user settings:

settings.json

{
    "go.autocompleteUnimportedPackages": true,
    "go.formatTool": "gofmt"
}

Windows

Create GOPATH envar:

set GOPATH=c:\Users\%USERNAME%\go

Also, we need to add GOPATH\bin to PATH in order to run compiler Go programs:

set PATH=%PATH%;%GOPATH%\bin

Create project

Move to your GOPATH/src directory. Create a directory for your project:

cd $GOPATH/src
mkdir -p github.com/alikhil/hello-world-with-go

Open it using vscode:

code github.com/alikhil/hello-world-with-go

Hello World

Let’s create a file named program.go and put the following code there:

program.go

package main

import "fmt"

func main() {
    fmt.Println("¡Hola, mundo!")
}

Run the program

Finally, to run the program by pressing the F5 button in VS Code and you should see the message printed to Debug Console.

That’s all! My congratulations, you have just written your first program in Go!

Troubleshooting

If you fail to run your program and there is some message like “Cannot find a path to go”. Try to add to your PATH envar with path directory where go binary is stored.

For example in MacOS I have added following line to my ~/.bash_profile:

export PATH=/usr/local/go/bin:$PATH