Kubernetes on alikhil

Kubernetes In-Place Pod Resize

Alik Khilazhev — Sun, 28 Dec 2025 21:14:12 +0300

About six years ago, while operating a large Java-based platform in Kubernetes, I noticed a recurring problem: our services required significantly higher CPU and memory during application startup. Heavy use of Spring Beans and AutoConfiguration forced us to set inflated resource requests and limits just to survive bootstrap, even though those resources were mostly unused afterwards.

This workaround never felt right. As an engineer, I wanted a solution that reflected the actual lifecycle of an application rather than its worst moment.

I opened an issue in the Kubernetes repository describing the problem and proposing an approach to adjust pod resources dynamically without restarts. The issue received little discussion but quietly accumulated interest over time (13 👍 emoji reaction). Every few months, an automation bot attempted to mark it as stale, and every time, I removed the label. This went on for nearly six years…

Until the release of Kubernetes 1.35 where In-Place Pod Resize feature was marked as stable.

What In-Place Pod Resize Brings

In-Place Pod Resize allows Kubernetes to update CPU and memory requests and limits without restarting pods, whenever it is safe to do so. This significantly reduces unnecessary restarts caused by resource changes, leading to fewer disruptions and more reliable workloads.

For applications whose resource needs evolve over time, especially after startup, this feature provides a long-missing building block.

Impact on VerticalPodAutoscaler

The new resizePolicy field is configured at the pod spec level. While it is technically possible to change pod resources manually, doing so does not scale. In practice, this feature should be driven by a workload controller.

At the moment, the only controller that supports in-place pod resize is the Vertical Pod Autoscaler (VPA).

There are two enhancement proposals enable this behavior:

AEP-4016: Support for in place updates in VPA which introduces InPlaceOrRecreate update mode
AEP-7862: CPU Startup Boost which is about temporarily boosting pod by giving more cpu during pod startup. This is conceptually similar to the approach proposed in my original issue.

Here is an example of Deployment and VPA using both AEP features:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
        - name: app
          image: my-heavy-java-app:stable
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 1000m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 2048Mi

---

apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: example-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: example
  updatePolicy:
    updateMode: "InPlaceOrRecreate"
  resourcePolicy:
    containerPolicies:
      - containerName: "app"
        minAllowed:
          cpu: "250m"
          memory: "512Mi"
        maxAllowed:
          cpu: "3000m"
          memory: "8192Mi"
        # The CPU boosted resources can go beyond maxAllowed.
        startupBoost:
          cpu:
            type: "Factor"
            quantity: "2"

With such configuration pod will have doubled cpu requests and limits during startup. During the boost period no resizing will happen.

Once the pod reaches the Ready state, the VPA controller scales CPU down to the currently recommended value.

After that, VPA continues operating normally, with the key difference that resource updates are applied in place whenever possible.

Limitations

Does this feature fully solve the problem described above? Only partially.

First, most application runtimes still impose fundamental constraints. Java and Python runtimes do not currently support resizing memory limits without a restart. This limitation exists outside of Kubernetes itself and is tracked in the OpenJDK project via an open ticket.

In-Place Pod Resize in Kubernetes: Dynamic Resource Management Without Restarts - Tim Allclair & Mofi Rahman, Google

Second, Kubernetes does not yet support decreasing memory limits, even with in-place Pod Resize enabled. This is a known limitation documented in the enhancement proposal for memory limit decreases.

As a result, while in-place Pod Resize effectively addresses CPU-related startup spikes, memory resizing remains an open problem.

Final thoughts

In place Pod Resize gives a foundation for cool new features like StartupBoost and makes use of VPA more reliable. While important gaps remain, such as memory decrease support and scheduling race condition, this change represents a meaningful step forward.

For workloads with distinct startup and steady-state phases, Kubernetes is finally beginning to model reality more closely.

kubectl-find - UNIX-find-like plugin to find resources and perform action on them

Alik Khilazhev — Fri, 17 Oct 2025 19:27:24 +0300

Recently, I have developed a plugin for kubectl inspired by UNIX find utility to find and perform action on resources. And few days ago number of stars in the repo reached 50! I think it’s a good moment to tell more about the project.

The problem

As engineer who works with kubernetes everyday I use kubectl a lot. Actually, more than 50% of my terminal history commands are related to kubernetes. Here is a top 10 commands:

1	2079  46.7611%    kubectl
2	425   9.55915%    git
3	324   7.28745%    helm
4	156   3.50877%    cd
5	146   3.28385%    ssh
6	130   2.92398%    kctx
7	114   2.5641%     kns
8	80    1.79937%    gcloud-auth
9	78    1.75439%    curl
10	66    1.48448%    docker
...

Run this command if you are curious what about yours the most popular commands in terminal history.

I use kubectl to check status of the pods, delete orphaned resources, trigger sync on ExternalSecrets and much more. When I realized half my terminal history was just kubectl commands, I thought — there must be a better way to find things in Kubernetes without chaining pipes with grep / awk / xargs. And I imagined how nice it would be to have a UNIX find-like tool — something that lets you search for exactly what you need in the cluster and then perform actions directly on the matching resources. I searched for a krew plugin like this but there was not any. For that reason, I decided to develop one!

Development

I used sample-cli-plugin as a starting point. Its clean repository structure and straightforward design make it a great reference for working with the Kubernetes API. Additionally, it allows easy reuse of the extensive Kubernetes client libraries.

Almost everything in the Kubernetes ecosystem is written in Go, and this plugin is no exception — which is great, as it allows building binaries for a wide range of CPU architectures and operating systems.

Features overview

Filters

Find by resource name using regex

kubectl fd pods -r dev

Find pods with restarted containers

kubectl fd pods --restarted

Find pods with image matching regex

kubectl fd pods --image bitnami

Find pods by status

kubectl fd pods --status Pending

Find pods running on nodes matching regex

kubectl fd pods --node '*spot'

Find any resource by custom condition

Use jq filter to find any resource by any custom condition. kubectl-find uses gojq implementation of jq.

# find Services with trafficDistribution set to PreferClose
kubectl fd svc -j '.spec.trafficDistribution == "PreferClose"'

# find pods with unset nodeSelector
kubectl fd pods -j '.spec.nodeSelector == null' -A

# find pods with undefined resources
kubectl fd pods -j 'any( .spec.containers[]; .resources == {} )' -A

Actions

By default, fd will print found resources to Stdout. However, there flags that you can provide to perform action on found resources:

--delete - to delete them
--patch - to patch with provided JSON
--exec - to run command on pods

Getting started

Use krew to install the plugin:

krew install fd

What’s next

I’m currently working on adding:

JSON/YAML output format
More filters
Saved queries

If you’re tired of writing long kubectl | grep | xargs chains, give kubectl fd a try — it’s already saved me countless keystrokes.

Check out the repo ⭐ github.com/alikhil/kubectl-find and share your ideas or issues — I’d love to hear how you use it!

OAuth2-proxy: protect services in kubernetes

Alik Khilazhev — Tue, 05 Aug 2025 21:14:12 +0300

The original post wrote about oauth2-proxy over seven years ago was quite popular at the time and attracted a lot of organic traffic to my blog, which still benefits my SEO today. Since the tutorial had become outdated, I decided to rewrite it.

What we have

We have a Kubernetes cluster with several web services deployed for internal use.

What we want to achieve

We want to expose our internal web services to the Internet, but restrict access by requiring authorization. Access should be granted only to users authenticated through our Identity Provider (such as Google, GitHub, Keycloak, etc.).

Assumptions

For simplicity, let’s assume that both ingress-nginx and cert-manager are already deployed in the cluster.

I will use Pocket ID as Identity Provider in this tutorial. Configuration slightly differs for different providers. Check the official documentation for your provider.

For the examples in this guide, I’ll use my alikhil.dev domain:

– pocket-id.k8s.alikhil.dev - will be used for Pocket ID

k8s.alikhil.dev - will be used for oauth2-proxy. I recommend to have higher domain for oauth2-proxy service for easier cookie setup.
*.k8s.alikhil.dev - reserved for services deployed for internal usage

Preparation

DNS

I have added two DNS records:

A record for k8s.alikhil.dev pointing to ingress-nginx LoadBalancer IP address in the cluster (kubectl get svc -n ingress-nginx | grep LoadBalancer)
CNAME record for *.k8s.alikhil.dev pointing to k8s.alikhil.dev

Install Pocket ID


helm repo add anza-labs https://anza-labs.github.io/charts
helm upgrade --install pocket-id anza-labs/pocket-id -f ./values/pocket-id.yaml

Values for pocket-id

Why Graceful Shutdown Matters in Kubernetes

Alik Khilazhev — Sun, 01 Jun 2025 20:31:06 +0300

Have you ever deployed a new version of your app in Kubernetes and noticed errors briefly spiking during rollout? Many teams do not even realize this is happening, especially if they are not closely monitoring their error rates during deployments.

There is a common misconception in the Kubernetes world that bothers me. The official Kubernetes documentation and most guides claim that “if you want zero downtime upgrades, just use rolling update mode on deployments”. I have learned the hard way that this simply it is not true - rolling updates alone are NOT enough for true zero-downtime deployments.

And it is not just about deployments. Your pods can be terminated for many other reasons: scaling events, node maintenance, preemption, resource constraints, and more. Without proper graceful shutdown handling, any of these events can lead to dropped requests and frustrated users.

In this post, I will share what I have learned about implementing proper graceful shutdown in Kubernetes. I will show you exactly what happens behind the scenes, provide working code examples, and back everything with real test results that clearly demonstrate the difference.

The Problem: Hidden Errors During Pod Termination

ChatGPT: draw funny picture of Kubernetes pod gracefully shutting down

If you are running services on Kubernetes, you have probably noticed that even with rolling updates (where Kubernetes gradually replaces pods), you might still see errors during deployment. This is especially annoying when you are trying to maintain “zero-downtime” systems.

When Kubernetes needs to terminate a pod (for any reason), it follows this sequence:

Sends a SIGTERM signal to your container
Waits for a grace period (30 seconds by default)
If the container does not exit after the grace period, it gets brutal and sends a SIGKILL signal

The problem? Most applications do not properly handle that SIGTERM signal. They just die immediately, dropping any in-flight requests. In the real world, while most API requests complete in 100-300ms, there are often those long-running operations that take 5-15 seconds or more. Think about processing uploads, generating reports, or running complex database queries. When these longer operations get cut off, that’s when users really feel the pain.

When Does Kubernetes Terminate Pods?

Rolling updates are just one scenario where your pods might be terminated. Here are other common situations that can lead to pod terminations:

Horizontal Pod Autoscaler Events: When HPA scales down during low-traffic periods, some pods get terminated.
Resource Pressure: If your nodes are under resource pressure, the Kubernetes scheduler might decide to evict certain pods.
Node Maintenance: During cluster upgrades, node draining causes many pods to be evicted.
Spot/Preemptible Instances: If you are using cost-saving node types like spot instances, these can be reclaimed with minimal notice.

All these scenarios follow the same termination process, so implementing proper graceful shutdown handling protects you from errors in all of these cases - not just during upgrades.

Let’s Test It: Basic vs. Graceful Service

Instead of just talking about theory, I built a small lab to demonstrate the difference between proper and improper shutdown handling. I created two nearly identical Go services:

Basic Service: A standard HTTP server with no special shutdown handling
Graceful Service: The same service but with proper SIGTERM handling

Both services:

Process requests that take about 4 seconds to complete (intentionally configured for easier demonstration)
Run in the same Kubernetes cluster with identical configurations
Serve the same endpoints

I specifically chose a 4-second processing time to make the problem obvious. While this might seem long compared to typical 100-300ms API calls, it perfectly simulates those problematic long-running operations that occur in real-world applications. The only difference between the services is how they respond to termination signals.

To test them, I wrote a simple k6 script that hammers both services with requests while triggering rolling restart of service’s deployment. Here is what happened:

Basic Service: The Error-Prone One

checks_total.......................: 695    11.450339/s
checks_succeeded...................: 97.98% 681 out of 695
checks_failed......................: 2.01%  14 out of 695

✗ status is 200
  ↳  97% — ✓ 681 / ✗ 14

http_req_failed....................: 2.01%  14 out of 696

Graceful Service: The Reliable One

checks_total.......................: 750     11.724824/s
checks_succeeded...................: 100.00% 750 out of 750
checks_failed......................: 0.00%   0 out of 750

✓ status is 200

http_req_failed........... ........: 0.00%  0 out of 751

The results speak for themselves. The basic service dropped 14 requests during the update (that is 2% of all traffic), while the graceful service handled everything perfectly without a single error.

You might think “2% it is not that bad” — but if you are doing several deployments per day and have thousands of users, that adds up to a lot of errors. Plus, in my experience, these errors tend to happen at the worst possible times.

So How Do We Fix It? The Graceful Shutdown Recipe

After digging into this problem and testing different solutions, I have put together a simple recipe for proper graceful shutdown. While my examples are in Go, the fundamental principles apply to any language or framework you are using.

Here are the key ingredients:

1. Listen for SIGTERM Signals

First, your app needs to catch that SIGTERM signal instead of ignoring it:

// Set up channel for shutdown signals
stop := make(chan os.Signal, 1)
signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

// Block until we receive a shutdown signal
<-stop
log.Println("Shutdown signal received")

This part is easy - you are just telling your app to wake up when Kubernetes asks it to shut down.

2. Track Your In-Flight Requests

You need to know when it is safe to shut down, so keep track of ongoing requests:

// Create a request counter
var inFlightRequests atomic.Int64

http.HandleFunc("/process", func(w http.ResponseWriter, r *http.Request) {
    // Increment counter when request starts
    inFlightRequests.Add(1)
    // do not forget to decrement when done!
    defer inFlightRequests.Add(-1)

    // Your normal request handling...
    time.Sleep(4 * time.Second)  // Simulating long-running work
})

This counter lets you check if there are still requests being processed before shutting down. it is especially important for those long-running operations that users have already waited several seconds for - the last thing they want is to see an error right before completion!

3. Separate Your Health Checks

Here is a commonly overlooked trick - you need different health check endpoints for liveness and readiness:

// Track shutdown state
var isShuttingDown atomic.Bool

// Readiness probe - returns 503 when shutting down
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
    if isShuttingDown.Load() {
        w.WriteHeader(http.StatusServiceUnavailable)
        fmt.Fprintf(w, "Shutting down, not ready")
        return
    }

    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "Ready for traffic")
})

// Liveness probe - always returns 200 (we are still alive!)
http.HandleFunc("/alive", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, "I'm alive")
})

This separation is crucial. The readiness probe tells Kubernetes to stop sending new traffic, while the liveness probe says “do not kill me yet, I’m still working!”

4. The Shutdown Dance

Now for the most important part - the shutdown sequence:

// Step 1: Mark service as shutting down
isShuttingDown.Store(true)

// Step 2: Let Kubernetes notice the readiness probe failing
time.Sleep(5 * time.Second)

// Step 3: Wait for in-flight requests to finish
for inFlightRequests.Load() > 0 {
    time.Sleep(1 * time.Second)
}

// Step 4: Finally, shut down the server gracefully
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

if err := server.Shutdown(ctx); err != nil {
    log.Fatalf("Forced shutdown: %v", err)
}

I’ve found this sequence to be optimal. First, we mark ourselves as “not ready” but keep running. We pause to give Kubernetes time to notice and update its routing. Then we patiently wait until all in-flight requests finish before actually shutting down the server.

5. Configure Kubernetes Correctly

Do not forget to adjust your Kubernetes configuration:

# Use different probes for liveness and readiness
livenessProbe:
  httpGet:
    path: /alive  # Always returns OK
    port: 8080
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready  # Returns 503 during shutdown
    port: 8080
  periodSeconds: 3
  failureThreshold: 2

# Give pods enough time to shut down gracefully
terminationGracePeriodSeconds: 30 # Stops routing traffic after 2 failed checks (6 seconds)

This tells Kubernetes to wait up to 30 seconds for your app to finish processing requests before forcefully terminating it.

TL; DR; Quick Tips

If you are in a hurry, here are the key takeaways:

Catch SIGTERM Signals: Do not let your app be surprised when Kubernetes wants it to shut down.
Track In-Flight Requests: Know when it is safe to exit by counting active requests.
Split Your Health Checks: Use separate endpoints for liveness (am I running?) and readiness (can I take traffic?).
Fail Readiness First: As soon as shutdown begins, start returning “not ready” on your readiness endpoint.
Wait for Requests: Do not just shut down - wait for all active requests to complete first.
Use Built-In Shutdown: Most modern web frameworks have graceful shutdown options; use them!
Configure Terminaton Grace Period: Give your pods enough time to complete the shutdown sequence.
Test Under Load: You will not catch these issues in simple tests - you need realistic traffic patterns.

Wrap Up: Is It Worth the Extra Code?

You might be wondering if adding all this extra code is really worth it. After all, we’re only talking about a 2% error rate during pod termination events.

From my experience working with high-traffic services, I would say absolutely yes - for three reasons:

User Experience: Even small error rates look bad to users. Nobody wants to see “Something went wrong” messages, especially after waiting 10+ seconds for a long-running operation to complete.
Cascading Failures: Those errors can cascade through your system, especially if services depend on each other. Long-running requests often touch multiple critical systems.
Deployment Confidence: With proper graceful shutdown, you can deploy more frequently without worrying about causing problems.

The good news is that once you have implemented this pattern once, it is easy to reuse across your services. You can even create a small library or template for your organization.

In production environments where I have implemented these patterns, we have gone from seeing a spike of errors with every deployment to deploying multiple times per day with zero impact on users. that is a win in my book!

Deploy SPA application to Kubernetes

Alik Khilazhev — Sat, 08 Jun 2019 22:30:00 +0300

Hello, folks!

Today I want you to share with you tutorial on how to deploy your SPA application to Kubernetes. Tutorial is oriented for those don’t very familiar with docker and k8s but want their single page application run in k8s.

Dockerize the application

I expect that you have docker installed in your machine. If it isn’t you can install it by following official installation guide.

As SPA project I will use vue-realworld-example-app as SPA project. You can your own SPA project if you have one.

So, I have cloned it, installed dependencies and built:

git clone https://github.com/gothinkster/vue-realworld-example-app
yarn
yarn build

Next step is to decide how our application will be served. There are bunch of possible solutions but I decided to use nginx since it recommends itself as one of the best http servers.

To serve SPA we need to return all requested files if they exist or otherwise fallback to index.html. To do so I wrote the following nginx config:

nginx.conf

# ...
# other configs

server {
  listen 80;
  root /app;

  location / {
    alias /app/;
    try_files $uri /index.html;
  }

}

Full config file can be found in my fork of the repo

Then, we need to write Dockerfile for building image with our application. Here it is:

FROM nginx
WORKDIR /root/
COPY ./dist /app
COPY ./nginx.conf /etc/nginx/conf.d/default.conf

We assume that artifacts of build placed in the dist directory and so that during the docker build the content of dist directory copied into containers /app directory.

Now, we are ready to build it:

docker build -t alikhil/my-spa:0.1 .

And run it:

docker run -p 8080:80 alikhil/my-spa:0.1

Then if we open http://localhost:8080 we will see something similar to:

Cool! It works!

We will need to use our newly builded docker image to deploy to k8s. So, we need to make it available from the k8s cluster by pulling to some docker registry. I will push image to DockerHub:

docker push alikhil/my-spa:0.1

Deploy to k8s

To run the application in k8s we will use Deployment resource type. Here it is:

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-spa
  labels:
    app: my-spa
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-spa
    spec:
      containers:
      - image: alikhil/my-spa:0.1
        name: spa
        ports:
          - containerPort: 80
        resources:
          limits:
            cpu: 150m
            memory: 250Mi

Then we create deployment by running kubectl apply -f deployment.yaml and newly created pods can found:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-spa-84b6dcd48d-mhv9f   1/1     Running   0          18s

Then we need to expose our app to the world. It can be done by using service of type NodePort or via Ingress. We will do it with Ingress. For that we will need service:

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-spa
  labels:
    app: my-spa
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: my-spa

And ingress itself:

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-spa-ing
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - my-spa.example.com
    secretName: my-spa-cert-secret
  rules:
  - host: my-spa.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-spa
          servicePort: 80

kubectl apply -f ingress.yaml -f service.yaml

And here it is! Our SPA runs in the k8s!

Oauth2 Proxy for Kubernetes Services

Alik Khilazhev — Sun, 20 May 2018 21:30:36 +0300

Hello, folks!

In this post, I will go through configuring Bitly OAuth2 proxy in a kubernetes cluster.

UPD 5/08/2025

There is a fresh tutorial about oauth2-proxy

A few days ago I was configuring SSO for our internal dev-services in KE Technologies.

And I spent the whole to make it work properly, and at the end I decided that I will share my experience by writing this post, hoping that it will help others(and possibly me in the future) to go through this process.

What do we want?

We have internal services in our k8s cluster that we want to be accessible for developers. It can be kubernetes-dashboard or kibana or anything else.

Before that we used Basic Auth, it’s easy to setup in ingresses. But this approach has several disadvantages:

We need to share a single pair of login and password for all services among all developers
Developers will be asked to enter credentials each time when they access service first time

What we want is that developer will log in once and will have access to all other services without additional authentication.

So, a possible scenario could be:

Developers open https://kibana.example.com which is internal service
Browser redirects them to https://auth.example.com where they sign in
After successful authentication browser redirects them to https://kibana.example.com

Preparation

Update

UPD 1.0(07/30/18)

Using kube-lego for configuring Let’s Encrypt certificates is depricated now. Consider using cert-manager instead.

UPD 2.0(08/24/18)

Initialy, when I was writing this post I was using old version of nginx 0.9.0, because it did not work correctly on newer version. Now, I found the problem and it have been fixed in 0.18.0 release. But ingress exposing private services should be updated(more details):

    nginx.ingress.kubernetes.io/auth-signin: https://auth.example.com/oauth2/start?rd=https://$host$request_uri$is_args$args

    nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.oauth-proxy.svc.cluster.local:4180/oauth2/auth

Kubernetes

First of all, we need a Kubernetes cluster. I will use the newly created cluster in Google Cloud Platform with version 1.8.10-gke.0. If you have a cluster with configured ingress and https you can skip this step.

Then we need to install nginx ingress and kube lego. Let’s do it using helm:

Init helm

With RBAC:

# giving default accout admin role:
ACCOUNT=$(gcloud info --format='value(config.account)')

kubectl create clusterrolebinding owner-cluster-admin-binding \
    --clusterrole cluster-admin \
    --user $ACCOUNT


kubectl -n kube-system create sa tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller

without RBAC:

helm init

Install nginx-ingress


helm install stable/nginx-ingress --name nginx-ing --namespace nginx-ing \
    --set controller.image.repository=gcr.io/google_containers/nginx-ingress-controller \
    --set controller.image.tag="0.9.0-beta.15"
    --set rbac.create=true # if RBAC is enabled in the cluster
    # see all options here: https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/values.yaml

After it’s installed we can retrieve controller IP address:

kubectl --namespace nginx-ing get services -o wide -w nginx-ing-nginx-ingress-controller

and create DNS record to point our domain and subdomains to this IP address.

A       example.com     xxx.xxx.xx.xxx
CNAME   *.example.com   example.com

Install kube-lego

helm install --name kube-lego stable/kube-lego --namespace kube-lego \
    --set config.LEGO_SUPPORTED_INGRESS_CLASS=nginx \
    --set config.LEGO_SUPPORTED_INGRESS_PROVIDER=nginx \
    --set config.LEGO_DEFAULT_INGRESS_CLASS=nginx \
    --set config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory \
    --set rbac.create=true \
    --set image.tag=0.1.5 \
    --set config.LEGO_LOG_LEVEL=debug

Test it!

Let’s run simple HTTP server as service and expose it using nginx ingress:

kubectl run simple-http --image=strm/helloworld-http --port=80

kubectl expose deployment simple-http --name example-service --port=80 --target-port=80 --type=NodePort

example-ing.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
spec:
  rules:
    - host: service.example.com
      http:
        paths:
          - backend:
              serviceName: example-service
              servicePort: 80
            path: /
  tls:
    - hosts:
        - "service.example.com"
      secretName: ing-tls

kubectl apply -f example-ing.yaml

Wait for a few seconds and open https://service.example.com and you should see something similar to this:

Example

GitHub application

In this post, we will use GitHub accounts for authentication.

So, go to https://github.com/settings/applications/new and create new OAuth application

Fill Authorization callback URL field with https://auth.example.com/oauth2/callback where example.com is your domain name.

GitHub Application

After creating an application you will have Client ID and Client Secret which we will need in next step.

Deploy OAuth Proxy

There are a lot of docker images for OAuth proxy, but we can not use them because they do not support domain white-listing. The problem is that such functionality has not implemented yet.

Actualy there are several PRs that solve that problem but seems to be they frozen for an unknown amount of time.

So, the only thing I could do is to merge one of the PRs to current master and build own image.

You also can use my image, but if you worry about security just clone my fork and build image yourself.

Let’s create a namespace and set it as current:

kubectl create ns oauth-proxy

kns oauth-proxy # I am using kubectx tool -> https://github.com/ahmetb/kubectx

Deploy secret

secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: oauth-proxy-secret
  namespace: oauth-proxy
data:
  github-client-id: base64(YOUR_CLIENT_ID)
  github-client-secret: base64(YOUR_CLIENT_SECRET)
  cookie-secret: base64(random_string)

kubectl create -f secret.yml

Deploy deployment

oauth-proxy.deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: oauth-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: alikhil/oauth2_proxy:2.2.2
        imagePullPolicy: Always
        args:
        - --provider=github
        - --email-domain=*
        - --upstream=file:///dev/null
        - --http-address=0.0.0.0:4180
        - --whitelist-domain=.example.com
        - --cookie-domain=.example.com
        # - --cookie-expire duration: expire timeframe for cookie (default 168h0m0s)
        # - --cookie-name string: the name of the cookie that the oauth_proxy creates (default "_oauth2_proxy")
        # - --cookie-refresh duration: refresh the cookie after this duration; 0 to disable
        # - --cookie-secret string: the seed string for secure cookies (optionally base64 encoded)
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: github-client-id
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: github-client-secret
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-proxy-secret
              key: cookie-secret
        ports:
        - containerPort: 4180
          protocol: TCP

kubectl create -f oauth-proxy.deployment.yml

Deploy service

oauth-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: oauth-proxy
spec:
  type: NodePort
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: oauth2-proxy

kubectl create -f oauth-service.yml

Deploy ingress

oauth-ing.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: oauth-proxy
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"

spec:
  rules:
  - host: auth.example.com
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 4180
        path: /oauth2
  tls:
  - hosts:
    - auth.example.com
    secretName: oauth-proxy-tls

kubectl create -f oauth-ing.yml

Test it!

You can update ingress that we used while configuring nginx-ingress or create a new one:

example-ing.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    ingress.kubernetes.io/auth-url: https://auth.example.com/oauth2/auth
    ingress.kubernetes.io/auth-signin: https://auth.example.com/oauth2/start?rd=https://$host$request_uri$is_args$args
spec:
  rules:
    - host: service.example.com
      http:
        paths:
          - backend:
              serviceName: example-service
              servicePort: 80
            path: /
  tls:
    - hosts:
        - service.example.com
      secretName: ing-tls

kubectl apply -f example-ing.yml

Then visit service.example.com and you will be redirected to GitHub authorization page:

GitHub Authorization page

And once you authenticate, you will have access to all your services under ingress that point to auth.example.com until cookie expires.

And that’s it! Now you can put any of your internal services behind ingress with OAuth.

Resources

Here is a list of resources that helped me to go through this proccess first time:

Kubernetes on alikhil

Kubernetes In-Place Pod Resize

What In-Place Pod Resize Brings

Impact on VerticalPodAutoscaler

Limitations

Final thoughts

kubectl-find - UNIX-find-like plugin to find resources and perform action on them

The problem

Development

Features overview

Filters

Find by resource name using regex

Find pods with restarted containers

Find pods with image matching regex

Find pods by status

Find pods running on nodes matching regex

Find any resource by custom condition

Actions

Getting started

What’s next

OAuth2-proxy: protect services in kubernetes

What we have

What we want to achieve

Assumptions

Preparation

DNS

Install Pocket ID

Why Graceful Shutdown Matters in Kubernetes

The Problem: Hidden Errors During Pod Termination

When Does Kubernetes Terminate Pods?

Let’s Test It: Basic vs. Graceful Service

Basic Service: The Error-Prone One

Graceful Service: The Reliable One

So How Do We Fix It? The Graceful Shutdown Recipe

1. Listen for SIGTERM Signals

2. Track Your In-Flight Requests

3. Separate Your Health Checks

4. The Shutdown Dance

5. Configure Kubernetes Correctly

TL; DR; Quick Tips

Wrap Up: Is It Worth the Extra Code?

Further Reading

Deploy SPA application to Kubernetes

Dockerize the application

Deploy to k8s

Oauth2 Proxy for Kubernetes Services

UPD 5/08/2025

What do we want?

Preparation

Update

UPD 1.0(07/30/18)

UPD 2.0(08/24/18)

Kubernetes

Init helm

Install nginx-ingress

Install kube-lego

Test it!

GitHub application

Deploy OAuth Proxy

Deploy secret

Deploy deployment

Deploy service

Deploy ingress

Test it!

Resources