Resolving 5xx Errors When Deploying on Kubernetes

When deploying on Kubernetes with a rolling deployment, 502 or 503 errors can occur along the way.

There are issues about this problem, as well as pod-graceful-drain, which was created to solve it.

Of course, I handle it by using sleep.

To explain, when a new pod is deployed, the existing pod starts terminating. Even after it starts terminating, requests continue to come in to the pod that is being terminated.

In this situation, even if a request comes in while the pod is terminating, the pod must not shut down midway through handling it. That is why sleep is used with nginx and php-fpm.

In other words, if php-fpm shuts down first, nginx will return a stream error. So php-fpm must always shut down later than nginx.

That said, it doesn't really matter much when php-fpm and nginx shut down.

The important thing is to sleep for at least the minimum amount of time it takes for connections to be removed from the ALB or LB in the AWS network.

That takes about 15 seconds.

In other words, requests can still come in for 15 seconds even after the pod starts terminating.

So you need to set sleep to at least 15 seconds. (nginx, php-fpm)

How much additional sleep to add after that will probably depend on the server environment.

For example, if a request takes 10 seconds:

15 (network) + 5 (buffer) + 10 (maximum request time) = 30 seconds should be enough.

Also, the value of terminationGracePeriodSeconds must always be greater than the sleep duration.

If sleep is 30 seconds, set terminationGracePeriodSeconds to around 40.

          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - sleep 20
			terminationGracePeriodSeconds: 30

This is usually how you should configure it.

Summary#

The minimum sleep duration is 15 seconds.
terminationGracePeriodSeconds > sleep
Since requests can still come in while a pod is terminating, the goal is to prevent pods from shutting down unexpectedly until the requests have finished.

Summary#

More posts