Protecting Your Pods from Kubelet
The title may weird because the Kubelet is our friend that manages Container stuff on a node. This friend sometimes runs our containers and sometimes it kills our containers. We are gonna talk about what we will do when resource starvations have arisen.
Pod resources are one of the most important things, especially in the case of mission-critical applications. Misconfigured or missed resource directions may affect your system unexpectedly. In a sense why is that an application has to not run. Let’s talk about setting Pod resources correctly.
Firstly you have to find convenient resource requests and limits for your application. You can use some tools for that or watch it on production by not providing any resource for short time. When you don’t set any resource definition for your application Kubernetes permits to use of whole resources. This is called Best-Effort. This is dangerous in production but I’m assuming that your purpose is just to test the application’s resource usage.
Assume that you found this data by monitoring with some tools
# rule you found
resources:
limit:
cpu: 500m
memory: 1Gi
request:
cpu: 100m
memory: 200MiNow we have to discuss Quality of Service because another application designing strategy is that.
Quality of Service Definition
Depending on whether you specify the requests, the limits, or both, the platform offers a different kind of Quality of Service (QoS).
Best-Effort
The pod does not have any requests and limits set for its containers. Such a Pod is considered as the lowest priority and is killed first when the node where the Pod is placed runs out of incompressible resources.
Burstable
Pod that has requests and limits defined, but they are not equal (and limits is larger than requests as expected). Such a Pod has minimal resource guarantees but is also willing to consume more resources up to its limit when available. When the node is under incompressible resource pressure, these Pods are likely to be killed if no Best-Effort Pods remain.
Guaranteed
The pod has an equal amount of requests and limits. These are the highest-priority Pods and guaranteed not to be killed before Best-Effort and Burstable Pods.
When we want to create a sorted list that represents which QoS is killed first by Kubelet this list below arises:
- Best-Effort
- Burstable
- Guaranteed
Remember, you found a set of resource rule
# rule you found
resources:
limit:
cpu: 500m
memory: 1Gi
request:
cpu: 100m
memory: 200MiThis rule you found is Burstable as you read above. If your application shouldn’t affect by any resource starvation that any other application can occur use Guaranteed QoC instead of Burstable.
QoC is matters but it's not everything. Even you set QoC right but the environment may not be convenient and Kubelet kills your containers again. You may be working at a cluster that there are multiple teams and so many applications. The cluster may not start an application with 1 replica because there is resource starvation. Maybe you or another one forgot to use resource limits. But it’s not just your fault to forget to set Pod resources because administrators should manage resources on a shared cluster that used by multiple teams because sometimes resources can’t be controlled by teams working on the same cluster and this causes too the Pod Evictions or Non-Scheduled. Both situations are a mess.
At this point limiting the resources per namespace maybe work. Currently, there are two are admission controllers as solutions for these problems.
Let’s discuss the LimitRanger.
LimitRanger
It enforces defaults and limits for all Pods and Containers that do not set compute resource requirements and tracks usage to ensure it does not exceed resource minimum, maximum, and ratio defined in any LimitRange present in the namespace. When you are gonna create/update a Container spec LimitRanger validates its resource spec.
A fully configured example is below. This definition explains itself. But we will go through cases
apiVersion: v1
kind: LimitRange
metadata:
name: lr-demo
namespace: team-a
spec:
limits:
- type: Pod
min:
cpu: 50m
memory: 50Mi
max:
cpu: 2
memory: 3Gi
- type: Container
default:
cpu: 300m
memory: 500Mi
defaultRequest:
cpu: 100m
memory: 200Mi
min:
cpu: 50m
memory: 50Mi
max:
cpu: 500m
memory: 1Gi
maxLimitRequestRatio:
cpu: 10Let’s inspect this spec.
Pod type means the maximum amount of CPU/Memory that a pod can request on a node across all containers.
Container type means the maximum amount of CPU/Memory that a single container in a pod can request.
default key specifies the default limits of resource of container’s resource.
defaultRequest key specifies the default requests of container’s resource.
maxLimitRequestRatio means the maximum amount of CPU burst that a container can make as a ratio of its limit over request.
Examples
When you are trying to create a Pod in lr-demo namespace and you are:
- forgotten to specify any resources spec on your Container then it puts the defaults into your Pod spec
- specified limit.cpu too much you will get forbidden error
- specified limit.memory too much you will get forbidden error
- specified request.cpu too low you will get forbidden error
- specified request.memory too low you will get forbidden error
So we learned some things about LimitRanger. My thought is, the administrator should use it on shared clusters so one team didn’t affect to another. But it comes with a maintenance cost. Some apps may use too much CPU/Memory, they are exceptions. But if you have an exceptional app then you may want to deploy your exceptional apps to another namespace.
We won’t discuss ResourceQuota but it is also great that provides you much more ruleset.
Conclusion
We were talked about protecting our applications from Kubelet and also other teams. I suggest setting QoC but LimitRanger is a helper for protecting your applications from resource starvations issues that other teams cause and it is not necessary if you are thinking you won’t be affected by this.
