When you’re architecting your infrastructure around Kubernetes, you need to make sure you’re securing it properly. In a talk at this week’s KubeCon in San Diego, Pushkar Joglekar, security engineer at Visa, explained some of the best practices for securing Kubernetes.
At Visa, the payment workflow boils down to: “Do it fast and with a lot of capacity and make sure it’s as secure as possible,” he explained.
According to Joglekar, if you have a good window of time before deployment, you have time to do a threat model. The first step is to start talking to colleagues about when they want to go live. Then you can start figuring out the threats and putting together a threat model.
One of the popular threat models is STRIDE, which stands for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege.
Spoofing is when an attacker acts as if they are someone they are not. A few ways to mitigate spoofing are to ensure the CA is trusted only within the cluster and to educate developers that when you run something on Kubernetes, it’s not automatically authenticated. It can also be helpful to ask if pods actually need service account tokens, Joglekar explained.
Tampering is the act of trying to change something so that it becomes unusable. There are three types of tampering: at-rest, in-transit, and in-memory. The ways to combat against each of these are to implement auditing, sign, and verify; TLS; and restricted access, respectively.
According to Joglekar, with repudiation, it’s important to understand the who, where, when, what, how, and why. These can be discovered with processes like API auditing, log forwarding, Linux audits, and other Kubernetes audits.
Information disclosure, or data breaches, can be combated against by good secret management. According to Joglekar, using Kubernetes Secrets is a good way of managing secrets, but it’s important to recognize that encryption is not enabled by default and that when encryption is in place, encryption keys are stored in configuration files, which isn’t good. A way around this is to enable KMS plugins.
Denial of Service is one of the harder things to combat against, he said. At the cluster level and pod level, there are things one can do to mitigate risk. At the cluster level, you should have more than one manager node, more than one worker node, network segregation of manager and worker nodes, and resources quotas. At the pod level, you can implement requests and limits, mutual AuthN, and AuthZ policies.
Elevation of privilege is another thing that can be mitigated against. A few steps to take include: not running things as root and setting allowPrivilegeEscalation to false.
Joglekar compared Kubernetes security to earthquakes in California. He lives in the Bay Area, and while he loves it there, earthquakes are prevalent in the area. But just because there is a risk of earthquakes doesn’t mean people are going to leave the state — they build more structurally sound buildings so that the risk of damage is at an acceptable level. The same is true with Kubernetes, you have to calculate what the risk is and compare it with the cost.
You need to understand that not all threats are going to be able to be mitigated by the controls you have. You have to ask: “What’s my risk and what’s the cost of the security product? If the cost of risk is higher, buy the security product,” Joglekar explained.
He also cautioned against becoming a “checklist bot,” explaining that often you can end up sounding more like a lawyer and less like an engineer. It’s fine to have a checklist of things you need to do, but you have to understand why something is in a checklist and what happens when you check the box.
As you move from Day 0 through Day 1 and Day 2, you’ll need to also scale your security as your cluster scales. ““Security is a continuous job and it never ends. You have to keep looking at stuff that is stable as you’re looking at things that come over the horizon,” said Joglekar.
