K8S done right is literally designed for multi tenancy. A separate cluster per customer would be insanely costly and terrible for the planet. Maybe in premium products where security is paramount, but a separate cluster per customer is basically setting your money on fire.
Multi-tenant Kubernetes[1] is a strcpy-level footgun IMO. It's perfectly fine as long as everyone involved does everything correctly and never makes a mistake.
Kubernetes itself is very complex. The "who needs a UI when you have configuration files and an API?" approach makes it even more opaque to the people who often end up responsible for it. The landscape changes very rapidly.[2] I'd trust Kelsey Hightower to set up a secure multi-tenant deployment, but probably not anyone else.
Is it not practical to deploy clusters on top of virtualization? That should make efficient use of hardware while still giving each tenant their own cluster, therefore providing stronger isolation than the typical Kubernetes configuration tends to.
[1] I am specifically referring to a Kubernetes deployment where different customers are running custom code on the same underlying hosts. Using Kubernetes to host a service that is multi-tenant at a higher level is not something I would recommend, but it's not as immediately dangerous as a model where customers run container-level custom code.
[2] This is not surprising for a relatively new technology, especially one that's as paradigm-shifting as Kubernetes was. But most people are not going to rearchitect and redeploy everything every six months just because the Kubernetes developers decided to replace a pod security or network security model with a non-backward-compatible alternative again.
> K8S done right is literally designed for multi tenancy
No it is not. Not in the way they are using it.
There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails
The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not.
There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small.
