Let a Kubernetes Container create another Container in the cluster

For one of the projects I am working on I have the requirement to let a container within a pod create another container in a new pod in a separate namespace. This to allow new abstraction layers to be created.

Naive Approach

At first it seemed straightforward, just create a piece of code that simply lists the pods running on the cluster (to demonstrate Kubernetes API access) through the Kubernetespackage in Python:

import kubernetes.client as k8s_client
import kubernetes.config as k8s_config

kube_api_v1 = k8s_client.CoreV1Api()

print("Listing pods with their IPs:")
ret = self.kube_api_v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

This will work locally! But this is obviously not being executed on the cluster. When we now run this on the cluster itself from inside a container in a pod, we will get:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4584536070>: Failed to establish a new connection: [Errno 111] Connection refused'))

Which makes sense as:

1. A container is isolated on network level
2. A container has specific rights within the cluster, preventing takeovers

Service Accounts

Now luckily for us, we can do this through Service Accounts! Service Accounts allow roles with permissions to be used that can perform certain actions within a namespace.

Now what if we are in namespace A and want to perform actions in namespace B? Well for that we allocate 2 roles within the 2 namespaces and create a rolebinding in namespace B that binds the service account of namespace A.

Creating the namespaces

kubectl create namespace poc-a
kubectl create namespace poc-b

Creating the service accounts

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: poc-a
  namespace: poc-a
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: poc-b
  namespace: poc-b
EOF

Creating the roles

cat <<EOF | kubectl apply -f -
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: poc-a-role
  namespace: poc-a
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["pods", "services", "deployments", "replicasets"]
  verbs: ["get", "watch", "list", "create", "update", "delete"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pob-b-role
  namespace: poc-b
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["pods", "services", "deployments", "replicasets"]
  verbs: ["get", "watch", "list", "create", "update", "delete"]
EOF

Create the default role bindings

cat <<EOF | kubectl apply -f -
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: poc-a-rolebinding
  namespace: poc-a
subjects:
- kind: ServiceAccount
  name: poc-a
roleRef:
  kind: Role
  name: poc-a-role
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: poc-b-rolebinding
  namespace: poc-b
subjects:
- kind: ServiceAccount
  name: poc-b
roleRef:
  kind: Role
  name: poc-b-role
  apiGroup: rbac.authorization.k8s.io
EOF

Allow SA (ns: poc-a) access to ns: poc-b

cat <<EOF | kubectl apply -f -
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: poc-a-sa-access-to-poc-b-rolebinding
  namespace: poc-b
subjects:
- kind: ServiceAccount
  name: poc-a
  namespace: poc-a
roleRef:
  kind: Role
  name: poc-b-role
  apiGroup: rbac.authorization.k8s.io
EOF

Testing through simple Python Script

To test this now we can create a simple Python script that lists the containers in its current namespace and in the other namespace:

k8s_config.load_incluster_config()
kube_api_v1 = k8s_client.CoreV1Api()

# Get the current namespace (poc-a)
namespace = open(
    '/var/run/secrets/kubernetes.io/serviceaccount/namespace'
).read()

print("Current namespace: " + namespace)

print(f"========= Pods in NS: {namespace} =========")
pods = self.kube_api_v1.list_namespaced_pod(namespace=namespace)
for pod in pods.items:
    print("- ", pod.metadata.name)
    
print(f"========= Pods in NS: poc-b =========")
pods = self.kube_api_v1.list_namespaced_pod(namespace="poc-b")
for pod in pods.items:
    print("- ", pod.metadata.name)

Which will print out the pods in both namespaces, showing that the container is able to access the service account poc-a and use the rolebinding to access poc-b its namespace!

Conclusion

While first it might seem daunting to use Service Accounts, Kubernetes is actually a super sweet platform with a lot of customization options. By this option, we are able to - securely - manage what containers can do in their namespaces and how we can interact between namespaces, isolating our software where needed!