Advanced Usage: Calico Troubleshooting

For advanced Kubernetes networking scenarios, such as troubleshooting the CNI (Container Network Interface) plugin itself, netshoot can be deployed with specific configurations to interact with the control plane of the networking solution.

This example demonstrates how to deploy netshoot as a pod with the necessary permissions and configuration to interact with a Calico backend that uses etcd for storage. This allows you to use calicoctl and other tools from within the netshoot container to inspect the state of the Calico network.

Example: netshoot-calico Deployment

This manifest creates a netshoot pod that runs on a master node and has access to Calico's certificates and control sockets.

---
# NOTE: use apps/v1beta1 before 1.9.x and apps/v1beta2 before v1.16.x
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kube-system
  name: netshoot-calico-deploy
  labels:
    app: netshoot-calico
spec:
  selector:
    matchLabels:
      app: netshoot
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: netshoot
    spec:
      serviceAccount: cni-plugin
      serviceAccountName: cni-plugin
      hostNetwork: true
      containers:
      - image: nicolaka/netshoot
        name: netshoot
        command: ["ping","localhost"]
        volumeMounts:
        - mountPath: /calico-secrets
          name: etcd-certs
        - mountPath: /var/run/calico/bird.ctl
          name: bird-ctl
        env:
        - name: ETCD_ENDPOINTS
          value: "https://localhost:12378"
        - name: ETCD_CA_CERT_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_ca
              name: calico-config
        - name: ETCD_KEY_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_key
              name: calico-config
        - name: ETCD_CERT_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_cert
              name: calico-config
      volumes:
        - name: etcd-certs
          secret:
           defaultMode: 420
           secretName: calico-etcd-secrets
        - name: bird-ctl
          hostPath:
            path: /var/run/calico/bird.ctl
      nodeSelector:
        node-role.kubernetes.io/master: ""

Breakdown of the Configuration

• namespace: kube-system: Deployed into the kube-system namespace where control plane components often reside.
• hostNetwork: true: The pod runs in the host's network namespace, allowing it to connect to services running on the node's localhost, such as etcd.
• nodeSelector: This ensures the pod is scheduled only on nodes with the node-role.kubernetes.io/master label, where the etcd cluster and Calico control plane components are typically running.
• volumeMounts & volumes:
  • etcd-certs: The calico-etcd-secrets secret, which contains the client certificates for authenticating with etcd, is mounted into the pod.
  • bird-ctl: The BIRD control socket (bird.ctl) is mounted from the host into the pod. This allows you to interact with the BIRD routing daemon using tools like birdctl.
• env: Environment variables are configured to point calicoctl to the correct etcd endpoint and certificate files, enabling it to communicate with the Calico datastore.

With this setup, you can kubectl exec into the netshoot pod and run calicoctl commands to debug Calico network policies, IP pools, and BGP peering sessions.