The process of downloading and using secrets in Kubernetes involves several components working together:

1. Secret Creation: First, a secret is created in the cluster, typically using kubectl or programmatically through the API:

   kubectl create secret generic db-credentials --from-literal=username=admin --from-literal=password=secret123
   

2. Pod Reference: A pod spec references this secret either as environment variables or as mounted volumes:

   # In a pod spec
   volumes:
   - name: secret-volume
    secret:
      secretName: db-credentials
   

3. API Server Role: When a pod with secret references is created, the API server:

   • Validates that the pod has permission to access the requested secrets
   • Does not store the full pod spec with embedded secret data in etcd

4. Kubelet's Secret Download Process:

   • When the kubelet starts a pod, it determines which secrets the pod needs
   • Kubelet makes authenticated, authorized requests to the API server to fetch only the secrets needed by its pods
   • Kubelet stores secrets in a tmpfs (memory-backed filesystem) on the node, not on persistent disk
   • For volume-mounted secrets, kubelet creates an in-memory tmpfs mount for the pod with the secret data
   • For environment variable secrets, kubelet injects them into the container's environment

5. Security Considerations:

   • Secrets are only distributed to nodes running pods that need them
   • On the node, secrets are only visible to the pods that requested them
   • When a pod is deleted, kubelet deletes its local copy of the secrets
   • Each secret is limited to 1MB in size

The key insight here is that secrets are never written to permanent storage on nodes - they exist only in memory (tmpfs) and only on nodes that need them.

One important note: by default, secrets are stored unencrypted in etcd. For production, you should enable encryption at rest for the etcd database to fully secure your secrets.