A Step-by-Step Guide to Using Secret Manager with GKE and CSI Driver

Introduction 

Management of sensitive information such as API keys, credentials and configuration secrets are an important part of developing safe and reliable Skylands applications. In Google Cloud Ecosystem, Secret Manager provides a centralized and safe way to save, access and manage these secrets.When running applications on Google Kubernetes Engine (GKE), including Secret Manager guarantees that your workloads are able to access secrets securely without having to hard-code them into your containers or codebase.

Objective :-

The objective of this blog is to provide a comprehensive guide on implementing the secret manager using the GCP Key Vault Provider for Secrets Store Container Storage Interface (CSI) Driver within an Google Kubernetes Engine (GKE) cluster. Through this blog we will understand the integration of secret managers in Google Kubernetes Engine by securely managing and accessing the sensitive information , such as secret and credentials in GKE.

Managing secrets through secret manager

What is Secret Manager ?

Secret Manager is a secure and centralized service on Google Cloud designed to store sensitive information such as API keys, passwords, and certificates. It simplifies the management, access, and auditing of secrets across your cloud environment, helping you keep critical data safe and organized.

Why to use secret manager ?

1. Security : Secrets are encrypted at rest using Google-managed or customer-managed keys (CMEK).
2. Centralised Secret Management : Google Cloud Secret Manager securely and centrally stores secrets.
3. Workload Identity-Based Access Control : GKE workloads can access secrets securely without service account keys using Workload Identity.
4. Versioning & Rotation : It has versioning, so you are able to update and roll back securely.
5. Integration with CI/CD Pipelines : Seamlessly integrates with Cloud Build, GitHub Actions, or other CI/CD to inject secrets during deployment or build.

Secret manager Setup :-

1. Create a gke cluster :- Create gke cluster either via console , terraform or by cli.

GKE Cluster

2. Turn on the secret manager add-on :- Be sure to activate the secret manager add-on when setting up a GKE cluster.Access the secrets kept in Secret Manager as volumes are mounted in Kubernetes pods by using the Secret Manager add-on.

GKE Cluster add-ons

3. Verify that after enabling the secret manager add-on Secrets Store CSI Driver installation :- Verify the installation is finished using the kubectl get pods command, which lists all pods with the csi-  secrets-store-gke and csi-secrets-store-provider-gke labels in the kube-system  namespace.

CSI Secret Pods

4. Create secrets in a secret manager :- A Secret Manager was created using the GCP Console to guard against unwanted access and safely store private information like passwords and API keys. To effectively handle updates and preserve a safe history of modifications, secret versions are made.

Secrets in secret manager

Secret Version

5.Create a Secret Provider Class :- To specify how secrets from the secret manager are accessed and mounted into your Kubernetes pods, create a SecretProviderClass using the YAML below.

Secretproviderclass.yaml :-

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: dev-secret-provider
namespace: dev
spec:
provider: gke
parameters:
secrets: |
- resourceName: "projects/954514138689/secrets/DB_USER/versions/1"
fileName: "DB_USER"
- resourceName: "projects/954514138689/secrets/DB_PASSWORD/versions/1"
fileName: "DB_PASSWORD"

secretObjects:
- secretName: dev-secrets
type: Opaque
data:
- objectName: "DB_USER"
key: "DB_USER"
- objectName: "DB_PASSWORD"
key: "DB_PASSWORD"

6. Apply the SecretProviderClass YAML : – Apply the SecretProviderClass YAML using the following command in GKE.

kubectl apply -f secretproviderclass.yaml

Note :-
1.No need to create k8s secret object separately. It will be created automatically when you create a pod with CSI driver volume.
2.The secretObjects block in the above YAML is optional and is only needed if you need to synchronize mounted content with a Kubernetes secret. You will still get the secret manager object mounted to the pod if you do not use this block.

6. Give permission to the CSI driver : – Give permission to CSI driver secret store to use cluster roles and cluster role binding to generate GKE secrets.

Secret-store-rbac.yaml:-

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secrets-store-csi-secrets-access
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch", "create", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: secrets-store-csi-secrets-access-binding
subjects:
- kind: ServiceAccount
  name: secrets-store-csi-driver-gke
  namespace: kube-system
roleRef:
kind: ClusterRole
name: secrets-store-csi-secrets-access
apiGroup: rbac.authorization.k8s.io

7. Set an environment variable to reference Kubernetes secrets :- Create a deployment.yaml for gke and pass the reference of gke secrets and mount volume on the deployment.yaml.
Below is an example of the deployment.

Example of Gke deployment.yaml

8.Create service file : – Create a service.yaml file to expose your deployment and allow network access to the application and apply them using the commands.

service.yaml

Kubectl apply  -f deployment.yaml
Kubectl apply  -f service.yaml

9. Verify the gke secrets : – After applying the service and deployment verify that the csi driver created the gke secrets or not using the following command.

Kubectl get secrets -n dev

Kubernetes Secrets

Conclusion :-

In conclusion, maintaining the security and integrity of your Google Kubernetes Engine applications depends heavily on efficient secret management. Organisations can greatly lower the risk of sensitive data exposure by utilising GKE’s built-in features, like Kubernetes Secrets, and integrating with reliable programs like Google Cloud Secret Manager. Developers and operations teams can create and implement secure, resilient applications in the GKE environment with confidence if the strategies discussed are put into practice, from appropriate secret creation and distribution to strong access controls and frequent rotation.