Terraform and Kubernetes

# CHAPTER 13

Terraform and Kubernetes #

1. Introduction #

2. Learning Objectives #

• Differentiate between provisioning a K8s Cluster vs. provisioning K8s Resources.

• Configure the Terraform Kubernetes Provider.

• Define Kubernetes Namespaces, Deployments, and Services using HCL.

• Understand the integration between the AWS/GCP providers and the Kubernetes provider.

• Evaluate the pros and cons of using Terraform versus Helm/YAML for K8s deployments.

3. Beginner-Friendly Explanation #

• The AWS Provider (Building the Port): You use Terraform to pour the concrete, build the cranes, and hire the port manager. (Provisioning the EKS/GKE Kubernetes Cluster).

• Kubernetes (The Manager): The port manager whose only job is to organize shipping containers.

• The Kubernetes Provider (Giving Orders): You use Terraform to hand a clipboard to the port manager. The clipboard says: "I need exactly 5 containers of shoes unloaded, and if one breaks, instantly replace it." (Deploying Pods and Services).

4. Phase 1: Provisioning the Cluster (Hardware) #

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "my-company-cluster"
  cluster_version = "1.27"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_groups = {
    # We define the worker nodes (the servers that actually run the containers)
    general_compute = {
      min_size     = 1
      max_size     = 3
      desired_size = 2
      instance_types = ["t3.large"]
    }
  }
}

5. Phase 2: Deploying to the Cluster (Software) #

provider "kubernetes" {
  # We dynamically pull the authentication details from the AWS EKS module above!
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

6. Mini Project: Deploy a K8s Application via Terraform #

Step-by-Step Architecture Concept:

# 1. Create a Namespace (A logical partition in the cluster)
resource "kubernetes_namespace" "web_app" {
  metadata {
    name = "frontend"
  }
}

# 2. Create the Deployment (The containers)
resource "kubernetes_deployment" "nginx" {
  metadata {
    name      = "nginx-deployment"
    namespace = kubernetes_namespace.web_app.metadata[0].name
  }

  spec {
    replicas = 3 # Kubernetes will ensure exactly 3 containers are always running

    selector {
      match_labels = { app = "nginx" }
    }

    template {
      metadata { labels = { app = "nginx" } }
      spec {
        container {
          image = "nginx:1.21.6"
          name  = "nginx"
          port { container_port = 80 }
        }
      }
    }
  }
}

*When you run terraform apply, Terraform talks to the Kubernetes API, which then pulls the Docker image and spins up 3 highly available containers.*

7. Real-World Scenarios #

8. Best Practices #

• Terraform vs Helm/YAML: The industry is divided on this. While Terraform *can* deploy Kubernetes resources, many experts prefer to use Terraform ONLY to build the cluster (Phase 1), and use tools like Helm or ArgoCD to deploy the applications (Phase 2). Terraform's state file struggles with Kubernetes resources that auto-update themselves (like auto-scaling pods), causing Terraform to constantly think the state has drifted. Evaluate carefully if you want to use the kubernetes provider for application deployment.

9. Security Recommendations #

• Authentication Execution: In the provider configuration, we used an exec block to dynamically generate an AWS authentication token (aws eks get-token). This is vastly superior to hardcoding a static, long-lived Kubernetes .kube/config file in your CI/CD pipeline, ensuring that authentication is ephemeral and tightly coupled to AWS IAM roles.

10. Troubleshooting Tips #

• Provider Initialization Failures: If you write the code to create the EKS cluster and the code to deploy the Kubernetes Deployment in the *exact same* Terraform apply, Terraform will often crash. It tries to initialize the kubernetes provider before the EKS cluster actually exists. The best practice is to separate cluster creation and application deployment into two distinct Terraform projects (workspaces).

11. Exercises #

1. 1. Explain the architectural difference between the aws provider (creating an EKS cluster) and the kubernetes provider.

1. 2. Why is using the exec block to dynamically generate an authentication token preferred over using a static kubeconfig file?

12. FAQs #

13. Interview Questions #

• Q: Describe a unified deployment workflow using Terraform to provision both the underlying cloud Kubernetes infrastructure (e.g., GKE or EKS) and the application workloads. What are the synchronization challenges of defining both in the same state file?

• Q: Contrast the operational philosophy of deploying Kubernetes workloads using Terraform's kubernetes_deployment resource versus utilizing native Kubernetes manifest files with a GitOps tool like ArgoCD.

14. Summary #

15. Next Chapter Recommendation #