Event-Driven GitOps: Deploying Polyglot Microservices on Amazon EKS [Terraform, GitHub Actions, Helm, Kustomize, Gateway API, ExternalDNS, ArgoCD, Image Updater, Prometheus, Elastic Stack]¶

Overview¶

I deployed Google's Online Boutique (10-service polyglot monorepo) on Amazon EKS with a production-grade, highly automated DevOps architecture built from scratch. I provisioned the underlying infrastructure using Terraform. For the CI/CD lifecycle, I engineered 3 CI pipelines for change detection, Trivy security scanning, and automated Helm chart releases to GHCR. On the cluster side, I configured ArgoCD for declarative GitOps deployments, utilizing Kustomize to combine raw Gateway API manifests alongside the packaged Helm chart. The platform utilizes the Kubernetes Gateway API for advanced traffic routing, ExternalDNS for automated Route 53 record provisioning, and ArgoCD Image Updater for zero-touch continuous delivery. Comprehensive observability is achieved via kube-prometheus-stack for monitoring, AlertManager and Slack for notifications, and the Elastic Stack for centralized logging.

Architecture at a Glance¶

The entire CI/CD pipeline operates as a fully automated, event-driven system. It reacts to events from source to cluster, guaranteeing continuous delivery with absolutely zero manual intervention, Git writes, or PR approvals for deployments.

[Infrastructure Provisioning]
Terraform ──> VPC, EKS Cluster, Route 53 Zone, ACM Certificate

[Event: Code Push] Developer pushes code to src/
    │
    ▼
GitHub Actions (3 CI Pipelines)
    ├── Change Detection & Trivy Scans
    ├── Docker Build & Helm Packaging
    └── [Event: Image Publish] Push artifacts to GHCR
          │
          ▼
ArgoCD Image Updater (Continuous Delivery)
    ├── Polls GHCR every 2 min
    ├── [Event: Registry Detection] Detects new :latest digest
    └── Patches ArgoCD Application
          │
          ▼
ArgoCD (GitOps Controller)
    ├── Combines Helm Chart + raw Gateway API manifests via Kustomize
    └── [Event: GitOps Sync] Deploys to EKS Cluster
          ├── 10 microservices in boutique-app namespace
          ├── Gateway API HTTPRoute -> shared ALB -> app.ibtisam.qzz.io
          └── ExternalDNS auto-creates Route 53 records

Observability Stack
    ├── kube-prometheus-stack (Monitoring & Slack Alerts)
    └── Elastic Stack (Centralized Logging)

5 subdomains, 1 ALB, 1 wildcard cert, 0 manual DNS records

Top Architectural Highlights¶

This project intentionally deviates from standard deployments by making several advanced engineering decisions. These are the core differentiators:

1. Automated Route 53 Provisioning: I entirely avoided manual DNS management. Instead, I integrated ExternalDNS with the Kubernetes Gateway API to automatically create, update, and reconcile AWS Route 53 records based on cluster HTTPRoutes.
2. Immutable Upstream Helm Charts via Kustomize: I refused to modify the upstream Helm chart to avoid breaking future syncs. I used the developer's provided values to configure the chart, and then used Kustomize to inject my own custom Gateway API manifests (HTTPRoutes, ALB target groups) alongside the Helm package directly within ArgoCD.
3. Decoupling Application from Platform: I intentionally deployed only the microservices application via ArgoCD. The observability stacks (kube-prometheus-stack and Elastic Stack) were deployed independently. If ArgoCD or the deployment pipeline fails, the monitoring and logging infrastructure remains fully operational to debug the outage.
4. Zero-Touch Continuous Delivery: The workflow requires absolutely zero manual intervention. A developer pushes code, the CI pipeline builds/scans/publishes the artifact to GHCR, ArgoCD Image Updater detects the new digest, ArgoCD autonomously rolls out the pods, and AlertManager pushes notifications to Slack.
5. Gateway API over Legacy Ingress: Following official Kubernetes documentation, I abandoned the frozen Ingress API in favor of the modern Gateway API. This provides a highly scalable, role-oriented networking architecture that elegantly handles advanced traffic routing and native resource sharing (one ALB mapping to multiple independent HTTPRoutes).

Phases¶

The project is documented across 6 phases. Each phase has its own runbook with step-by-step commands, decisions, bugs encountered, and terminal session recordings as evidence.

Key Decisions¶

Architectural and engineering decisions made across all 6 phases. Each links to the runbook where it is documented in detail with full rationale.

Repository and Code Strategy¶

• Never modify upstream files. The fork stays pristine and syncable. All customization lives in files added alongside upstream. (Phase 1)
• Two-repo separation. CI repo owns code and pipelines. CD repo owns deployment intent. CI has zero knowledge of the cluster for code pushes. (Phase 1, Phase 4)
• All deployment manifests in CD repo, nothing in CI repo. The source repo contains no Kubernetes resources, no values files, no Helm overrides. (Phase 4)
• Deleted 2 upstream workflows, kept 3. ci-main.yaml and ci-pr.yaml were replaced with a monorepo-aware matrix pipeline. helm-chart-ci.yaml was kept for chart validation. (Phase 1)

Image Tagging and Continuous Delivery¶

• Image tagging evolved through 3 iterations. Chart version tags (broken for CD), immutable SHA tags (noisy), and finally ArgoCD Image Updater with digest strategy. (Phase 1)
• Chose Approach B (Image Updater) over SHA-based GitOps. CI pushes images and stops. Image Updater watches GHCR and handles deployments automatically. Eliminated reusable-gitops.yaml entirely and removed the GIT_TOKEN dependency for code pushes. (Phase 1)
• Update strategy: digest. BuildKit sets the image config's created timestamp to epoch (1970-01-01T00:00:00Z) for reproducible builds. newest-build was evaluated but cannot differentiate tags. digest compares the :latest tag's digest in GHCR against what is currently running, bypassing timestamps. (Phase 4)
• Update method: argocd (default). Image overrides are patched directly onto the Application CR as kustomize image entries. No Git write-back is performed, which avoids noisy commits. If the Application is ever deleted and recreated from Git, Image Updater re-applies the overrides on its next poll cycle.
• Traceability Tradeoff. Pods show frontend:latest@sha256:<digest> instead of frontend:sha-<commit>. Commit traceability requires looking up the digest in GHCR. To support this audit trail, CI pushes :sha-<40char>, :sha-<7char>, and :latest tags on every build.
• tag: "latest" constraint. In values-eks.yaml, tag: "latest" is explicitly set rather than an empty string. Image Updater requires a version constraint to know which tag to track for digest changes. Without it, the strategy fails. (Phase 4)

Helm Chart and Values¶

• Chart packaged from upstream as-is. No upstream files modified. The chart on GHCR ships with Google's defaults. EKS overrides live in the CD repo's values-eks.yaml. (Phase 1)
• Patch-only values file. Only 5 fields that differ from upstream: registry, tag, externalService, platform, loadGenerator. Everything else uses upstream defaults. (Phase 4)
• loadGenerator disabled. Excluded from CI matrix (no image in GHCR). Setting create: false in values prevents ImagePullBackOff. (Phase 4)
• chart-release.yaml absorbs CD repo update. reusable-gitops.yaml was deleted. Chart version writes moved into chart-release.yaml. Image tag writes eliminated entirely. (Phase 1)

Networking and DNS¶

• Gateway API instead of Ingress. The Kubernetes project recommends Gateway and states the Ingress API has been frozen. (Phase 3)
• Single shared ALB via Gateway API. One Gateway, one ALB, 5 HTTPRoutes across namespaces. All subdomains served through one load balancer. (Phase 3)
• Wildcard ACM certificate. *.ibtisam.qzz.io covers all subdomains. No new certificate needed when a subdomain is added. (Phase 2)
• Free domain from digitalplat.org. No cost for the qzz.io domain used across the project. (Phase 2)
• ExternalDNS with Gateway API sources. Auto-creates Route 53 records from HTTPRoutes. Zero manual DNS records in the entire project. (Phase 3)
• HTTPRoute defaults included in Git. Gateway API controller injects group, kind, weight, and matches defaults post-creation. Including them in the manifest prevents ArgoCD OutOfSync drift. (Phase 4)

Platform Architecture¶

• ArgoCD manages the app, not the platform. Monitoring and logging stacks deployed via Helm, not ArgoCD. If ArgoCD breaks, observability tools remain operational for debugging. (Phase 4, Phase 5)
• gp3 as default StorageClass. Cheaper and better baseline than gp2. Required for Elasticsearch PVCs. (Phase 3)
• Self-managed nodes (managed blocked by SCP). KodeKloud AWS Playground enforces SCPs that block managed node groups. Self-managed nodes via CloudFormation. (Phase 2)
• Scaled ASG from 3 to 4 nodes. 3 nodes could not fit the full stack (app + monitoring + logging). Kibana pod stuck Pending until the 4^th node joined. (Phase 5)

Security and Scanning¶

• Trivy CRITICAL gate temporarily relaxed. Designed as exit-code 1 (hard gate). Currently exit-code 0 because upstream base images carry known CRITICAL CVEs. Restore once patched. (Phase 1)
• GIT_TOKEN scoped to CD repo only. Fine-grained PAT with Contents: Read+Write on platform-engineering-systems only. Token never in process argv. (Phase 1)

Terminal Sessions¶

Every phase was recorded. The terminal sessions capture the exact commands, outputs, and errors encountered.

Screenshots¶