End-to-End DevOps: CI/CD, GitOps, and Observability on Amazon EKS¶

Overview¶

I deployed Google's Online Boutique (10-service polyglot monorepo) on Amazon EKS with a production-grade DevOps pipeline built from scratch: GitHub Actions CI with Trivy security scanning, Helm chart packaging to GHCR, ArgoCD GitOps with Image Updater for continuous delivery, full observability (Prometheus, Grafana, ELK, Slack alerting), Gateway API networking, and HPA autoscaling.

Architecture at a Glance¶

Developer pushes code to src/
    │
    ▼
GitHub Actions (CI)
    ├── Trivy filesystem + image scan
    ├── Docker build (BuildKit + GHA cache)
    └── Push 3 tags to GHCR (:sha-<40>, :sha-<7>, :latest)
          │
          ▼
ArgoCD Image Updater (on cluster)
    ├── Polls GHCR every 2 min
    ├── Detects new :latest digest
    └── Patches ArgoCD Application
          │
          ▼
ArgoCD syncs, pods roll
    ├── 10 microservices in boutique-app namespace
    ├── Gateway API HTTPRoute -> shared ALB -> app.ibtisam.qzz.io
    └── ExternalDNS auto-creates Route 53 records

Observability
    ├── Prometheus + Grafana + AlertManager (Slack)
    └── Elasticsearch + Filebeat + Kibana

5 subdomains, 1 ALB, 1 wildcard cert, 0 manual DNS records

Phases¶

The project is documented across 6 phases. Each phase has its own runbook with step-by-step commands, decisions, bugs encountered, and terminal session recordings as evidence.

Key Decisions¶

Architectural and engineering decisions made across all 6 phases. Each links to the runbook where it is documented in detail with full rationale.

Repository and Code Strategy¶

• Never modify upstream files. The fork stays pristine and syncable. All customization lives in files added alongside upstream. (Phase 1)
• Two-repo separation. CI repo owns code and pipelines. CD repo owns deployment intent. CI has zero knowledge of the cluster for code pushes. (Phase 1, Phase 4)
• All deployment manifests in CD repo, nothing in CI repo. The source repo contains no Kubernetes resources, no values files, no Helm overrides. (Phase 4)
• Deleted 2 upstream workflows, kept 3. ci-main.yaml and ci-pr.yaml were replaced with a monorepo-aware matrix pipeline. helm-chart-ci.yaml was kept for chart validation. (Phase 1)

Image Tagging and Continuous Delivery¶

• Image tagging evolved through 3 iterations. Chart version tags (broken for CD), immutable SHA tags (noisy), and finally ArgoCD Image Updater with digest strategy. (Phase 1)
• Chose Approach B (Image Updater) over SHA-based GitOps. CI pushes images and stops. Image Updater watches GHCR and handles deployments. Eliminated reusable-gitops.yaml entirely and removed the GIT_TOKEN dependency for code pushes. (Phase 1)
• Digest strategy, not newest-build. BuildKit sets the image config's created timestamp to epoch (1970-01-01) for reproducible builds. newest-build cannot differentiate tags. digest tracks :latest tag's digest directly, no timestamps involved. (Phase 4)
• tag: "latest" in values, not empty string. tag: "" falls back to Chart.AppVersion, but CI never pushes images with the chart version tag. Image Updater needs :latest to track. (Phase 4)

Helm Chart and Values¶

• Chart packaged from upstream as-is. No upstream files modified. The chart on GHCR ships with Google's defaults. EKS overrides live in the CD repo's values-eks.yaml. (Phase 1)
• Patch-only values file. Only 5 fields that differ from upstream: registry, tag, externalService, platform, loadGenerator. Everything else uses upstream defaults. (Phase 4)
• loadGenerator disabled. Excluded from CI matrix (no image in GHCR). Setting create: false in values prevents ImagePullBackOff. (Phase 4)
• chart-release.yaml absorbs CD repo update. reusable-gitops.yaml was deleted. Chart version writes moved into chart-release.yaml. Image tag writes eliminated entirely. (Phase 1)

Networking and DNS¶

• Gateway API instead of Ingress. The Kubernetes project recommends Gateway and states the Ingress API has been frozen. (Phase 3)
• Single shared ALB via Gateway API. One Gateway, one ALB, 5 HTTPRoutes across namespaces. All subdomains served through one load balancer. (Phase 3)
• Wildcard ACM certificate. *.ibtisam.qzz.io covers all subdomains. No new certificate needed when a subdomain is added. (Phase 2)
• Free domain from digitalplat.org. No cost for the qzz.io domain used across the project. (Phase 2)
• ExternalDNS with Gateway API sources. Auto-creates Route 53 records from HTTPRoutes. Zero manual DNS records in the entire project. (Phase 3)
• HTTPRoute defaults included in Git. Gateway API controller injects group, kind, weight, and matches defaults post-creation. Including them in the manifest prevents ArgoCD OutOfSync drift. (Phase 4)

Platform Architecture¶

• ArgoCD manages the app, not the platform. Monitoring and logging stacks deployed via Helm, not ArgoCD. If ArgoCD breaks, observability tools remain operational for debugging. (Phase 4, Phase 5)
• gp3 as default StorageClass. Cheaper and better baseline than gp2. Required for Elasticsearch PVCs. (Phase 3)
• Self-managed nodes (managed blocked by SCP). KodeKloud AWS Playground enforces SCPs that block managed node groups. Self-managed nodes via CloudFormation. (Phase 2)
• Scaled ASG from 3 to 4 nodes. 3 nodes could not fit the full stack (app + monitoring + logging). Kibana pod stuck Pending until the 4^th node joined. (Phase 5)

Security and Scanning¶

• Trivy CRITICAL gate temporarily relaxed. Designed as exit-code 1 (hard gate). Currently exit-code 0 because upstream base images carry known CRITICAL CVEs. Restore once patched. (Phase 1)
• GIT_TOKEN scoped to CD repo only. Fine-grained PAT with Contents: Read+Write on platform-engineering-systems only. Token never in process argv. (Phase 1)

Terminal Sessions¶

Every phase was recorded. The terminal sessions capture the exact commands, outputs, and errors encountered.

Screenshots¶