Multi-Environment Orchestration: Deploying Polyglot Microservices from Bare-Metal to Amazon EKS [Helmfile, ALB Ingress Controller, EBS CSI Driver, DynamoDB, AWS SQS, SNS, Lambda, Fluent Bit, CloudWatch]¶

Overview¶

Although this application was originally built as a demo project by the AWS Containers team specifically for Amazon EKS, I set out to deploy it across three entirely different target environments: an ephemeral bare-metal cluster, a persistent bare-metal cluster, and finally, Amazon EKS.

It models a real-world heterogeneous stack: five independent services, five different runtimes, and five different persistence backends.

The Approach (Decoupled Values):

AWS provided the base Helm charts for the microservices. However, rather than manually editing their charts to accommodate my three target environments, I left the upstream charts completely untouched. Instead, I navigated into each service's src/ directory and authored my own environment-specific values-*.yaml overrides.

Bare-Metal First:

Because this application is heavily AWS-oriented, deploying it to a bare-metal Kubernetes cluster required explicitly severing its cloud dependencies. Using the custom values-*.yaml overrides, I systematically ripped out AWS DynamoDB (Cart), AWS SQS/SNS (Orders), AWS ElastiCache (Checkout), and the ALB Ingress Controller (UI). I rewired the microservices to run entirely on local, containerized alternatives—such as a local DynamoDB container, RabbitMQ/in-memory messaging, local Redis, and standard NodePorts. This successfully proved the application's portability completely outside of the AWS ecosystem.

Helmfile Orchestration:

While the bare-metal deployment was successful, it required executing five separate helm install commands in strict dependency order. To eliminate this manual toil, I engineered a Helmfile orchestration layer. This allowed me to deploy the entire stack—with databases provisioning before microservices—via a single command.

For the bare-metal validations, I authored and applied the following two Helmfile targets:

# Bare-metal — ephemeral
helmfile -f helmfile/helmfile-baremetal-ephemeral.yaml apply

# Bare-metal — persistent
helmfile -f helmfile/helmfile-baremetal-persistent.yaml apply

The Final Target (Amazon EKS):

With the Helmfile orchestration successfully automating the deployments, I returned the application to its original intended target: Amazon EKS. Because deploying to EKS required complex, deep native AWS infrastructure provisioning (ALB, EBS CSI, DynamoDB, SQS, SNS, Lambda, and CloudWatch), the vast majority of this runbook is dedicated exclusively to documenting this final, production-grade deployment.

Application Architecture¶

Multi-Environment Configuration¶

Three Helmfile Targets¶

Rather than running five separate helm install commands in the right order every time, I authored three Helmfile configurations - one per deployment target - each declaring all five releases with explicit dependency ordering via needs::

Layered values-*.yaml Override Matrix¶

This matrix demonstrates how environment-specific overrides were injected across the three deployment targets while leaving the base values.yaml untouched.

Amazon EKS Architectural Highlights¶

The rest of this document exclusively covers the third platform: Amazon EKS. The following highlights define its deep cloud-native integration:

1. Native AWS Observability: Because this is an AWS-centric deployment, I explicitly abandoned the ELK stack and Prometheus. I deployed Fluent Bit to stream logs natively into CloudWatch Container Insights, demonstrating deep alignment with the AWS managed ecosystem.
2. Event-Driven Order Notifications: I engineered a serverless, event-driven pipeline where AWS SQS queues capture order events. These events immediately trigger an AWS Lambda function that executes custom business logic to publish order confirmations to an AWS SNS topic for direct email delivery.
3. Bypassing Service Control Policies (SCPs): As the lab environment blocked iam:PassRole required by EKS Managed Node Groups, I bypassed this restriction by engineering self-managed worker nodes via custom AWS CloudFormation templates.

EKS Deployment Phases¶

As established in the Final Target overview, running a single helmfile apply against Amazon EKS is only possible after an extensive amount of native AWS infrastructure has been provisioned and configured.

To document this deep cloud integration, I have broken the entire EKS rollout down into a rigorous 10-Phase deployment lifecycle. The runbook pages below detail every step from the first IAM role creation to the final end-to-end TLS validation.

Key Decisions¶

Architectural and engineering decisions made across the deployment phases to accommodate environment constraints and optimize the platform.

Infrastructure and Lab Constraints¶

• Simplified VPC Creation via eksctl. Instead of explicitly defining a VPC and its private subnets in the cluster manifest, I leveraged eksctl's automated provisioning. This kept the infrastructure code simple while still ensuring the EKS cluster was securely placed into private subnets. (Phase 1-3)
• Dev Machine over Bastion for Administration. Instead of installing kubectl, helmfile, and other DevOps tools onto a bastion host, I administered the cluster directly from my SilverStack Dev Machine which had the complete toolchain pre-installed. (Phase 1-3)
• Bastion Host for Node Troubleshooting. While cluster administration was done from the Dev Machine, a bastion host was still utilized to SSH into the self-managed worker nodes for troubleshooting. When nodes initially failed to join the cluster due to an incorrect authentication mode, SSH access via the bastion proved critical for identifying the issue before tearing down and recreating the node stack with the correct parameters. (Phase 1-3)
• Authentication Mode: API_AND_CONFIG_MAP. I explicitly set the authentication mode to API_AND_CONFIG_MAP (the modern 2023 standard using EKS Access Entries alongside the traditional aws-auth ConfigMap), bypassing legacy restrictions and ensuring robust access control. (Phase 1-3)
• IAM Roles via Terraform, not eksctl. The KodeKloud lab user lacks iam:PassRole permissions. Allowing eksctl to automatically create roles during cluster creation would fail. Roles were pre-provisioned via Terraform and explicitly passed to eksctl. (Phase 1-3)
• Self-Managed Nodes over Managed Node Groups. EKS Managed Node Groups also trigger the blocked iam:PassRole permission. To bypass this, worker nodes were deployed manually using an AWS CloudFormation template. (Phase 1-3)
• Delayed OIDC Association. OIDC was disabled in cluster.yaml (withOIDC: false) because it also triggers permission failures during cluster creation. It was associated manually via eksctl utils after the control plane was up. (Phase 1-3)

Deployment Orchestration¶

• Why Helmfile instead of ArgoCD? Because this was my first deep-dive into microservices, I intentionally prioritized mastering deployment orchestration, multi-environment configurations, and release dependencies natively via Helmfile before abstracting the workflow behind a GitOps controller (which I subsequently implemented in my next project).
• Single Runtime Override. Because of the layered values*.yaml strategy, all infrastructure configurations were pre-defined. The only manual edit required during the entire deployment phase is injecting the dynamically generated ACM Certificate ARN into the UI service's ingress values file right before executing the grand helmfile -f command.

Cloud-Native Integrations¶

• Offloading State to AWS Managed Services. Rather than running databases inside the cluster, the EKS Helmfile configuration binds the microservices to DynamoDB (Cart) and SQS/SNS (Orders) via IAM Roles for Service Accounts (IRSA). (Phase 6)
• Shared ALB via Ingress Group. The UI, Prometheus, and Grafana all share a single Application Load Balancer using alb.ingress.kubernetes.io/group.name: ecom-eks. The ALB routes traffic based on the Host header, eliminating the cost of multiple load balancers. (Phase 8-9)
• gp3 as Default StorageClass. gp2 was patched out and gp3 was set as the default StorageClass for the EBS CSI driver, providing a cheaper and more performant storage baseline for the stateful databases (MySQL, PostgreSQL). (Phase 4-5)

Screenshots¶