Cluster Topology

This page describes the ArgoCD deployment architecture across Titanbay’s Kubernetes clusters.

Hub-and-Spoke Architecture

Titanbay operates a hub-and-spoke model where a central ArgoCD instance on the Infra Management Cluster manages deployments to all clusters, including itself and the tb-platform environment clusters.

The Clusters

Infra Management Cluster

The infra management cluster runs the primary ArgoCD instance in High Availability (HA) mode. This ArgoCD instance is responsible for:

1. Managing itself - The ArgoCD application syncs its own configuration
2. Managing infrastructure services - Atlantis, Netbox, monitoring stack, etc.
3. Managing tb-platform clusters - Via ApplicationSets that deploy to remote clusters
4. Managing ArgoCD on tb-platform clusters - Each spoke cluster gets its own ArgoCD via Helm

ArgoCD HA Deployment

The management cluster runs ArgoCD using a standalone HA manifest (argocd-ha-3.2.1.yaml), not a Helm chart. This provides:

• Multiple replicas of critical components
• Redis HA for caching and state
• Pod Disruption Budgets for safe updates
• Custom resource limits tuned for production workloads

Applications Managed

The hub ArgoCD manages approximately 20+ applications on the infra management cluster itself:

TB Platform Clusters

Each tb-platform cluster (dev, qa, prod) runs its own ArgoCD instance deployed via Helm chart. These instances are managed by the hub ArgoCD, creating a cascading management structure.

How It Works

1. Hub ArgoCD deploys an Application that references the Helm chart configuration for each spoke
2. The Helm chart installs ArgoCD on the tb-platform cluster
3. The spoke ArgoCD manages workloads local to that cluster

ApplicationSets for TB Platform

The hub uses ApplicationSet resources to dynamically generate applications for each environment:

# Example: tb-platform-init ApplicationSet
generators:
  - list:
      elements:
      - environment: dev
        cluster: https://10.64.128.34
      - environment: qa
        cluster: https://10.64.128.50
      - environment: prod
        cluster: https://10.64.128.66

Key ApplicationSets include:

Spoke ArgoCD Configuration

The tb-platform ArgoCD instances are deployed via Helm with configuration in k8s/infra-services/argocd/tb-platform/:

• Base configuration (base/argocd-helm.yaml): Common Helm values including OIDC SSO, RBAC policies
• Environment overlays: Domain-specific patches for each environment
  • Dev: argocd-dev.nessie-chimera.ts.net
  • QA: argocd-qa.nessie-chimera.ts.net
  • Prod: argocd-prod.nessie-chimera.ts.net

Infra Build Services Cluster

infra-build-services (GCP project tb-engi-build-services-eu, internal IP 10.64.128.83) is a fifth cluster with a hybrid management model, distinct from both the infra-management hub and the tb-platform spokes:

• Cluster and edge services are Hub-managed remotely, deployed via the tb-engi-build-services-init-services and tb-engi-build-services-init-resources Applications on the hub. Hub-managed resources include cluster services (cert-manager, envoy-gateway, tailscale-operator, external-secrets, agentgateway), cluster-scoped controllers/CRDs (such as agent-sandbox), and Play edge and security resources (Hub init-resources manages Play Service, Certificate, NetworkPolicy, Secrets, Gateway, routes, policies, and admission, including the Gateway, HTTPRoute, GCPBackendPolicy, and ExternalSecrets in play-gateway/).
• Workload applications (bufo-build/bufo-system and delegated play workloads) are managed by a dedicated spoke ArgoCD instance, deployed via a Hub-managed Application installing the argo-cd Helm chart (cluster-init/base/cluster-services/infra-build-services/argocd.yaml), reachable at argocd-build.nessie-chimera.ts.net via the Tailscale Ingress in cluster-resources/infra-build-services/argocd/ingress.yaml. The spoke manages only delegated Play workload resources (ServiceAccount, ConfigMap, and Deployment), while Hub init-resources manages Play edge and security resources (Service, Certificate, NetworkPolicy, Secrets, Gateway, routes, policies, and admission).
• Unlike tb-platform (one spoke per environment/cluster), this single spoke manages both dev and prod namespaces (bufo-build-dev, bufo-system-dev, bufo-build-prod, bufo-system-prod, play-dev, play-prod) within the one cluster. It uses dedicated AppProjects (bufo-build, play, and agent-gateway) and bootstrap Applications in cluster-init/overlays/infra-build-services/init-resources/argocd-projects/.
• The cluster-scoped agent-sandbox controller/CRDs (from bufo-build’s own k8s/gke/agent-sandbox) remain Hub-managed, not delegated to the spoke. Spoke AppProjects restrict cluster-scoped resources according to their workload boundaries: play and agent-gateway deny all cluster resources (clusterResourceWhitelist: []), while bufo-build permits only PriorityClass for its sandbox balloon pod scheduling. The Hub’s build-services AppProject retains a broad cluster resource whitelist (group: '*', kind: '*') because it bootstraps infrastructure CRDs, namespaces, and cluster controllers.
• Sync policies differ by workload and environment:
  • play-dev enables automated sync with automated prune and self-heal (automated.prune: true, automated.selfHeal: true).
  • play-prod uses manual sync (automated sync disabled) to preserve controlled release gates for production.
  • bufo-system-dev enables automated sync with self-heal to apply image tag overrides from ArgoCD Image Updater.
  • The remaining Bufo bootstrap Applications (bufo-build-dev, bufo-build-prod, bufo-system-prod) and cluster-services currently operate with manual sync while workload migration is managed by hand.
• The spoke’s repo credentials for Titanbay/bufo-build (cluster-resources/infra-build-services/argocd/bufo-build-repo-credentials.yaml) depend on github-app-install-id and github-private-key in this project’s Secret Manager (see that directory’s README for provisioning steps).

Cross-Cluster Authentication

ArgoCD on the management cluster authenticates to remote clusters using:

1. Cluster CA certificates stored as Secrets (e.g., tb-platform-cluster-ca.yaml)
2. Service Account tokens for Kubernetes API access
3. Workload Identity for GCP resource access

Network Connectivity

All clusters communicate over private networking:

• Clusters are in the same GCP VPC or connected via VPC peering
• ArgoCD uses internal cluster IPs (10.64.x.x range)
• Tailscale provides secure access for human operators

Sync Strategies

Different sync strategies are applied based on the application type:

The conservative approach for the argocd application itself prevents accidental destructive changes.