Cloud Run Services

Technical details of the Cloud Run services that support the Titanbay platform

Tyk API Gatwway

Tyk API Gateway is an open-source API Gatway that is used to manage and aggregate API service endpoints.

We currently deploy Tyk API Gateway to the engi-vpc-*-eu projects in the europe-west2 region. Each deployment proxies to the GCP API Gatway instances that are managed by service domain repos. Deployment of this is managed through Atlantis, and the terraform module for each deployment is in the infra-terraform/gcp/modules/deployment-environment/external-lb directory.

Tyk is configured by registering OpenAPI v3 specifications via the Tyk Management API. Each API endpoint in the specification is mapped to a service domain in a GCP API Gateway. The OAS spec for Tyk requires the use of a custom extension, x-tyk-api-gateway. The Tyk extension informs Tyk via the management API which service domain API Gateway to proxy to.

Tyk management API expects OAS v3 specifications however service domain GCP API Gateway instances are configured using OAS v2 specifications. To bridge this gap we deploy a service called tykctl that translates OAS v2 to OAS v3, and also handles the creation of the x-tyk-api-gateway extension in the OAS v3 spec to be submitted to Tyk management API

Tyk API Gateway does not manage AuthN or AuthZ, this is managed by the service domains. In that regard Tyk API Gateway is a reverse/pass-through proxy that manages rate limiting, header exclusion/addition, request/response transformations and so forth (if configured to do so).

The ingress path to a service domain HTTP endpoint is as follows:

sequenceDiagram
    autonumber
    User Agent->>Tyk API Gateway: GET|POST|PUT|PATCH|DELETE /v1/{api_group}/{resource}/...
    Tyk API Gateway->>GCP API Gateway: HTTP Request
    GCP API Gateway->>GCP API Gateway: AuthN
    GCP API Gateway->>Cloud Function: HTTP Request
    Cloud Function-->>User Agent: HTTP Response
    Note over User Agent,Cloud Function: HTTP Response egresses through the ingress path

tykctl service

Tykctl is a restful API service that accepts requests to translate OAS v2 to OAS v3, and to submit the OAS v3 to the Tyk Management API. The service is deployed to the engi-vpc-*-eu projects in the europe-west2

The service is a fairly simple Go programme that is built into a container image using the Ko image builder. The source for the service is found under infra-services/cmd/tykctl, and the packages it consumes are found in infra-services/pkg. The reason for choosing Go is primarily familiarity with the OpenAPI packages and tooling consumed by the tykctl code, however the following secondary reasons are also considered:

• Simplicity of deployment
• Google Cloud go-sdk leverages gRPC with protobuf as the underlying wire transport which makes maintenence less of a burden as GCP APIs evolve

tykctl -> Tyk Management API interaction:

sequenceDiagram
    autonumber
    GitHub Action->>Tykctl: POST /{project_id}/{region}/{api_id}/{gateway_id}
    Tykctl->>GCP: Get API Gateway and API Gateway config
    Tykctl->>Tykctl: Validate project environment
    Tykctl->>Tykctl: Validate OAS v2
    Tykctl->>Tykctl: Convert OAS v2 to OAS v3
    Tykctl->>Tykctl: Add x-tyk-api-gateway extension
    Tykctl->>Tyk API Gatway: POST|PUT /tyk/apis/oas
    Tykctl->>Tyk API Gatway: GET /tyk/reload

Authentication

Authentication for calls to tykctl are through GCP IAM, run invoker role in the project is needed to call the service.

Development

The tykctl service interacts with GCP, so you’ll need to have gcloud authenticated to proceed. Additionally a few environment variables are needed to run the service locally, check the .envrc.example file for the required variables.

The programme code layout is roughly as follows:

• cmd/tykctl - main package
  • main.go - entry point
  • handlers.go - http handlers
• pkg/gcp/apigatway - GCP API Gateway client
  • api_gateway.go - gcp client and top level transformation function
• pkg/gcp/resourcemanager - GCP Resource Manager client
  • project.go - gcp client for projects
• pkg/tyk - tyk management API client
  • client.go - client for tyk management API
  • openapi_spec.go - openapi spec model and lower level OpenAPI spec transformation

There’s no hot-reload support for development (yet) but to compile and run in dev mode, you can use the following commands:

make run-dev

Assuming you have Go & Ko installed, you can compile the service and publish a new container image with the following command:

make container-image

Deployment

tykctl is deployed to Cloud Run via GitHub actions, the workflow is found in .github/workflows/tykctl-cloud-run.yaml. In infra-service repo. The base template for the cloud run instance is under cmd/tykctl//manifests/tykctl/base/tyk-service-template.yaml. In the GitHub actions workflow a Kustomization step is used to apply an overlay to the base for each environment.

A sample cloud run template for tykctl (not necessarily current) is as follows:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: tykctl-service
  annotations:
    run.googleapis.com/ingress: all
    run.googleapis.com/launch-stage: BETA
  labels:
    cloud.googleapis.com/location: europe-west1

spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/maxScale: '1'
        autoscaling.knative.dev/minScale: '1'
        run.googleapis.com/cpu-throttling: 'true'
        run.googleapis.com/execution-environment: gen2
        run.googleapis.com/network-interfaces: '[{"network":"engi-vpc-dev-europe-west1","subnetwork":"engi-vpc-vpcd-dev-europe-west1-1"}]'
        run.googleapis.com/vpc-access-egress: all-traffic
    spec:
      containerConcurrency: 80
      containers:
        - env:
            - name: TYK_API_KEY
              valueFrom:
                secretKeyRef:
                  name: TYK_GW_SECRET
                  key: latest
          image: europe-west1-docker.pkg.dev/tb-infra-management-395208/infra/tykctl:latest
          ports:
            - containerPort: 8080
              name: http1
          resources:
            limits:
              cpu: 1000m
              memory: 512Mi
          startupProbe:
            failureThreshold: 1
            periodSeconds: 240
            tcpSocket:
              port: 8080
            timeoutSeconds: 240
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100