Apigee X Eval Org Stuck at 502? Check These 2 Things First

Quick-fix reference. For the full debugging story behind this, see: The 502 That Wouldn’t Die

Symptom

Your Apigee X evaluation org shows fully provisioned:

• Console wizard: all 4 setup steps green ✅
• organizations.get API: "state": "ACTIVE" ✅
• instances.list API: "state": "ACTIVE", real host/port ✅
• Environment attached, proxy deployed, envgroup hostname bound — all clean ✅

And yet every request — even to a brand-new proxy — returns:

HTTP/2 502
Error: Server Error
The server encountered a temporary error and could not complete your request.

Enter fullscreen mode Exit fullscreen mode

Waiting longer doesn’t fix it. Deploying a different proxy doesn’t fix it.

First: confirm this is actually your bug

gcloud compute backend-services list
gcloud compute backend-services get-health apigee-proxy-backend --global

Enter fullscreen mode Exit fullscreen mode

If this shows healthState: UNHEALTHY on the apigee-proxy-* instances, keep reading — this is the load-balancer layer that sits between the external HTTPS LB and your actual Apigee runtime, and it’s separate from Apigee’s own control plane. That’s why everything above reports “ACTIVE” while requests still 502: Apigee’s config is correct, but the forwarding instances behind the LB aren’t actually serving traffic.

Cause #1: Missing service account on the instance template

Check the boot log of one of the unhealthy instances:

gcloud compute instances get-serial-port-output <INSTANCE_NAME> --zone=<ZONE> | tail -60

Enter fullscreen mode Exit fullscreen mode

Look for:

Instance has service account: false, ...
Failed to download from GCS: ... credentials: cannot fetch token ...
Trying unauthenticated download

Enter fullscreen mode Exit fullscreen mode

Confirm it:

gcloud compute instance-templates describe apigee-proxy-<REGION> \
  --format="yaml(properties.serviceAccounts)"

Enter fullscreen mode Exit fullscreen mode

If this prints null, the template has no service account attached, so the VM can never authenticate to Cloud Storage to pull its real startup script.

Fix — clone the template with a service account attached, then roll the MIG onto it:

# Get every field from your existing template first so you replicate it exactly:
gcloud compute instance-templates describe apigee-proxy-<REGION> --format=yaml

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=<PROJECT_NUMBER>[email protected] \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>

Enter fullscreen mode Exit fullscreen mode

Match every field from your describe --format=yaml output — machine type, disk, network, tags, and especially the scheduling block. --preemptible, --no-restart-on-failure, and --maintenance-policy=TERMINATE must be specified together or gcloud rejects the combination.

Cause #2: Blank ENDPOINT metadata

Even after fixing the service account, health checks can still fail. These forwarding VMs don’t run a proxy application themselves — they install an iptables DNAT rule redirecting incoming port-443 traffic to your real Apigee runtime instance’s internal IP. That IP comes from an instance metadata key called ENDPOINT.

Check it from inside an instance:

gcloud compute ssh <INSTANCE_NAME> --zone=<ZONE>
curl -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ENDPOINT"

Enter fullscreen mode Exit fullscreen mode

If this returns nothing, that’s the second bug. Get your runtime instance’s real internal IP:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/<ORG>/instances"

Enter fullscreen mode Exit fullscreen mode

Look for the "host" field in the response (e.g. 10.51.204.98).

Fix — patch the metadata, then force each VM to re-run its startup script (no reboot needed):

gcloud compute instances add-metadata <INSTANCE_NAME> \
  --zone=<ZONE> --metadata=ENDPOINT=<RUNTIME_INTERNAL_IP>

gcloud compute ssh <INSTANCE_NAME> --zone=<ZONE>
sudo google_metadata_script_runner startup

Enter fullscreen mode Exit fullscreen mode

Repeat for every instance in the group. Verify the NAT rule landed:

sudo iptables -t nat -L -n -v
# Look for: DNAT tcp dpt:443 to:<RUNTIME_INTERNAL_IP>

Enter fullscreen mode Exit fullscreen mode

Confirm it’s fixed

gcloud compute backend-services get-health apigee-proxy-backend --global

Enter fullscreen mode Exit fullscreen mode

Both instances should now show healthState: HEALTHY. Then:

curl "https://<YOUR_HOSTNAME>.nip.io/hello-world"

Enter fullscreen mode Exit fullscreen mode

You should get a real response instead of the 502 page.

This is one specific failure mode out of many possible causes of a 502 on Apigee X — always confirm the backend health check first before assuming this applies to you. Full context and the debugging process that led here: The 502 That Wouldn’t Die.