Gateway populated with incorrect address and blocked from deletion if a cluster has multiple gateways.

[chetanatole]

When multiple gateways are deployed in an EKS cluster, each with their respective associated HTTPRoutes, the status of a gateway object is getting populated with the address of an HTTPRoute to which it is not actually associated and the deletion of the gateway object is blocked, even after the correctly associated HTTPRoute has been deleted.

Steps to reproduce the behavior:

1. Deploy the gateway controller as mentioned here using helm chart. Set the version to 1.1.2
2. Upgrade the controller config to set default service network

helm upgrade gateway-api-controller \
oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart \
--version=v1.1.2 \
--reuse-values \
--namespace aws-application-networking-system \
--set=defaultServiceNetwork=my-hotel

3. Edit the controller deployment to add ENABLE_SERVICE_NETWORK_OVERRIDE env variable.
4. Create 2 namespaces, deploy sample pod with associated service

k create ns ns-a
k create ns ns-b

k run test-pod --image=nginx -n ns-a
k run test-pod --image=nginx -n ns-b

k expose pod test-pod --name=test-svc-b -n ns-b --port=80 --target-port=80
k expose pod test-pod --name=test-svc-a -n ns-a --port=80 --target-port=80

5. Create Gateway and HTTPRoutes for the app in ns-a

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-a
  namespace: ns-a
spec:
  gatewayClassName: amazon-vpc-lattice
  listeners:
    - name: listener1
      protocol: HTTP
      port: 80

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: route-a
  namespace: ns-a
spec:
  parentRefs:
    - kind: Gateway
      name: gateway-a
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: test-svc-a
          port: 80

k get gateway,httproute -n ns-a
NAME                                          CLASS                ADDRESS   PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/gateway-a   amazon-vpc-lattice             True         94s

NAME                                          HOSTNAMES   AGE
httproute.gateway.networking.k8s.io/route-a               84s

The httproute gets the domain name in annotations

k describe httproute.gateway.networking.k8s.io/route-a -n ns-a
Name:         route-a
Namespace:    ns-a
Labels:       <none>
Annotations:  application-networking.k8s.aws/lattice-assigned-domain-name: route-a-ns-a-02dd5c0852f615d5e.7d67968.vpc-lattice-svcs.us-east-2.on.aws
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
...

6. Create Gateway and HTTPRoutes for the app in ns-b

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-b
  namespace: ns-b
spec:
  gatewayClassName: amazon-vpc-lattice
  listeners:
    - name: listener1
      protocol: HTTP
      port: 80

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: route-b
  namespace: ns-b
spec:
  parentRefs:
    - kind: Gateway
      name: gateway-b
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: test-svc-b
          port: 80

The created gateway 'gateway-b' is getting address of route 'route-a' which are not related.

k get gateway,httproute -A
NAMESPACE   NAME                                          CLASS                ADDRESS                                                                    PROGRAMMED   AGE
ns-a        gateway.gateway.networking.k8s.io/gateway-a   amazon-vpc-lattice                                                                              True         13m
ns-b        gateway.gateway.networking.k8s.io/gateway-b   amazon-vpc-lattice   route-a-ns-a-02dd5c0852f615d5e.7d67968.vpc-lattice-svcs.us-east-2.on.aws   True         2m47s

NAMESPACE   NAME                                          HOSTNAMES   AGE
ns-a        httproute.gateway.networking.k8s.io/route-a               13m
ns-b        httproute.gateway.networking.k8s.io/route-b               2m28s

7. Now, if we want to delete the gateway 'gateway-b' for which we will first delete the actual associated httproute which is 'route-b', the gateway deletion is blocked.

k delete httproute route-b -n ns-b
httproute.gateway.networking.k8s.io "route-b" deleted
[root@chetan-swift01 gateway]# k get gateway,httproute -A
NAMESPACE   NAME                                          CLASS                ADDRESS                                                                    PROGRAMMED   AGE
ns-a        gateway.gateway.networking.k8s.io/gateway-a   amazon-vpc-lattice                                                                              True         20m
ns-b        gateway.gateway.networking.k8s.io/gateway-b   amazon-vpc-lattice   route-a-ns-a-02dd5c0852f615d5e.7d67968.vpc-lattice-svcs.us-east-2.on.aws   True         9m22s

NAMESPACE   NAME                                          HOSTNAMES   AGE
ns-a        httproute.gateway.networking.k8s.io/route-a               20m
[root@chetan-swift01 gateway]# k delete gateway gateway-b -n ns-b
gateway.gateway.networking.k8s.io "gateway-b" deleted
# deletion is stuck

8. After deleting httproute 'route-a' from 'ns-a' namespace, the gateway 'gateway-b' is deleted.

k delete httproute route-a -n ns-a
httproute.gateway.networking.k8s.io "route-a" deleted

k get gateway,httproute -A
NAMESPACE   NAME                                          CLASS                ADDRESS   PROGRAMMED   AGE
ns-a        gateway.gateway.networking.k8s.io/gateway-a   amazon-vpc-lattice             True         29m

Expected behavior

1. Gateway's address should be domain name from associated httproutes only
2. Gateway deletion should work if all httproutes associated with it are deleted.

Controller logs: controller_logs.log