Kubernetes Migration: From Ingress-NGINX to Envoy Gateway

A practical look at our migration from Ingress-NGINX to the Kubernetes Gateway API using Envoy. Learn how we tackled real-world challenges like OIDC, CORS, and traffic policies.

Raphaël Parrée
Raphaël ParréePublished on

Our Ingress Migration (to Envoy Gateway)

In this blog article, we would like to share some of the challenges we faced and the solutions we implemented during our migration from the deprecated Ingress-NGINX to Envoy Gateway. We performed this migration over a period of 10 days in April 2026. We will not delve into the selection process or detail why we ultimately chose Envoy Gateway. In short, our decision was largely driven by our extensive prior experience with Envoy via Istio, as well as our use of Envoy as sidecars in the vanilla Podman and Docker Compose services we maintain.

Important: This article does not serve as an introduction to the Kubernetes Gateway API or Envoy itself. Instead, it illustrates the specific issues we encountered and how we resolved them. While we have included the relevant YAML definitions, they are provided without detailed explanations. Ultimately, this article is not intended as a step-by-step tutorial, but rather as a practical collection of real-world problems and solutions from our migration journey.

Our cluster

We operate a self-managed "on-premises" Kubernetes cluster, which is hosted on Hetzner cloud servers. This cluster serves as the backbone for our public website, various internal applications, and, most importantly, our interactive training courses. All of our course materials are delivered directly to students via their web browsers.

Prior to the migration, we managed approximately 100 Ingress objects to route our north-south traffic using Ingress-NGINX. These configurations catered to a diverse set of requirements and generally fell into four distinct categories:

  • Public-facing applications: Front-ends and back-ends requiring standard reverse proxy logic, such as Cross-Origin Resource Sharing (CORS), redirects, and path rewrites.
  • Internal, bespoke applications: Self-built front-ends and back-ends secured primarily through OpenID Connect (OIDC) with strict, role-based authorisation.
  • Back-office infrastructure: Off-the-shelf, open-source software utilised by our team. The majority of these tools are deployed via Helm.
  • Course materials: Highly structured environments that rely on a significantly more elaborate and granular authentication and authorisation scheme (which is also OIDC-based).

Fortunately, TLS certificate management was one area we did not have to worry about during the migration. All our Let's Encrypt certificates are automatically provisioned and renewed using cert-manager alongside the cert-manager-webhook-hetzner plugin. Because we utilise the DNS-01 challenge for domain validation, we did not rely on Ingress-NGINX for HTTP-01 challenge routing. This separation of concerns made the transition slightly smoother.

Gateway Controller and CRDs

We utilise GitOps for managing our cluster, specifically Flux CD. When making the Gateway API Custom Resource Definitions (CRDs) and the Envoy Gateway Controller available to our cluster, we initially contemplated separating their installation. This would typically involve distinct OCIRepository and HelmRelease objects, along with a cluster-level GitRepository for the CRDs. Ultimately, we opted for a more straightforward approach: installing the CRDs directly as part of a single Helm release.

Below is the OCIRepository and HelmRelease manifest for the Envoy Gateway. You will notice that we explicitly enabled the Envoy Patch Policy (enableEnvoyPatchPolicy: true) in the Helm values. This is a crucial extension that allows us to modify the underlying Envoy Proxy configuration directly when the standard Gateway API does not cover our specific use cases. We will explore how we utilised this feature later in the article.

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: envoy-gateway-chart
  namespace: flux-system
spec:
  interval: 1h
  url: oci://docker.io/envoyproxy/gateway-helm
  ref:
    tag: 1.7.3
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: envoy-gateway
spec:
  releaseName: eg
  targetNamespace: envoy-gateway-system
  interval: 1h
  chartRef:
    kind: OCIRepository
    name: envoy-gateway-chart
  upgrade:
    crds: CreateReplace
  install:
    createNamespace: true
  values:
    config:
      envoyGateway:
        extensionApis:
          enableEnvoyPatchPolicy: true

Proxy and GatewayClass

Before configuring the GatewayClass, we needed to define a custom EnvoyProxy resource. If you do not explicitly reference an EnvoyProxy in either your GatewayClass or Gateway, the controller falls back to its built-in defaults. However, we had two specific requirements that necessitated a custom configuration:

  1. Fixed NodePorts: Because we are using Hetzner cloud servers, we needed to expose our Gateway via specific NodePorts (just as we did with Ingress-NGINX). To achieve this, we applied a StrategicMerge patch to the underlying Service object generated by the Gateway Controller.
  2. Access Log Filtering: We wanted to eliminate the log noise generated by our separately hosted uptime monitor, Uptime Kuma, as well as Hetzner's health checks.

Additionally, because we rely on NodePorts and only run a few replicas of the Envoy Proxy, we had to modify the externalTrafficPolicy. Envoy Gateway's default is Local (which preserves the client source IP but requires a proxy replica to be present on the node receiving the traffic). By changing it to Cluster, we ensure traffic is reliably routed to available proxy pods regardless of which node receives the initial request, maintaining high availability.

Here is the resulting EnvoyProxy definition. Notice the Common Expression Language (CEL) syntax in the telemetry section, which we used to elegantly filter out specific user-agent strings before they reach standard output.

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        patch:
          type: StrategicMerge
          value:
            spec:
              externalTrafficPolicy: Cluster
              ports:
                - name: http
                  nodePort: 30080
                  port: 80
                - name: https
                  nodePort: 30443
                  port: 443
  telemetry:
    accessLog:
      settings:
        - matches:
            - "!request.headers['user-agent'].contains('Uptime-Kuma') && !request.headers['user-agent'].contains('HCLB-HealthCheck')"
          sinks:
            - type: File
              file:
                path: /dev/stdout

The GatewayClass is then defined as follows, using the parametersRef field to link it to the custom EnvoyProxy we just created.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: proxy-config  # The custom proxy defined above
    namespace: envoy-gateway-system

Proxy Protocol

To ensure the original client IP addresses are preserved and passed through via the X-Forwarded-For header, our Hetzner Load Balancer is configured to use the PROXY protocol. In Envoy Gateway, this behaviour is managed by attaching a ClientTrafficPolicy to the Gateway.

By setting proxyProtocol.optional: true, we instructed Envoy to accept connections both with and without the PROXY protocol header. This is particularly useful for allowing direct, internal traffic to hit the proxy without failing, whilst still correctly parsing the header when traffic arrives via the external Load Balancer.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-proxy-protocol
  namespace: envoy-gateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: main-gateway
  proxyProtocol:
    optional: true

Gateway

We operate two primary top-level domains (*.edc4it.com and *.course-delivery.com). For simplicity, we decided to handle all traffic using a single Gateway object. In retrospect, we probably should have separated these domains into two distinct Gateways. Nonetheless, this consolidated setup was easier to manage initially and meant we only required a single Hetzner Load Balancer.

Below is the resulting Gateway object, which references the custom GatewayClass we defined earlier. It specifies two HTTPS listeners, one for each wildcard domain.

Crucially, we also configured allowedRoutes.namespaces.from: All for both listeners. By default, the Gateway API restricts route attachments to resources residing in the same namespace as the Gateway itself. Modifying this setting permits HTTPRoute objects deployed across our various application namespaces to dynamically attach themselves to these centralised listeners.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: https-edc4it
      protocol: HTTPS
      port: 443
      hostname: "*.edc4it.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: edc4it-tls
      allowedRoutes:
        namespaces:
          from: All
    - name: https-course
      protocol: HTTPS
      port: 443
      hostname: "*.course-delivery.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: course-delivery-tls
      allowedRoutes:
        namespaces:
          from: All

With the installation of the CRDs and Controller we could now start to define the Gateway object (s)

HTTPRoute Migration Plan

There are automated tools available that promise to perform the migration from Ingress to the Gateway API for you, including one provided by Envoy itself. However, we decided against using them. We knew that a manual migration would give us the perfect opportunity to properly restructure and optimise a significant portion of our north-south traffic.

With Ingress, annotations apply globally to all rules within a single object. Because we frequently had slightly different requirements per URL segment (e.g., specific security policies, CORS configurations, or body size limits), we were often forced to create many separate Ingress objects for what was logically a single service. The Gateway API's more granular approach allows us to consolidate these configurations elegantly.

To ensure a smooth transition with zero downtime, we adopted a parallel running strategy. We provisioned a new Hetzner Load Balancer specifically for the Envoy Gateway while keeping the existing Ingress-NGINX controller and its Load Balancer fully operational. A few days prior to the migration, we significantly lowered the Time-To-Live (TTL) on our DNS records to ensure the eventual switchover would propagate quickly.

For the individual routes, we followed a methodical, step-by-step process:

  • Retain the existing Ingress object.
  • Create an equivalent (or improved) HTTPRoute object pointing to the new Gateway.
  • Verify the routing through the new Gateway using local DNS overrides. For example:
    • Using CLI tools like cURL, HTTPie, or xh with the --resolve flag: 
      $ curl -v --resolve www.edc4it.com:443:46.225.40.55 https://www.edc4it.com
    • Or using a Chromium-based browser with the --host-resolver-rules option to test the UI locally: 
      $ chromium-browser --host-resolver-rules='MAP management.course-delivery.com 46.225.40.55' &!
    • For more complex reverse proxy logic (such as OIDC flows), we ran automated Python tests (more on this later).

Once every route had been successfully duplicated and verified on the Gateway:

  • We updated our DNS records to point to the new Envoy Gateway Load Balancer.
  • We waited two days to ensure full DNS propagation globally before finally deleting the old Ingress objects and decommissioning the NGINX controller.

The remainder of this article details the specific migration scenarios we encountered.

Simple Routes

We had a couple of Ingress objects that were straightforward enough to be almost mechanically migrated to the Gateway API. By "simple", we mean standard routes that merely map a hostname and a path prefix to a backend service without any complex reverse proxy logic.

Even with these basic routes, we immediately noticed some major architectural improvements:

  • Centralised TLS Management: We no longer needed to use tools like Reflector to copy TLS certificates to every namespace that contained an Ingress. (With standard Ingress, an Ingress object must reference a Secret residing in its own namespace for TLS). The Gateway API handles TLS centrally at the Gateway listener level.
  • Consolidated Hostnames: Several of our routes served multiple hostnames. With Ingress, we had to duplicate the entire rule block (often relying on YAML anchors to keep the file somewhat DRY). HTTPRoute natively supports an array of hostnames, making the configuration significantly cleaner.

The only minor friction we encountered was with backend ports. With Ingress, we frequently routed traffic to Kubernetes named ports. Currently, the Gateway API does not support named ports; it requires an explicit integer for the backendRef port. This meant we had to look up and hardcode the actual port numbers in our HTTPRoute definitions.

Additionally, we had to write custom HTTPRoute objects for third-party Helm charts that did not yet natively support the Gateway API. For these, we disabled their built-in ingress generation (usually via ingress.enabled: false) and added a separate HTTPRoute manifest to the Kustomization containing the HelmRelease.

Below is an example of a simple HTTPRoute for one such service (Kutt). Notice how the parentRefs elegantly links this route back to our central main-gateway (which resides in a different namespace, permitted by our listener configuration), and how easily it handles multiple hostnames:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kutt
  labels:
    app.kubernetes.io/name: kutt
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "link.edc4it.com"
    - "link.course-delivery.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: kutt
          port: 3000

CORS

Moving beyond basic routing, many of our public-facing APIs and front-ends rely on Cross-Origin Resource Sharing (CORS). In the Ingress-NGINX ecosystem, CORS is managed entirely through a series of specific annotations. While this works, it often feels like a bolted-on solution; the configuration relies heavily on loosely typed strings, making it brittle, prone to typos, and difficult to validate programmatically.

Here is an example of one of our legacy Ingress definitions relying on these CORS annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app.kubernetes.io/component: "frontend"
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://*.edc4it.com,https://*.course-delivery.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "POST, GET"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-max-age: "86400"

Fortunately, the Kubernetes Gateway API elevates CORS to a first-class citizen. It is now handled natively via a built-in CORS filter directly within the HTTPRoute rules. This shift makes the migration incredibly straightforward and provides a much more declarative, type-safe, and readable configuration.

However, there is one critical "gotcha" to be aware of during this specific migration: implicit defaults. Ingress-NGINX automatically applies default values for properties like allowHeaders and maxAge if you omit them from your annotations. The Gateway API, by design, is entirely explicit. If you do not define your allowed headers in the HTTPRoute, Envoy will not inject them for you, which will immediately result in CORS failures in the browser. You must meticulously map out and define every header your application expects.

Here is the resulting HTTPRoute, showcasing the structured and explicit CORS filter:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app.kubernetes.io/component: "frontend"
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "sso.edc4it.com"
    - "sso.course-delivery.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /auth
      filters:
        - type: CORS
          cors:
            allowCredentials: true
            allowOrigins:
              - https://*.edc4it.com
              - https://*.course-delivery.com
            allowMethods:
              - POST
              - GET
            allowHeaders:
              - Authorization
              - Content-Type
              - X-Requested-With
              - X-Forwarded-For
              - Accept
            maxAge: 86400
      backendRefs:
        - name: kc-service
          port: 8080

Redirects

Managing HTTP redirects is another area where the Gateway API shines. For our Keycloak instance, we needed a simple redirect from the root path (/) to the login path (/auth). Because this functionality is natively supported by the Gateway API's Domain Specific Language (DSL), the configuration is entirely declarative.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app.kubernetes.io/component: "frontend"
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "sso.edc4it.com"
    - "sso.course-delivery.com"
  rules:
    - matches:
        - path:
            type: Exact
            value: /
      filters:
        - type: RequestRedirect
          requestRedirect:
            path:
              type: ReplaceFullPath
              replaceFullPath: /auth
            statusCode: 302

In the past, to achieve similar redirection logic with Ingress-NGINX (when simple annotations were insufficient), we were often forced to rely on raw NGINX configuration injected via snippets. Relying on snippets is generally discouraged as it tightly couples your manifests to the specific underlying controller implementation and requires explicit administrative enablement due to security risks.

Here is an example of an old Ingress object using a server snippet for a permanent redirect:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      location = /training/open-enrollment/oil {
        rewrite ^ https://www.edc4it.com/training/open-enrollment/ permanent;
      }

Complex Redirects

While the standard RequestRedirect filter covers most use cases, we also had a series of highly complex redirect rules. These rules relied heavily on regular expressions (regex) to strip specific language prefixes, remove .html extensions from legacy blog posts, and handle a rather messy, static list of historical URL changes.

Translating complex NGINX regex rewrites directly into standard Gateway API objects is currently quite cumbersome (or entirely unsupported depending on the exact regex capabilities required by the implementation). To solve this cleanly, we leveraged Envoy's extensibility.

By defining an EnvoyExtensionPolicy, we were able to inject a custom Lua script directly into the Envoy Proxy request lifecycle. This allowed us to write flexible, programmatic routing logic tailored exactly to our legacy URL structures.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: public-site-redirects
  namespace: public-site
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: public-site-main
  lua:
    - type: Inline
      inline: |
        function envoy_on_request(request_handle)
          local path = request_handle:headers():get(":path")

          -- 1. i18n Redirect: /en/foo -> /foo
          local i18n_match = path:match("^/[efnd][nrle]/(.*)")
          if i18n_match then
            request_handle:respond({[":status"] = "301", ["location"] = "/" .. i18n_match}, "")
            return
          end

          -- 2. Blog Category & HTML Strip: /blog/java/post.html -> /blog/post
          local b_cat, b_slug = path:match("^/blog/[%a%-]+/(.-)%.?h?t?m?l?$")
          if b_cat and b_slug ~= "" then
            request_handle:respond({[":status"] = "301", ["location"] = "/blog/" .. b_slug}, "")
            return
          end

          -- 3. Simple .html strip for blogs
          local b_simple = path:match("^/blog/(.-)%.html$")
          if b_simple then
            request_handle:respond({[":status"] = "301", ["location"] = "/blog/" .. b_simple}, "")
            return
          end

          -- 4. Static Redirects (The "Messy" List)
          local redirects = {
            ["/training/course/K8s-NONTECH"] = "/training/course/K8S-NONTECH",
            ["/training/open-enrollment/oil"] = "/training/open-enrollment/",
            -- … omitted for brevity
          }

          if redirects[path] then
            request_handle:respond({[":status"] = "301", ["location"] = redirects[path]}, "")
            return
          end
        end

Easier Prefix-stripping

Many of our routes require stripping a prefix from the URL. With Ingress this worked through the ImplementationSpecific pathType and annotations. In this particular situation we had many Ingress objects because different REST "nouns" (such as "classroom" below) had different authorisation roles associated with it (more on this later, but these authorisation were part of annotations and hence applied to all rules).

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: attendee-service-classroom
  namespace: delivery-management
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1$2$3
  labels:
    app.kubernetes.io/component: "backend"

spec:
  ingressClassName: nginx
  rules:
    - host: www.course-delivery.com
      http:
        paths:
          - path: /api/(classroom)(/|$)(.*)
            pathType: ImplementationSpecific

Not only did it become more declarative, we were also able to use a single HTTPRoute object for all "nouns", because the various authentication roles were defined elsewehre (again, more on this later)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: delivery-management
  namespace: delivery-management
  labels:
    app.kubernetes.io/component: "backend"
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - …
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /
      …

Backend Traffic Policies

We had a specific Ingress object that required the nginx.ingress.kubernetes.io/proxy-body-size annotation to permit larger file uploads to a back-end service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
# …

In Envoy Gateway, this behaviour is configured using the BackendTrafficPolicy Custom Resource. This is a highly versatile Envoy extension that allows you to manage a wide array of Quality of Service (QoS) and traffic management features, rather than just body sizes. It can configure:

  • Load Balancing (loadBalancer): Round Robin, Random, Least Request, Consistent Hash, and Backend Utilisation.
  • Rate Limiting (rateLimit): Local or global rate limits per route.
  • Circuit Breaking (circuitBreaker): Maximum connections, pending requests, and retries.
  • Retries (retry): Number of retries, retry conditions, and fallback strategies.
  • Health Checks (healthCheck): Active health checking on back-end endpoints.
  • Timeouts (timeout): Back-end connection timeouts.
  • TCP Keepalive (tcpKeepalive): Upstream connection keepalive settings.
  • Connection Settings (connection): Back-end connection limits and buffering.
  • Fault Injection (via BTP targeting): Injecting delays or aborts for testing resilience.
  • Bandwidth Limiting (bandwidthLimit): Limiting traffic bandwidth to and from the back-end.
  • Zone-Aware Routing (loadBalancer.zoneAware): Preferring local zones or using weighted zone distribution.
  • Proxy Protocol (proxyProtocol): Enabling PROXY protocol communication to back-ends.

During our training courses, we consistently advocate for extracting these types of QoS requirements out of your application code and delegating them to a proxy or sidecar (for example, using plain Envoy or a service mesh like Istio). The BackendTrafficPolicy makes implementing this architectural pattern at the edge incredibly straightforward.

A significant advantage of this API design is reusability: you can define a standard policy once and apply it to multiple HTTPRoute targets simultaneously using targetRefs.

Below is the configuration we used to replicate our old NGINX body size annotation. It sets a requestBuffer limit of 50Mi for a specific route (the target route name has been obfuscated here to avoid inviting targeted DoS attacks):

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: request-limit-large
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: xxxx
  requestBuffer:
    limit: 50Mi

OIDC Integration

We anticipated that this would be the most complex and stressful phase of the migration. Many of our services require robust authentication and, more importantly, fine-grained, role-based authorisation. Our security back-end is Keycloak (handling both internal and external users), and we rely entirely on OpenID Connect (OIDC).

Historically, to achieve this role-based security, we utilised the brilliant and versatile oauth2-proxy in conjunction with NGINX auth-url annotations.

Here is what the legacy configuration looked like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "http://oauth2-proxy-cd.oauth-proxy.svc.cluster.local/oauth2/auth?allowed_groups=role:planning"
    nginx.ingress.kubernetes.io/auth-signin: "https://sso.xxxx.com/oauth2/start?rd=https://$host$request_uri"
    nginx.ingress.kubernetes.io/auth-response-headers: "x-auth-request-user, x-auth-request-groups, x-auth-request-email, x-auth-request-preferred-username"
# …

We maintained two separate oauth2-proxy instances: one for instructors, students, and training delivery accessing course materials, and another for internal services and partner management. Both interfaced with Keycloak. The security requirements are highly granular, dictating role access down to specific URL paths. For our training environments, we even maintain a scheduled pipeline (using Cypress) that continuously verifies these security boundaries to ensure no regressions slip through during infrastructure updates. Consequently, we were extremely cautious about migrating this setup.

Ultimately, we decided to retire oauth2-proxy entirely and adopt Envoy Gateway's native OIDC authentication and JWT authorisation mechanisms. This architectural shift removes an extra network hop and leaves us with fewer moving parts to maintain. It did, however, mean we were executing two migrations simultaneously: moving from Ingress to Gateway, and moving from oauth2-proxy to Envoy native security.

Tests

Our experience with configuring OIDC is that it is notoriously difficult to get right on the first try. The slightest misconfiguration usually results in opaque, generic errors. Furthermore, the root cause could hide in multiple places: the ingress, the proxy, Keycloak, or the backend application itself, meaning several different logs must be consulted simultaneously.

Because OIDC configuration inherently involves many trial-and-error cycles, fast developer feedback is crucial. To facilitate this, we wrote an automated test suite in Python using the following libraries:

  • pytest - Testing framework for writing and running the tests.
  • mechanize - Browser automation library for programmatic web browsing.
  • beautifulsoup4 - HTML/XML parser for extracting data from web pages.
  • pytest-dotenv - Plugin for loading environment variables from .env files.
  • requests & requests-oauthlib - HTTP libraries with OAuth authentication extensions.

Here is an example test case to illustrate our approach. This test ensures that a user logged in as a "student" receives a 403 Forbidden when attempting to access instructor-only paths:

def test_student():
    br = mechanize.Browser()
    br.set_handle_robots(False)
    url = "https://edc4it.course-delivery.com/deck-showcase"
    br.open(url)
    login(br)
    br.open("https://edc4it.course-delivery.com/…/labs/")

    for c in ["slides", "notes", "admin", "…"]:
        with raises(mechanize.HTTPError) as excinfo:
            br.open(f"https://edc4it.course-delivery.com/…/{c}/")

        assert excinfo.value.code == 403, (
            f"Expected 403, on /{c}/ got {excinfo.value.code}"
        )

We also had tests designed specifically to verify that Envoy was correctly forwarding JWT claims as HTTP headers (which back-end applications rely on for user context). For this, we deployed one of our own training container images, generic-service, which features an endpoint that simply echoes back all received headers.

This allowed us to write precise assertions against the headers Envoy injected:

def test_gw():
    br = mechanize.Browser()
    url = "https://…/test-gw/headers"
    br.open(url)
    res = login(br)
    r = json.loads(res.read())
    
    assert r["x-auth-request-email"] == "e2e-testuser@edc4it.com", \
        f"Expected email 'e2e-testuser@edc4it.com', got {r.get('x-auth-request-email')}"
        
    assert r["x-auth-request-preferred-username"] == "e2e-testuser", \
        f"Expected username 'e2e-testuser', got {r.get('x-auth-request-preferred-username')}"
        
    raw_bytes = base64.b64decode(r["x-auth-request-groups"])
    roles_list = json.loads(raw_bytes.decode('utf-8'))
    assert "deck-showcase-student" in roles_list, \
        f"Expected role 'deck-showcase-student', got {roles_list}"

Security Policies

In Envoy Gateway, OIDC flows and JWT verification are defined using the SecurityPolicy Custom Resource.

Here is a straightforward example of a policy that restricts access to our Prometheus instance, ensuring only users with the k8s-admin role in Keycloak can view it:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: prometheus-policy
  namespace: lab-security
spec:
  targetSelectors:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      matchLabels:
        security.edc4it.com/auth-name: "prometheus"
  oidc:
    provider:
      issuer: https://sso.edc4it.com/auth/realms/edc4it
    clientIDRef:
      name: oidc-client-management
    clientSecret:
      name: oidc-client-management
    redirectURL: https://www.edc4it.com/prometheus/oauth2/callback
    logoutPath: /prometheus/logout
    cookieNames:
      accessToken: "AccessToken"
    scopes:
      - profile
      - email
      - roles
  jwt:
    providers:
      - name: kc-provider
        issuer: "https://sso.edc4it.com/auth/realms/edc4it"
        remoteJWKS:
          uri: "https://sso.edc4it.com/auth/realms/edc4it/protocol/openid-connect/certs"
        extractFrom:
          cookies:
            - AccessToken
  authorization:
    rules:
      - name: admin
        action: Allow
        principal:
          jwt:
            provider: kc-provider
            claims:
              - name: "roles"
                valueType: StringArray
                values: [ "k8s-admin" ]

Implementing security roles for our individual training courses proved to be a more complex challenge. Without going too far off-topic, the difficulty arose because we needed entirely different JWT roles to access different URL paths within the same application.

To solve this, we utilised the Gateway API's ability to name specific rules within an HTTPRoute using the name field (e.g., instructor-admin below).

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
# …
spec:
  rules:
    - name: instructor-admin
      matches:
        - path: { type: PathPrefix, value: "/{{ $courseId }}/admin" }
      backendRefs: *backendRef
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /admin

With the route sections explicitly named, we could target them individually in our SecurityPolicy using targetRefs and the sectionName property.

Notice in the policy below how the claimToHeaders section elegantly replaces the old nginx.ingress.kubernetes.io/auth-response-headers annotation. This instructs Envoy to parse the validated JWT and inject specific claims as HTTP headers before passing the request to the back-end application.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: {{ include "deck.fullname" . }}-instructors-policies
# …
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: {{ include "deck.fullname" . }}
      sectionName: instructor-slides
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: {{ include "deck.fullname" . }}
      sectionName: instructor-admin
    # …
  oidc:
    provider:
      issuer: https://sso.course-delivery.com/auth/realms/course-delivery
    redirectURL: "%REQ(x-forwarded-proto)%://%REQ(:authority)%/{{ $appRoot }}/callback2"
    logoutPath: "/{{ $appRoot }}/logout2"
    clientIDRef:
      name: oidc-client-cwdecks
    clientSecret:
      name: oidc-client-cwdecks
    cookieNames:
      accessToken: "AccessToken"
    scopes:
      - profile
      - email

  jwt:
    providers:
      - name: kc-provider
        issuer: "https://sso.course-delivery.com/auth/realms/course-delivery"
        remoteJWKS:
          uri: "https://sso.course-delivery.com/auth/realms/course-delivery/protocol/openid-connect/certs"
        extractFrom:
          cookies:
            - AccessToken
        claimToHeaders:
          - claim: email
            header: x-auth-request-email
          - claim: realm_access.roles
            header: x-auth-request-groups
          - claim: sub
            header: x-auth-request-user
          - claim: preferred_username
            header: x-auth-request-preferred-username
  authorization:
    rules:
      - name: "instructor"
        action: Allow
        principal:
          jwt:
            provider: kc-provider
            claims:
              - name: "realm_access.roles"
                valueType: StringArray
                values: [ "{{ include "deck.deckname" . }}-instructor", "cm" ]

403 Redirect

One final user-experience challenge was ensuring that when Envoy rejected a request due to authorisation failure, the application displayed a user-friendly "403 Forbidden" HTML page, rather than Envoy's stark default text response.

We accomplished this by once again reaching for the BackendTrafficPolicy. By defining a responseOverride, we instructed Envoy to catch any 403 status codes originating from the specified routes and automatically redirect the user's browser to our custom error page.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: custom-403-override
  namespace: course-delivery
spec:
  targetSelectors:
    - kind: HTTPRoute
      matchLabels:
        app.kubernetes.io/component: deck
  # targetRef can also be used for a single route:
  # targetRef:
  #   group: gateway.networking.k8s.io
  #   kind: HTTPRoute
  #   name: deck-showcase
  responseOverride:
    - match:
        statusCodes:
          - type: Value
            value: 403
      redirect:
        statusCode: 302
        scheme: https
        hostname: www.course-delivery.com
        path:
          type: ReplaceFullPath
          replaceFullPath: "/403.html"

Known SecurityPolicy/OIDC Limitations

A Major Shortcoming: AccessTokens Cannot Be Shared Across SecurityPolicies

One significant limitation we encountered (and that cost us a non-trivial amount of debugging time) is that an AccessToken cookie set by one SecurityPolicy cannot be reused by another SecurityPolicy, even if both policies share an identical OIDC configuration.

The root cause is architectural: Envoy Gateway’s OIDC filter does not store a raw JWT in the AccessToken cookie. Instead, it stores an encrypted, opaque token that is bound to the specific filter instance — and therefore to the specific SecurityPolicy — that created it. Attempting to validate this encrypted cookie as a JWT in a second, independent policy will always fail with an invalid_token error.

For us, this became a real problem. Several of our services live in different namespaces, each with their own HTTPRoute. Some of these routes needed to share the same authenticated session, but because they were targeted by separate SecurityPolicy objects, each login attempt resulted in a fresh, isolated session. The user experience was broken, and the fix was far less obvious than it should have been.

Our immediate workaround was to consolidate: we moved several HTTPRoute objects (including some across namespace boundaries) so that they could all be targeted by a single shared SecurityPolicy. This worked, but it introduced coupling between resources that were previously cleanly separated, and it forced some organisational compromises in our GitOps repository layout.

In hindsight, one might argue that separate Gateway objects per domain or security context could have alleviated some of this, by giving each gateway its own isolated OIDC session scope. But even that feels like an imperfect solution at best. Our routes are granular by nature: a single service serves a mix of fully public paths, paths requiring and paths requiring specific JWT roles. Carving up Gateway objects to match this access matrix would quickly become unwieldy and would defeat much of the consolidation benefit the Gateway API offers in the first place.

The issues #8649 and #9082 track related aspects of the problem (but not this issue). A partial improvement — forwarding the decrypted ID token as a header upstream — is planned for a future release, but as of the time of writing, there is no direct solution for sharing an OIDC session across multiple SecurityPolicy instances.

Perhaps there's something we missed, but we were able to salvage the problem as described above.

No Path-Based Matching in Authorization Rules (Fix Pending Release)

A second limitation we ran into during the April migration was that SecurityPolicy authorization rules did not support path-based matching. It was only possible to differentiate access based on HTTP methods and headers — not on the request path.

This hit us squarely in our course delivery setup, where students and instructors access different URL paths within the same application. Student paths are a subset of what instructors can access, so ideally both roles would be governed by a single policy with path-aware rules. Without path matching, there was no way to express "allow students on /slides/" and "allow instructors on /slides/ and /admin/" within one SecurityPolicy.

The workaround was to split the policy in two: one SecurityPolicy for student access and a separate one for instructor access. This was manageable in itself, but it came with an unfortunate side effect — each SecurityPolicy requires its own OIDC redirectURL and logoutPath. That meant maintaining two distinct callback and logout URLs per course environment, doubling the configuration surface and adding noise to our Keycloak client setup.

The good news is that this limitation has been addressed upstream. GitHub issue #8952 tracked the missing feature, and PR #9008 — "feat: authorization path match" — has been merged. The authorization rules in SecurityPolicy will support a path field with Exact, PathPrefix, and RegularExpression match types, making it possible to consolidate what we split apart. The fix is not yet part of a released version, but is expected to ship in Envoy Gateway 1.9.0. We plan to update our course security configuration to take advantage of this improvement once that release is available.

Conclusion

Migrating from Ingress-NGINX to Envoy Gateway was not a trivial undertaking, taking us a total of 10 days to complete from start to finish. This timeframe reflects the complexity of our OIDC role-based security, the sheer volume of our legacy rules, and the care we took to avoid downtime. However, choosing to perform the migration manually rather than relying on automated conversion tools proved to be the right decision. It forced us to rethink and optimise our north-south traffic architecture from the ground up.

The shift from the brittle, annotation-heavy approach of standard Ingress objects to the highly granular, declarative model of the Kubernetes Gateway API has dramatically improved our infrastructure. We were able to:

  • Consolidate configurations: Features like native CORS support, declarative redirects, and prefix-stripping allowed us to reduce the number of manifests and eliminate YAML duplication.
  • Enhance traffic management: Leveraging BackendTrafficPolicy gave us a clean, reusable way to handle QoS requirements like body size limits and custom error pages without relying on opaque controller snippets.
  • Simplify our security stack: Retiring our separate oauth2-proxy instances in favour of Envoy’s native OIDC and JWT verification removed an entire network hop and left us with fewer moving parts to maintain.

That said, the migration was not without friction. The inability to share an AccessToken across multiple SecurityPolicy instances forced us to restructure our HTTPRoute layout in ways that introduced coupling we would have preferred to avoid. Similarly, the absence of path-based matching in authorisation rules required us to split policies that logically belonged together, adding unnecessary configuration overhead. The latter has been fixed upstream and is expected to ship in Envoy Gateway 1.9.0 — we plan to revisit that part of our configuration once the release is available.

While there are still a few limitations in the Gateway API, it is undeniably the future of cluster networking. The robust extensibility of Envoy Gateway, particularly through features like the EnvoyPatchPolicy and SecurityPolicy, provided exactly the flexibility we needed to handle our complex training environments safely.

We hope sharing these challenges and our resulting configurations proves helpful if you are considering embarking on a similar migration journey!