Kubernetes: Ingress External Authentication (Api Keys)

Learn how to secure Kubernetes services with API key authentication using NGINX Ingress. This step-by-step guide covers setting up external authentication with a Scala http4s service, forwarding custom identity headers, and testing with tools like httpie and curl. Perfect for enhancing security while keeping your configuration flexible and maintainable.

...
Raphaël ParréePublished on

Most access to our internal web applications, services, and course content is secured using OAuth2/OIDC via Keycloak and oauth2-proxy.

For comprehensive guidance on that, we cover setting up OAuth authentication for Ingress in our K8S-CORE course and using OIDC with kubectl in our K8S-ADMIN course.

However, for certain previously public services, we required a simpler authentication mechanism based on API keys. Fortunately, Ingress makes it easy to integrate external authentication using the nginx.ingress.kubernetes.io/auth-url annotation:

Note: The F5 NGINX and various Gateways do have built-in support for API Keys. Our cluster, as many others, is, however, using a "standard" community nginx

  1. Request Initiation: A client sends a request to the Ingress controller.
  2. Authentication Forwarding: The Ingress controller forwards the request to the URL specified by the nginx.ingress.kubernetes.io/auth-url annotation on the matching Ingress resource.
  3. External Authentication: The external authentication service processes the request and responds with:
    • 200 OK: Authentication succeeded.
    • 401 Unauthorized, 403 Forbidden, or 3xx Redirect: Depending on the authentication logic.
    • Optionally includes headers like WWW-Authenticate or custom headers.
  4. Response Handling:
    • If 200 OK, the request is forwarded to the target endpoint.
    • For non-200 responses, the client is redirected to the URL specified by nginx.ingress.kubernetes.io/auth-signin (if configured).
    • If no auth-signin URL is specified, the response is sent directly back to the client.

Let's go through an example using the Api Key Authentication Service. You'll need:

  • cluster with an Nginx Ingress Controller installed. You could run minikube: (see instructions below for a minikube cluster)
  • Kubectl
  • Helm
  • HTTPie or translate the http commands we supply to curl/…

Minikube (optional)

Below are instructions to quickly set up a minikube cluster:

  • Start a new cluster named "external-auth-demo" (using the profile option)

    minikube -p external-auth-demo start --addons="ingress"

  • To make it easier to access the Ingress controller, patch its service and change its type to LoadBalancer

    $ kubectl patch service ingress-nginx-controller \
  -n ingress-nginx \
  --type json \
  -p '[{"op": "replace", "path": "/spec/type", "value": "LoadBalancer"}]'

  • In order for minikube to assign a load balancer IP, run the minikube tunnel in another terminal session (in case you are unfamiliar, this process runs in the foreground, and henceL blocks your session):

    $ minikube -p external-auth-demo tunnel

  • You should now be able to get a load balancer IP for your Ingress controller (supplied by the tunnel above)

    $ kubectl -n ingress-nginx \
   get svc ingress-nginx-controller \
   -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Sample Service to Secure

To demonstrate API key protection, let's deploy a target service. For this, we'll use our generic service, which provides convenient utility endpoints, such as:

  • /version: Displays the service version, controllable via the BEHAVIOUR_VERSION environment variable. This feature is particularly useful in courses like our Service Mesh course.
  • /headers: Responds with all request headers, which can help debug or observe request properties.
  • …

We'll start by creating a namespace, deploying the application, and adding an unauthenticated Ingress to the base URL /service.

Let's create a sample namespace, deploy the application, and add an ingress. We'll use Helm:

helm upgrade --install sample oci://repo.course-delivery.com/chart-generic-service/generic-service \
  --version v6.5.3 \
  -n sample --create-namespace

Next, create an Ingress resource for unauthenticated access to the service. This will expose the service at /service.

  1. Save the following YAML to a file named gen-service-ingress.yaml:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-generic-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - http:
        paths:
          - path: /service(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: sample-generic-service
                port:
                  number: 80

  2. Apply the Ingress resource:

    kubectl apply -f gen-service-ingress.yaml

You can now access the application. For example, the /value endpoint will return information about the responding pod and a value controlled via the BEHAVIOUR_RETURN_VALUE environment variable.

Run the following command to test:

http http://$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/service/value

The external Authentication Service

The external authentication application is available on our GitLab repository: apikey-auth.
This project is a minimalistic Scala http4s implementation, intentionally kept "bare-bones" by avoiding dependencies like Tapir or Circe for simplicity.

The repository includes a ready-to-use Kustomization configuration. To get started, clone the repository:

$  git clone https://gitlab.edc4it.com/oss/apikey-auth.git

Take a look at the sample Kustomization located in the apikey-auth/deploy/sample directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: api-key-auth

secretGenerator:
  - name: apikeys
    files:
      - ./files/apikeys.properties

resources:
  - ../base

Here's a rundown of its contents:

  • Base Extension (../base):

    • Defines the api-key-auth namespace.
    • Includes a deployment and corresponding service (provider-auth).
    • Uses the repo.edc4it.com/apikey-auth container image for the deployment.
    • Mounts a secret named apikeys into the deployment.
    • The mounted secret is read by the application as the AUTH_API_KEY_FILE.

  • sample Kustomization:

    • Extends the base configuration.
    • Generates the apikeys secret using the ./files/apikeys.properties file.
    • The apikeys.properties file defines two API keys mapped to the following identities: 
      1111=AllSafe
2222=Evil Corp

Applying the Configuration

After reviewing the sample Kustomization, apply it to your cluster using:

kubectl apply -k apikey-auth/deploy/sample

You should now have a service, with endpoints pointing to your pod:

kubectl -n api-key-auth get service,ep,pod
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/provider-auth   ClusterIP   10.107.87.254   <none>        80/TCP    28s

NAME                      ENDPOINTS          AGE
endpoints/provider-auth   10.244.2.14:8080   28s

NAME                                READY   STATUS    RESTARTS   AGE
pod/api-key-auth-6ff65b9cb7-l82w9   1/1     Running   0          28s

To verify that the authentication service is working correctly, follow these steps:

  1. Start a Port Forward
    In one terminal session, forward port 8080 to the service's port 80:

    kubectl -n api-key-auth port-forward services/provider-auth 8080:80

  2. Test the /auth Endpoint
    In another terminal session, use the /auth endpoint with a valid API key.
    If you prefer curl, the equivalent command is shown alongside the httpie example:

    # Using httpie
http :8080/auth Authorization:"Bearer 2222"

# Using curl
curl -H "Authorization: Bearer 2222" http://localhost:8080/auth

    Example response:

    HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 10
Content-Type: text/plain; charset=UTF-8
Date: Thu, 05 Dec 2024 11:41:35 GMT
X-Identity: Evil Corp

Authorized
    • Status Code: 200 OK, indicating successful authentication.
    • Response Header: Includes an X-Identity header with the value Evil Corp. This header is controlled by the AUTH_IDENTITY_HEADER environment variable in the authentication service.

  3. Test with an Invalid API Key
    Try using an invalid API key (anything other than 1111 or 2222) and observe the response:

    http :8080/auth Authorization:"Bearer invalid-key"

    Example response:

    HTTP/1.1 401 Unauthorized
Content-Length: 12
Content-Type: text/plain; charset=UTF-8
Date: Thu, 05 Dec 2024 11:42:15 GMT

Invalid Token
    • Status Code: 401 Unauthorized, indicating the provided API key is invalid.

Secure our Application

After all the preparation, we’ve reached the exciting final step: securing access to our service.

At this point, our service is accessible without any authentication, as demonstrated below:

http http://$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
   -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/service/value

This command fetches the /value endpoint, currently unprotected. Let’s change that by enabling API key authentication! 🚀

Return to the gen-service-ingress.yaml file, which defines the Ingress for our web application. Currently, it includes only two annotations to rewrite the URL for the target service:

annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  nginx.ingress.kubernetes.io/use-regex: "true"

Now, let’s enhance this configuration to enable external authentication via our api-key-auth service.

The fully qualified domain name (FQDN) in a standard cluster is: provider-auth.api-key-auth.svc.cluster.local (which is <service name>.<namespace>,<service sub domain>.<cluster domain>).

We’ll use this FQDN to specify the /auth endpoint of the authentication service. Add the following annotation to the gen-service-ingress.yaml file:

annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  nginx.ingress.kubernetes.io/use-regex: "true"
  # Add this:
  nginx.ingress.kubernetes.io/auth-url: http://provider-auth.api-key-auth.svc.cluster.local/auth

This configuration instructs the Ingress controller to:

  1. Perform external authentication by forwarding requests to /auth on the provider-auth service.
  2. Use the response from the api-key-auth service to determine whether the request should proceed.

After saving the updated file, apply it to your cluster:

$ kubectl apply -f gen-service-ingress.yaml

Let’s test the updated Ingress configuration by making a request to the /service/value endpoint. Since we’ve added external authentication, the request should now fail with a 401 Unauthorized response if no valid API key is provided:

http http://$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/service/value

Expected response:

HTTP/1.1 401 Unauthorized
…

If the request fails with a 500 Internal Server Error, it’s likely that the Ingress controller is unable to resolve the provider-auth.api-key-auth.svc.cluster.local hostname. This can happen if there’s a DNS resolution issue or network misconfiguration.

To debug this, check the logs of the Ingress controller:

kubectl -n ingress-nginx logs -l app.kubernetes.io/name=ingress-nginx

Now, let’s test the /service/value endpoint with a valid API key. Use the Authorization header to include the token 2222:

http http://$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/service/value Authorization:"Bearer 2222"

Expected response:

HTTP/1.1 200 OK
…

This confirms that the authentication service successfully validates the API key and allows the request to proceed. You can now access the service with the proper token while unauthenticated requests are blocked with a 401 Unauthorized response.

In many cases, the application needs to know the identity of the authenticated user or client. Recall that our API key authentication service includes the identity in the X-Identity response header. To ensure the Ingress controller forwards this header to the target service, we can add the nginx.ingress.kubernetes.io/auth-response-headers annotation to our Ingress configuration:

annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /$2
  nginx.ingress.kubernetes.io/use-regex: "true"
  nginx.ingress.kubernetes.io/auth-url: http://provider-auth.api-key-auth.svc.cluster.local/auth
  # Add this annotation to forward the identity header:
  nginx.ingress.kubernetes.io/auth-response-headers: X-Identity

Our generic service provides a /headers endpoint that echoes the headers it receives. Let’s use it to verify that the X-Identity header is being forwarded to the backend.

Run the following command with a valid API key:

http http://$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/service/headers Authorization:"Bearer 2222"

Expected response:

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 471
Content-Type: application/json
Date: Thu, 05 Dec 2024 13:30:47 GMT

{
    "Accept": "*/*",
    "Authorization": "Bearer 2222",
    "X-Identity": "Evil Corp",
    …
}

Notice that the backend now receives the X-Identity header, which contains the identity information ("Evil Corp" in this case). This confirms that the authentication service's response header is successfully passed through the Ingress controller to the application.

By forwarding this identity header, you enable the application to make decisions based on the authenticated user's identity, such as authorization or custom logic.

Conclusion

Having the infrastructure, such as an Ingress reverse proxy, handle external authentication is straightforward. By simply pointing it to an HTTP service using nginx.ingress.kubernetes.io/auth-url, you can delegate authentication responsibilities. This service determines access by replying with standard HTTP response codes (200, 401, etc.). Additionally, the Ingress controller can forward custom headers to backend applications using nginx.ingress.kubernetes.io/auth-response-headers, enabling flexible integration with application logic.

This approach is significantly better than embedding authentication (and this level of authorization) directly into individual applications, as it centralizes security and simplifies management.

Before we wrap up, here are some other useful annotations related to external authentication:

  • nginx.ingress.kubernetes.io/auth-method: Specify the HTTP method (GET, POST, etc.) used for authentication.
  • nginx.ingress.kubernetes.io/auth-signin: Define the URL where users are redirected when authentication is required.
  • nginx.ingress.kubernetes.io/auth-cache-key: Configure a cache key for storing authentication results to improve performance.
  • nginx.ingress.kubernetes.io/auth-cache-duration: Specify how long authentication results are cached.
  • nginx.ingress.kubernetes.io/auth-snippet: Inject custom NGINX configuration snippets for advanced use cases.
  • .. see annotations

With these tools and annotations, you can create a powerful and efficient authentication layer that enhances both security and maintainability. Now, go and explore how this can simplify your infrastructure! 🚀