Kubeflow with oauth2-proxy and envoyExtAuthzHttp

For a quick install, see Example installation.

Description

Kubeflow authorization operates using custom authentication headers:

kubeflow-userid: Contains the user's email address.
kubeflow-groups: Holds a comma-separated list of user groups.
- Note: The functionality for kubeflow-groups is not fully operational at this time.

This feature was implemented using a custom, minimalistic authorization tool from Arrikto named oidc-authservice. This tool was integrated into Istio using EnvoyFilter.

The adoption of envoyExtAuthzHttp for authentication offers several advantages:

Simplified Authorization Process: envoyExtAuthzHttp extension streamlines adding external authorization to the Envoy proxy within an Istio service mesh. It allows for declarative policy configuration, reducing the complexity associated with direct EnvoyFilter modifications.
Standardization: Istio recommends envoyExtAuthzHttp for its standardized approach. This promotes consistency across Istio deployments and simplifies understanding and maintenance of the authorization logic.
Separate Policy Management: Authorization policies are defined in distinct configuration resources, not directly in EnvoyFilter. This separation eases policy management and facilitates updates or replacements of authorization logic without altering EnvoyFilter configurations.
Seamless Integration with Istio: envoyExtAuthzHttp harmonizes with Istio features like AuthorizationPolicy and VirtualService, enabling smoother integration within Istio's ecosystem and taking advantage of its architecture and capabilities.
Future-Proofing: There is a high likelihood that Kubeflow's internal authentication decisions will transition to relying directly on JWTs instead of custom auth headers.

Additional information on the benefits of moving away from EnvoyFilter can be found in these resources:

While envoyExtAuthzHttp could potentially integrate with oidc-authservice, oauth2-proxy emerges as a more advanced authentication proxy. It boasts broader community support and is widely used in the industry, including in the official Istio documentation on External Authorization.

For more details on the oauth2-proxy, refer to the official documentation.

Available Components

Below is a list of the available Kustomize Components with brief descriptions. Click on each for more details.

allow-unauthenticated-issuer-discovery - Creates a ClusterRoleBinding for anonymous access to Kubernetes OIDC discovery.
central-dashboard - Configures the central dashboard to use oauth2-proxy logout URL.
istio-external-auth - Modifies Istio configuration to define oauth2-proxy as external authentication middleware via envoyExtAuthzHttp extension provider. Adds RequestAuthentication to trust Dex as IdP and AuthorizationPolicies to delegate authentication to oauth2-proxy.
istio-m2m - Creates RequestAuthentication for Istio to trust the OIDC Issuer specified in parameters. This allows the generation of JWTs for authenticating requests, typically as Bearer Tokens in the Authorization header. By default, the OIDC Issuer is the in-cluster Kubernetes OIDC.

CloudFlare

CloudFlare requires that certain static, standard web browser assets are accessible without user authentication. This is crucial because CloudFlare aims to cache these assets for enhanced performance. If these assets necessitate user authentication, CloudFlare robots will be redirected to the authentication page, potentially causing access issues with the Kubeflow instance behind CloudFlare.

This issue can be resolved by defining a set of assets in the Istio AuthorizationPolicy that do not require authentication. An example AuthorizationPolicy for this purpose is provided in the file authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml.

Explaining the Auth Routine

Istio is configured with the envoyExtAuthzHttp extension provider pointing to the oauth2-proxy service. This configuration enables the use of this extension in AuthorizationPolicy for adding external authorization to the service mesh.
The Istio service mesh has an AuthorizationPolicy named istio-ingressgateway-oauth2-proxy in the istio-system (Istio root) namespace. This policy is set with action: CUSTOM and specifies the oauth2-proxy provider. Consequently, every request to the istio-ingressgateway must pass through the external authorization service, oauth2-proxy.
oauth2-proxy decides based on the cookie named oauth2_proxy_kubeflow. If this cookie is absent, expired, or invalid, oauth2-proxy redirects to the configured OIDC provider, typically dex. The authentication redirect includes:
- A redirect_uri for redirecting the user post-authentication,
- A code challenge to guard against interception and replay attacks,
- A state parameter to validate the authorization response's authenticity and ensure it originates from the initial request.
Post-authentication, a cookie is set in the user's browser, used by Istio and oauth2-proxy for authorizing requests.

oauth2-proxy also checks for an Authorization header with a JWT bearer token. It trusts JWTs issued by the configured OIDC provider (default dex) and can be configured to trust additional JWTs. If a valid JWT is present, oauth2-proxy forwards the request to Istio along with the authorization header.

The key objective here is to supply an Authorization header with a JWT Bearer Token, later used in RequestAuthentication. This step configures Istio to trust the JWT, parse its claims to include user email and groups in custom Kubeflow authorization headers, and make routing and authorization decisions based on these JWT claims.
With this setup, Istio always receives the JWT in requests, enabling authorization decisions based on JWT claims.

Using HTTPS

oauth2-proxy is initially set up with an http endpoint, secured by the Istio Service Mesh. As a result, oauth2-proxy may default to assuming that the authentication redirect URI should also use http. To enforce the use of https, modify the variable FORCE_HTTPS in kustomization.yaml to true. This adjustment leverages the oauth2-proxy configuration option --cookie-secure, ensuring redirection occurs with https.

Istio JWT Public Key Refresh Interval

In the initial setup of Kubeflow, it's common for istiod to become available before dex. Istio's RequestAuthentication is configured to retrieve the JWT Public Key from the Issuer URL. If the Issuer is not yet operational, placeholder keys are set, which can render the setup nonfunctional until Istio can access the correct JWT Public Key. To address this, istiod is configured with the environment variable PILOT_JWT_PUB_KEY_REFRESH_INTERVAL="1m". This setting ensures the JWT Public Key is refreshed every minute, rather than the default 20 minutes.

Without this configuration, users may encounter the following Istio error:

Jwks doesn't have key to match kid or alg from Jwt

Issues with This Setup

While not an inherent issue with Istio or oauth2-proxy, the current Kubeflow configuration automatically redirects to a URL specified in the logout response body's afterLogoutURL key. This behavior stems from custom integration with the oidc-authservice component. While oauth2-proxy is capable of redirecting to the base Kubeflow page, this custom setup results in users being logged out (with the authentication cookie removed) but not redirected back to the Kubeflow Home Page.

Details of this custom integration are available at:

To log in again, users must manually refresh the page.

Example Installation

To install Kubeflow configured to use oauth2-proxy with Istio's envoyExtAuthzHttp extension, make the following changes to the example/kustomization.yaml file:

use oauth2-proxy overlay for istio-install

# from
- ../common/istio-1-24/istio-install/base
# to
- ../common/istio-1-24/istio-install/overlays/oauth2-proxy

change OIDC Authservice to oauth2-proxy for OIDC and use overlay for m2m bearer tokens with self-signed in-cluster issuer
```
# from
- ../common//oidc-authservice/base
# to
- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
```

change Dex overlay

# from
- ../common/dex/overlays/istio
# to
- ../common/dex/overlays/oauth2-proxy

change Central Dashboard overlay to use oauth2-proxy for logout

# from
- ../apps/centraldashboard/upstream/overlays/kserve
# to
- ../apps/centraldashboard/manuel-patches/overlays/oauth2-proxy

All those changes combined can be done with this single command:

$ git apply <<EOF
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
index c1a85789..4a50440c 100644
--- a/example/kustomization.yaml
+++ b/example/kustomization.yaml
@@ -38,11 +38,11 @@ resources:
 # Istio
 - ../common/istio-1-24/istio-crds/base
 - ../common/istio-1-24/istio-namespace/base
-- ../common/istio-1-24/istio-install/base
-# OIDC Authservice
-- ../common//oidc-authservice/base
+- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
+# oauth2-proxy for OIDC
+- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
 # Dex
-- ../common/dex/overlays/istio
+- ../common/dex/overlays/oauth2-proxy
 # KNative
 - ../common/knative/knative-serving/overlays/gateways
 - ../common/knative/knative-eventing/base
@@ -60,7 +60,7 @@ resources:
 # Katib
 - ../apps/katib/upstream/installs/katib-with-kubeflow
 # Central Dashboard
-- ../apps/centraldashboard/upstream/overlays/kserve
+- ../apps/centraldashboard/overlays
 # Admission Webhook
 - ../apps/admission-webhook/upstream/overlays/cert-manager
 # Jupyter Web App
EOF

10 KiB Raw Blame History