tenant-catalog/kubeflow/common/oauth2-proxy/components/README.md

# Kubeflow with oauth2-proxy and envoyExtAuthzHttp

For a quick install, see [Example installation](#example-installation).


## Description

Kubeflow authorization operates using custom authentication headers:
* `kubeflow-userid`: Contains the user's email address.
* `kubeflow-groups`: Holds a comma-separated list of user groups.
    * Note: The functionality for `kubeflow-groups` is not fully operational at this time.

This feature was implemented using a custom, minimalistic authorization tool from Arrikto
named `oidc-authservice`. This tool was integrated into Istio using `EnvoyFilter`.

The adoption of `envoyExtAuthzHttp` for authentication offers several advantages:
* **Simplified Authorization Process**: `envoyExtAuthzHttp` extension streamlines adding
  external authorization to the Envoy proxy within an Istio service mesh. It allows for
  declarative policy configuration, reducing the complexity associated with direct
  `EnvoyFilter` modifications.
* **Standardization**: Istio recommends `envoyExtAuthzHttp` for its standardized approach.
  This promotes consistency across Istio deployments and simplifies understanding and
  maintenance of the authorization logic.
* **Separate Policy Management**: Authorization policies are defined in distinct
  configuration resources, not directly in `EnvoyFilter`. This separation eases policy
  management and facilitates updates or replacements of authorization logic without
  altering `EnvoyFilter` configurations.
* **Seamless Integration with Istio**: `envoyExtAuthzHttp` harmonizes with Istio features
  like `AuthorizationPolicy` and `VirtualService`, enabling smoother integration within
  Istio's ecosystem and taking advantage of its architecture and capabilities.
* **Future-Proofing**: There is a high likelihood that Kubeflow's internal authentication
  decisions will transition to relying directly on JWTs instead of custom auth headers.

Additional information on the benefits of moving away from `EnvoyFilter` can be found in these
resources:
* [Istio GitHub Issue #27790](https://github.com/istio/istio/issues/27790)
* [Istio Documentation on Authorization with Custom Authentication](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)

While `envoyExtAuthzHttp` could potentially integrate with `oidc-authservice`, `oauth2-proxy`
emerges as a more advanced authentication proxy. It boasts broader community support and is
widely used in the industry, including in the official Istio documentation on [External
Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom).

For more details on the `oauth2-proxy`, refer to the [official documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/behaviour).

## Available Components

Below is a list of the available Kustomize Components with brief descriptions. Click on each for more details.

* **[allow-unauthenticated-issuer-discovery](./allow-unauthenticated-issuer-discovery.md)** -
  Creates a ClusterRoleBinding for anonymous access to Kubernetes OIDC
  discovery.

* **[central-dashboard](./central-dashboard.md)** - Configures the central
  dashboard to use oauth2-proxy logout URL.

* **[istio-external-auth](./istio-external-auth.md)** - Modifies Istio
  configuration to define oauth2-proxy as external authentication middleware via
  envoyExtAuthzHttp extension provider. Adds RequestAuthentication to trust Dex
  as IdP and AuthorizationPolicies to delegate authentication to oauth2-proxy.

* **[istio-m2m](./istio-m2m.md)** - Creates RequestAuthentication for Istio to
  trust the OIDC Issuer specified in parameters. This allows the generation of
  JWTs for authenticating requests, typically as Bearer Tokens in the
  Authorization header. By default, the OIDC Issuer is the in-cluster Kubernetes
  OIDC.

## CloudFlare

CloudFlare requires that certain static, standard web browser assets are accessible without
user authentication. This is crucial because CloudFlare aims to cache these assets for
enhanced performance. If these assets necessitate user authentication, CloudFlare robots
will be redirected to the authentication page, potentially causing access issues with the
Kubeflow instance behind CloudFlare.

This issue can be resolved by defining a set of assets in the Istio `AuthorizationPolicy`
that do not require authentication. An example `AuthorizationPolicy` for this purpose is
provided in the file `authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml`.

## Explaining the Auth Routine

1. Istio is configured with the `envoyExtAuthzHttp` extension provider pointing to the
   `oauth2-proxy` service. This configuration enables the use of this extension in
   `AuthorizationPolicy` for adding external authorization to the service mesh.
2. The Istio service mesh has an `AuthorizationPolicy` named `istio-ingressgateway-oauth2-proxy`
   in the `istio-system` (Istio root) namespace. This policy is set with `action: CUSTOM` and
   specifies the `oauth2-proxy` provider. Consequently, every request to the
   `istio-ingressgateway` must pass through the external authorization service, `oauth2-proxy`.
3. `oauth2-proxy` decides based on the cookie named `oauth2_proxy_kubeflow`. If this cookie
   is absent, expired, or invalid, `oauth2-proxy` redirects to the configured OIDC provider,
   typically `dex`. The authentication redirect includes:
    * A `redirect_uri` for redirecting the user post-authentication,
    * A code challenge to guard against interception and replay attacks,
    * A state parameter to validate the authorization response's authenticity and
      ensure it originates from the initial request.

   Post-authentication, a cookie is set in the user's browser, used by Istio and `oauth2-proxy`
   for authorizing requests.

   `oauth2-proxy` also checks for an `Authorization` header with a JWT bearer token. It trusts
   JWTs issued by the configured OIDC provider (default `dex`) and can be configured to trust
   additional JWTs. If a valid JWT is present, `oauth2-proxy` forwards the request to Istio
   along with the authorization header.

   The key objective here is to supply an `Authorization` header with a JWT Bearer Token,
   later used in `RequestAuthentication`. This step configures Istio to trust the JWT, parse
   its claims to include user email and groups in custom Kubeflow authorization headers, and
   make routing and authorization decisions based on these JWT claims.
4. With this setup, Istio always receives the JWT in requests, enabling authorization
   decisions based on JWT claims.

## Using HTTPS

`oauth2-proxy` is initially set up with an `http` endpoint, secured by the Istio Service Mesh.
As a result, `oauth2-proxy` may default to assuming that the authentication redirect URI should
also use `http`. To enforce the use of `https`, modify the variable `FORCE_HTTPS` in
`kustomization.yaml` to `true`. This adjustment leverages the `oauth2-proxy` configuration
option `--cookie-secure`, ensuring redirection occurs with `https`.

## Istio JWT Public Key Refresh Interval

In the initial setup of Kubeflow, it's common for `istiod` to become available before `dex`.
Istio's `RequestAuthentication` is configured to retrieve the JWT Public Key from the Issuer
URL. If the Issuer is not yet operational, placeholder keys are set, which can render the
setup nonfunctional until Istio can access the correct JWT Public Key. To address this,
`istiod` is configured with the environment variable `PILOT_JWT_PUB_KEY_REFRESH_INTERVAL="1m"`.
This setting ensures the JWT Public Key is refreshed every minute, rather than the default
20 minutes.

Without this configuration, users may encounter the following Istio error:
```
Jwks doesn't have key to match kid or alg from Jwt
```

## Issues with This Setup

While not an inherent issue with Istio or `oauth2-proxy`, the current Kubeflow configuration
automatically redirects to a URL specified in the logout response body's `afterLogoutURL` key.
This behavior stems from custom integration with the `oidc-authservice` component. While
`oauth2-proxy` is capable of redirecting to the base Kubeflow page, this custom setup results
in users being logged out (with the authentication cookie removed) but not redirected back to
the Kubeflow Home Page.

Details of this custom integration are available at:
* [oidc-authservice server.go](https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509)
* [Kubeflow logout-button.js](https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50)

To log in again, users must manually refresh the page.

## Example Installation

To install Kubeflow configured to use `oauth2-proxy` with Istio's `envoyExtAuthzHttp` extension,
make the following changes to the `example/kustomization.yaml` file:
* use `oauth2-proxy` overlay for istio-install
  ```
  # from
  - ../common/istio-1-24/istio-install/base
  # to
  - ../common/istio-1-24/istio-install/overlays/oauth2-proxy
  ```
* change `OIDC Authservice` to `oauth2-proxy for OIDC` and use overlay for m2m
  bearer tokens with self-signed in-cluster issuer
  ```
  # from
  - ../common//oidc-authservice/base
  # to
  - ../common/oauth2-proxy/overlays/m2m-dex-and-kind
  ```
* change Dex overlay
  ```
  # from
  - ../common/dex/overlays/istio
  # to
  - ../common/dex/overlays/oauth2-proxy
* change Central Dashboard overlay to use oauth2-proxy for logout
  ```
  # from
  - ../apps/centraldashboard/upstream/overlays/kserve
  # to
  - ../apps/centraldashboard/manuel-patches/overlays/oauth2-proxy
  ```

All those changes combined can be done with this single command:
```diff
$ git apply <<EOF
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
index c1a85789..4a50440c 100644
--- a/example/kustomization.yaml
+++ b/example/kustomization.yaml
@@ -38,11 +38,11 @@ resources:
 # Istio
 - ../common/istio-1-24/istio-crds/base
 - ../common/istio-1-24/istio-namespace/base
-- ../common/istio-1-24/istio-install/base
-# OIDC Authservice
-- ../common//oidc-authservice/base
+- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
+# oauth2-proxy for OIDC
+- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
 # Dex
-- ../common/dex/overlays/istio
+- ../common/dex/overlays/oauth2-proxy
 # KNative
 - ../common/knative/knative-serving/overlays/gateways
 - ../common/knative/knative-eventing/base
@@ -60,7 +60,7 @@ resources:
 # Katib
 - ../apps/katib/upstream/installs/katib-with-kubeflow
 # Central Dashboard
-- ../apps/centraldashboard/upstream/overlays/kserve
+- ../apps/centraldashboard/overlays
 # Admission Webhook
 - ../apps/admission-webhook/upstream/overlays/cert-manager
 # Jupyter Web App
EOF
```