update kubeflow dip-catalog

This commit is contained in:
ChanghoWoo
2025-01-13 02:31:27 +00:00
parent 1dc1181a03
commit 5451f16d72
1959 changed files with 602337 additions and 0 deletions
@@ -0,0 +1,3 @@
approvers:
- axel7083
- kromanow94
@@ -0,0 +1,231 @@
# Kubeflow with oauth2-proxy and envoyExtAuthzHttp
For a quick install, see [Example installation](#example-installation).
## Description
Kubeflow authorization operates using custom authentication headers:
* `kubeflow-userid`: Contains the user's email address.
* `kubeflow-groups`: Holds a comma-separated list of user groups.
* Note: The functionality for `kubeflow-groups` is not fully operational at this time.
This feature was implemented using a custom, minimalistic authorization tool from Arrikto
named `oidc-authservice`. This tool was integrated into Istio using `EnvoyFilter`.
The adoption of `envoyExtAuthzHttp` for authentication offers several advantages:
* **Simplified Authorization Process**: `envoyExtAuthzHttp` extension streamlines adding
external authorization to the Envoy proxy within an Istio service mesh. It allows for
declarative policy configuration, reducing the complexity associated with direct
`EnvoyFilter` modifications.
* **Standardization**: Istio recommends `envoyExtAuthzHttp` for its standardized approach.
This promotes consistency across Istio deployments and simplifies understanding and
maintenance of the authorization logic.
* **Separate Policy Management**: Authorization policies are defined in distinct
configuration resources, not directly in `EnvoyFilter`. This separation eases policy
management and facilitates updates or replacements of authorization logic without
altering `EnvoyFilter` configurations.
* **Seamless Integration with Istio**: `envoyExtAuthzHttp` harmonizes with Istio features
like `AuthorizationPolicy` and `VirtualService`, enabling smoother integration within
Istio's ecosystem and taking advantage of its architecture and capabilities.
* **Future-Proofing**: There is a high likelihood that Kubeflow's internal authentication
decisions will transition to relying directly on JWTs instead of custom auth headers.
Additional information on the benefits of moving away from `EnvoyFilter` can be found in these
resources:
* [Istio GitHub Issue #27790](https://github.com/istio/istio/issues/27790)
* [Istio Documentation on Authorization with Custom Authentication](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
While `envoyExtAuthzHttp` could potentially integrate with `oidc-authservice`, `oauth2-proxy`
emerges as a more advanced authentication proxy. It boasts broader community support and is
widely used in the industry, including in the official Istio documentation on [External
Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom).
For more details on the `oauth2-proxy`, refer to the [official documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/behaviour).
## Available Components
Below is a list of the available Kustomize Components with brief descriptions. Click on each for more details.
* **[allow-unauthenticated-issuer-discovery](./allow-unauthenticated-issuer-discovery.md)** -
Creates a ClusterRoleBinding for anonymous access to Kubernetes OIDC
discovery.
* **[central-dashboard](./central-dashboard.md)** - Configures the central
dashboard to use oauth2-proxy logout URL.
* **[configure-self-signed-kubernetes-oidc-issuer](./configure-self-signed-kubernetes-oidc-issuer.md)** -
Creates a Kubernetes Job with RBAC to discover and configure the in-cluster,
self-signed default Kubernetes OIDC issuer, essential for using Kubernetes
OIDC for M2M Tokens.
* **[istio-external-auth](./istio-external-auth.md)** - Modifies Istio
configuration to define oauth2-proxy as external authentication middleware via
envoyExtAuthzHttp extension provider. Adds RequestAuthentication to trust Dex
as IdP and AuthorizationPolicies to delegate authentication to oauth2-proxy.
* **[istio-m2m](./istio-m2m.md)** - Creates RequestAuthentication for Istio to
trust the OIDC Issuer specified in parameters. This allows the generation of
JWTs for authenticating requests, typically as Bearer Tokens in the
Authorization header. By default, the OIDC Issuer is the in-cluster Kubernetes
OIDC.
## OpenShift
This deployment of `oauth2-proxy` does not support OpenShift out of the box. To facilitate
integration with OpenShift:
* Utilize the OpenShift-specific distribution of `oauth2-proxy`, accessible at:
* [OpenShift OAuth Proxy on GitHub](https://github.com/openshift/oauth-proxy)
* Enable RBAC for token reviews by incorporating the `rbac.tokenreviews.yaml` file into
your kustomize. For more details, refer to the relevant comment in `kustomization.yaml`.
## CloudFlare
CloudFlare requires that certain static, standard web browser assets are accessible without
user authentication. This is crucial because CloudFlare aims to cache these assets for
enhanced performance. If these assets necessitate user authentication, CloudFlare robots
will be redirected to the authentication page, potentially causing access issues with the
Kubeflow instance behind CloudFlare.
This issue can be resolved by defining a set of assets in the Istio `AuthorizationPolicy`
that do not require authentication. An example `AuthorizationPolicy` for this purpose is
provided in the file `authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml`.
## Explaining the Auth Routine
1. Istio is configured with the `envoyExtAuthzHttp` extension provider pointing to the
`oauth2-proxy` service. This configuration enables the use of this extension in
`AuthorizationPolicy` for adding external authorization to the service mesh.
2. The Istio service mesh has an `AuthorizationPolicy` named `istio-ingressgateway-oauth2-proxy`
in the `istio-system` (Istio root) namespace. This policy is set with `action: CUSTOM` and
specifies the `oauth2-proxy` provider. Consequently, every request to the
`istio-ingressgateway` must pass through the external authorization service, `oauth2-proxy`.
3. `oauth2-proxy` decides based on the cookie named `oauth2_proxy_kubeflow`. If this cookie
is absent, expired, or invalid, `oauth2-proxy` redirects to the configured OIDC provider,
typically `dex`. The authentication redirect includes:
* A `redirect_uri` for redirecting the user post-authentication,
* A code challenge to guard against interception and replay attacks,
* A state parameter to validate the authorization response's authenticity and
ensure it originates from the initial request.
Post-authentication, a cookie is set in the user's browser, used by Istio and `oauth2-proxy`
for authorizing requests.
`oauth2-proxy` also checks for an `Authorization` header with a JWT bearer token. It trusts
JWTs issued by the configured OIDC provider (default `dex`) and can be configured to trust
additional JWTs. If a valid JWT is present, `oauth2-proxy` forwards the request to Istio
along with the authorization header.
The key objective here is to supply an `Authorization` header with a JWT Bearer Token,
later used in `RequestAuthentication`. This step configures Istio to trust the JWT, parse
its claims to include user email and groups in custom Kubeflow authorization headers, and
make routing and authorization decisions based on these JWT claims.
4. With this setup, Istio always receives the JWT in requests, enabling authorization
decisions based on JWT claims.
## Using HTTPS
`oauth2-proxy` is initially set up with an `http` endpoint, secured by the Istio Service Mesh.
As a result, `oauth2-proxy` may default to assuming that the authentication redirect URI should
also use `http`. To enforce the use of `https`, modify the variable `FORCE_HTTPS` in
`kustomization.yaml` to `true`. This adjustment leverages the `oauth2-proxy` configuration
option `--cookie-secure`, ensuring redirection occurs with `https`.
## Istio JWT Public Key Refresh Interval
In the initial setup of Kubeflow, it's common for `istiod` to become available before `dex`.
Istio's `RequestAuthentication` is configured to retrieve the JWT Public Key from the Issuer
URL. If the Issuer is not yet operational, placeholder keys are set, which can render the
setup nonfunctional until Istio can access the correct JWT Public Key. To address this,
`istiod` is configured with the environment variable `PILOT_JWT_PUB_KEY_REFRESH_INTERVAL="1m"`.
This setting ensures the JWT Public Key is refreshed every minute, rather than the default
20 minutes.
Without this configuration, users may encounter the following Istio error:
```
Jwks doesn't have key to match kid or alg from Jwt
```
## Issues with This Setup
While not an inherent issue with Istio or `oauth2-proxy`, the current Kubeflow configuration
automatically redirects to a URL specified in the logout response body's `afterLogoutURL` key.
This behavior stems from custom integration with the `oidc-authservice` component. While
`oauth2-proxy` is capable of redirecting to the base Kubeflow page, this custom setup results
in users being logged out (with the authentication cookie removed) but not redirected back to
the Kubeflow Home Page.
Details of this custom integration are available at:
* [oidc-authservice server.go](https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509)
* [Kubeflow logout-button.js](https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50)
To log in again, users must manually refresh the page.
## Example Installation
To install Kubeflow configured to use `oauth2-proxy` with Istio's `envoyExtAuthzHttp` extension,
make the following changes to the `example/kustomization.yaml` file:
* use `oauth2-proxy` overlay for istio-install
```
# from
- ../common/istio-1-22/istio-install/base
# to
- ../common/istio-1-22/istio-install/overlays/oauth2-proxy
```
* change `OIDC Authservice` to `oauth2-proxy for OIDC` and use overlay for m2m
bearer tokens with self-signed in-cluster issuer
```
# from
- ../common/oidc-client/oidc-authservice/base
# to
- ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed
```
* change Dex overlay
```
# from
- ../common/dex/overlays/istio
# to
- ../common/dex/overlays/oauth2-proxy
* change Central Dashboard overlay to use oauth2-proxy for logout
```
# from
- ../apps/centraldashboard/upstream/overlays/kserve
# to
- ../apps/centraldashboard/upstream/overlays/oauth2-proxy
```
All those changes combined can be done with this single command:
```diff
$ git apply <<EOF
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
index c1a85789..4a50440c 100644
--- a/example/kustomization.yaml
+++ b/example/kustomization.yaml
@@ -38,11 +38,11 @@ resources:
# Istio
- ../common/istio-1-22/istio-crds/base
- ../common/istio-1-22/istio-namespace/base
-- ../common/istio-1-22/istio-install/base
-# OIDC Authservice
-- ../common/oidc-client/oidc-authservice/base
+- ../common/istio-1-22/istio-install/overlays/oauth2-proxy
+# oauth2-proxy for OIDC
+- ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed
# Dex
-- ../common/dex/overlays/istio
+- ../common/dex/overlays/oauth2-proxy
# KNative
- ../common/knative/knative-serving/overlays/gateways
- ../common/knative/knative-eventing/base
@@ -60,7 +60,7 @@ resources:
# Katib
- ../apps/katib/upstream/installs/katib-with-kubeflow
# Central Dashboard
-- ../apps/centraldashboard/upstream/overlays/kserve
+- ../apps/centraldashboard/upstream/overlays/oauth2-proxy
# Admission Webhook
- ../apps/admission-webhook/upstream/overlays/cert-manager
# Jupyter Web App
EOF
```
@@ -0,0 +1,24 @@
# oauth2-proxy
## `oauth2-proxy` Deployment
This deployment of `oauth2-proxy` has been configured to align closely with the official
`oauth2-proxy` Helm installation. This approach facilitates easier integration with any
existing `oauth2-proxy` deployments that may already be present on the cluster.
### Upgrading `oauth2-proxy`
The `oauth2-proxy` component is designed for easy upgrading, thanks to its foundation on the
official `oauth2-proxy` Helm chart. The use of the standard Helm chart simplifies the upgrade
process, closely following the upgrades of the official `oauth2-proxy` releases.
### Stateless Nature of `oauth2-proxy`
`oauth2-proxy` operates as a stateless application. This statelessness simplifies many
aspects of its operation, particularly upgrades, as there are no concerns about complex state
management or data migration. Additionally, while `oauth2-proxy` is integrated into the
Kubernetes environment, this integration is limited to running the application, thereby
minimizing the impact on Kubernetes infrastructure during upgrades.
These characteristics make the upgrade process for `oauth2-proxy` more predictable and
manageable in Kubernetes environments.
@@ -0,0 +1,85 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
labels:
app: oauth2-proxy
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: oauth2-proxy
template:
metadata:
labels:
app.kubernetes.io/name: oauth2-proxy
spec:
volumes:
- name: configmain
configMap:
name: oauth2-proxy
defaultMode: 420
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:latest
args:
- --http-address=0.0.0.0:4180
- --config=/etc/oauth2_proxy/oauth2_proxy.cfg
ports:
- name: http
containerPort: 4180
protocol: TCP
- name: metrics
containerPort: 44180
protocol: TCP
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
- name: OAUTH2_PROXY_COOKIE_SECURE
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: FORCE_HTTPS
- name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: ALLOW_SELF_SIGNED_ISSUER
- name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: ENABLE_M2M_TOKENS
- name: OAUTH2_PROXY_EXTRA_JWT_ISSUERS
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: EXTRA_JWT_ISSUERS
volumeMounts:
- name: configmain
mountPath: /etc/oauth2_proxy/oauth2_proxy.cfg
subPath: oauth2_proxy.cfg
livenessProbe:
httpGet:
path: /ping
port: http
scheme: HTTP
readinessProbe:
httpGet:
path: /ping
port: http
scheme: HTTP
resources: {}
@@ -0,0 +1,64 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: oauth2-proxy
resources:
- namespace.yaml
- deployment.yaml
- serviceaccount.yaml
- service.yaml
- oauth2-proxy-config.yaml
# To use oauth2-proxy in OpenShift, use openshift distribution of oauth2-proxy
# available here: https://github.com/openshift/oauth-proxy
# and add RBAC to enable creation of tokenreviews
# Token Reviews for OpenShift
# - rbac.tokenreviews.yaml
secretGenerator:
- name: oauth2-proxy
type: Opaque
envs:
- secrets.env
configMapGenerator:
- name: oauth2-proxy-parameters
literals:
# This will configure oauth2-proxy option --cookie-secure which can force
# auth redirect with redirect_uri parameter using https.
- FORCE_HTTPS=false
# If Kubernetes is managed by kind, vCluster, minikube or similar tool,
# most probably the Kubernetes OIDC Issuer will be server in-cluster
# behind self-signed certs. This option will configure
# --ssl-insecure-skip-verify which will accept self-signed-certificates.
- ALLOW_SELF_SIGNED_ISSUER=true
# This will configure oauth2-proxy option --skip-jwt-bearer-tokens which
# will pass the requests with 'Authorization' header with Bearer Token
# matching --extra-jwt-issuers.
- ENABLE_M2M_TOKENS=true
# Bearer tokens issued by these OIDC Issuers will be verified against the
# Issuer and accepted on success.
# The format is exactly like in --extra-jwt-issuers.
# If extra jwt issuers are meant to be used with m2m bearer tokens,
# each issuer has to be added with RequestAuthentication so Istio can
# verify, trust and use the JWT. See
# 'common/oidc-client/oauth2-proxy/components/istio-m2m' for details.
# Examples:
# - EXTRA_JWT_ISSUERS=https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local
# - EXTRA_JWT_ISSUERS=https://oidc.eks.region.amazonaws.com/id/1234abcd=https://kubernetes.default.svc
- EXTRA_JWT_ISSUERS=
patches:
- target:
kind: ConfigMap
name: oauth2-proxy
path: patch-configmap.yaml
images:
- name: quay.io/oauth2-proxy/oauth2-proxy
newName: quay.io/oauth2-proxy/oauth2-proxy
newTag: v7.6.0
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
@@ -0,0 +1,57 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: oauth2-proxy
labels:
app: oauth2-proxy
data:
oauth2_proxy.cfg: |
provider = "oidc"
oidc_issuer_url = "http://dex.auth.svc.cluster.local:5556/dex"
scope = "profile email groups openid"
upstreams = "static://200"
email_domains = [ "*" ]
skip_auth_regex=["/dex/.*"]
# ---
# OIDC Discovery has to be skipped and login url has to be provided directly
# in order to enable relative auth redirect.
# Turning On OIDC Discovery would set the auth redirect location as the dex
# Issuer URL which is http://dex.auth.svc.cluster.local:5556 in the default,
# example installation. This address is usuallynot available through the Web
# Browser. If you have a setup where dex has it's url as other than the
# in-cluster service, this is optional.
skip_oidc_discovery = true
login_url = "/dex/auth"
redeem_url = "http://dex.auth.svc.cluster.local:5556/dex/token"
oidc_jwks_url = "http://dex.auth.svc.cluster.local:5556/dex/keys"
# ---
# Go to dex login page directly instead of showing the oauth2-proxy login
# page.
skip_provider_button = true
# ---
# Set Authorization Bearer response header. This is needed in order to
# forward the Authorization Bearer token to Istio and enable authorization
# based on JWT.
set_authorization_header = true
# ---
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
# may have other services that use standard auth headers.
set_xauthrequest = true
# ---
cookie_name = "oauth2_proxy_kubeflow"
# ---
# Dex default cookie expiration is 24h. If set to 168h (default oauth2-proxy),
# Istio will not be able to use the JWT after 24h but oauth2-proxy will still
# consider the cookie valid.
# It's possible to configure the JWT Refresh Token to enable longer login
# session.
cookie_expire = "24h"
cookie_refresh = 0
# ---
code_challenge_method = "S256"
# ---
redirect_url = "/oauth2/callback"
relative_redirect_url = true
binaryData: {}
@@ -0,0 +1,5 @@
varReference:
- path: spec/configPatches/patch/value/typed_config/http_service/server_uri/uri
kind: EnvoyFilter
- path: spec/configPatches/patch/value/typed_config/http_service/server_uri/cluster
kind: EnvoyFilter
@@ -0,0 +1,53 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: oauth2-proxy
labels:
app: oauth2-proxy
data:
oauth2_proxy.cfg: |
provider = "keycloak-oidc"
oidc_issuer_url = ""
scope = "profile email roles openid"
upstreams = "static://200"
email_domains = [ "*" ]
skip_auth_regex=["/dex/.*"]
# ---
# OIDC Discovery has to be skipped and login url has to be provided directly
# in order to enable relative auth redirect.
# Turning On OIDC Discovery would set the auth redirect location as the dex
# Issuer URL which is http://dex.auth.svc.cluster.local:5556 in the default,
# example installation. This address is usuallynot available through the Web
# Browser. If you have a setup where dex has it's url as other than the
# in-cluster service, this is optional.
# ---
# Go to dex login page directly instead of showing the oauth2-proxy login
# page.
skip_provider_button = true
# ---
# Set Authorization Bearer response header. This is needed in order to
# forward the Authorization Bearer token to Istio and enable authorization
# based on JWT.
set_authorization_header = true
# ---
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
# may have other services that use standard auth headers.
set_xauthrequest = true
# ---
cookie_name = "oauth2_proxy_kubeflow"
# ---
# Dex default cookie expiration is 24h. If set to 168h (default oauth2-proxy),
# Istio will not be able to use the JWT after 24h but oauth2-proxy will still
# consider the cookie valid.
# It's possible to configure the JWT Refresh Token to enable longer login
# session.
cookie_expire = "24h"
cookie_refresh = "5m"
# ---
code_challenge_method = "S256"
# ---
redirect_url = "/oauth2/callback"
relative_redirect_url = true
binaryData: {}
@@ -0,0 +1,24 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: authn-delegator
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: authn-delegators
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authn-delegator
subjects:
- kind: ServiceAccount
name: oauth2-proxy
@@ -0,0 +1,3 @@
client-id=kubeflow-oidc-authservice
client-secret=5kD2WwXGUYQ3lEUpy6dkTIVXHpVeemuA
cookie-secret=094f9651100c4ee4a3a7337e405d8650
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: oauth2-proxy
ports:
- port: 80
name: http
targetPort: http
publishNotReadyAddresses: true
@@ -0,0 +1,4 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: oauth2-proxy
@@ -0,0 +1,43 @@
# Unauthenticated Issuer Discovery
If you are using Kubernetes with tools like kind, vCluster, minikube, or similar solutions for local
development, it's highly likely that the Kubernetes OIDC Issuer operates within the cluster and is
secured with self-signed certificates.
To facilitate the use of m2m tokens with `oauth2-proxy` and Istio, both tools must perform OIDC
Connect Discovery on the Token Issuer. Kubernetes offers OIDC Discovery functionality at a specific
URI:
```
https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
```
Access to this endpoint is blocked by default due to RBAC policies, but it can be enabled by
creating a `ClusterRoleBinding`. This binding associates the predefined `ClusterRole`
`system:service-account-issuer-discovery` with the `system:unauthenticated` group. The
configuration for this is detailed in the resource file `clusterrolebinding.unauthenticated-oidc-viewer.yaml`.
Once this step is completed, the endpoint can be accessed to reveal the Issuer URL:
```bash
$ curl -k https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
# Example output in kind:
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://172.18.0.5:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
# Example output in vCluster:
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://1.2.3.4:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
# Example output in AWS EKS:
{"issuer":"https://oidc.eks.region.amazonaws.com/id/123abc","jwks_uri":"https://ip-1-2-3-4.eu-central-1.compute.internal:443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
```
If you're operating in a `vCluster`, access to the endpoint specified under `jwks_uri` (for example,
`https://1.2.3.4:6443/openid/v1/jwks`) is managed separately, and merely creating the previously
mentioned `ClusterRoleBinding` is insufficient. To circumvent this limitation, you can configure the
`kube-apiserver` to allow anonymous authentication by setting `--anonymous-auth=true`. This is
achieved by appending `--kube-apiserver-arg=anonymous-auth=true` to the list of arguments in the
Helm Chart Values file:
```yaml
vcluster:
extraArgs:
- --kube-apiserver-arg=anonymous-auth=true
```
@@ -0,0 +1,12 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: unauthenticated-oidc-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:unauthenticated
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- clusterrolebinding.unauthenticated-oidc-viewer.yaml
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/deployment.logout-url.yaml
@@ -0,0 +1,19 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: centraldashboard
namespace: kubeflow
spec:
template:
spec:
containers:
- name: centraldashboard
env:
# This successfully logs out the user but the user is not redirected to
# the home page. This is because of custom integration with
# oidc-authservice which will provide response with 'afterLogoutURL'.
# https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509
# https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50
# Please refresh the page after logging out.
- name: LOGOUT_URL
value: /oauth2/sign_out
@@ -0,0 +1,41 @@
# Configure Istio with Self-Signed Kubernetes OIDC Issuer
## Overview
This kustomize component is designed for scenarios where the Kubernetes OIDC Issuer is behind
self-signed certificates, causing trust issues with Istio in retrieving the JWKS. It creates a
Kubernetes Job and necessary RBAC configurations to address this issue. The job facilitates the
retrieval of JWKS from the OIDC Issuer and embeds it directly into Istio's configuration, thereby
bypassing trust issues with self-signed certificates.
## Configuration Persistence and JWKS Public Accessibility
The configuration created by this job is stored exclusively in etcd and is not persisted elsewhere.
This setup is compatible with ArgoCD, which by default does not delete properties absent in the
desired manifest. However, if the `RequestAuthentication` resource is modified erroneously or
deleted, the configuration would be lost, necessitating a rerun of the job.
To circumvent the need for rerunning the job, it is advisable to store the JWKS in a repository.
JWKS are typically publicly accessible and contain no sensitive information, making them safe for
repository storage. By persisting the JWKS in a repository, consistent access is ensured regardless
of changes in the cluster or accidental deletions.
## Functionality
- **Reading OIDC Issuer URL**: The Job reads the `RequestAuthentication` resource to extract the
OIDC Issuer URL.
- **Fetching JWKS**: After identifying the Issuer URL, the Job retrieves the JWKS.
- **Patching RequestAuthentication**: The JWKS is used to patch the `RequestAuthentication`
resource, embedding the JWKS directly.
- **Static JWKS Configuration for Istio**: Ensures that Istio uses the JWKS provided by the Job
instead of requesting it independently.
## Use Case
This setup is particularly useful when:
- Kubernetes serves as an OIDC provider.
- The Kubernetes API is not served with publicly trusted certificates.
This component ensures seamless M2M authentication by handling JWKS retrieval and configuration
internally, thus circumventing certificate validation issues for Istio fetching JWKS from an OIDC
provider with self-signed or private CA certificates.
@@ -0,0 +1,39 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: kubeflow-m2m-oidc-configurator
namespace: istio-system
spec:
schedule: '*/5 * * * *'
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 3
ttlSecondsAfterFinished: 600
template:
metadata:
labels: {}
spec:
restartPolicy: OnFailure
serviceAccountName: kubeflow-m2m-oidc-configurator
containers:
- image: docker.io/curlimages/curl
name: kubeflow-m2m-oidc-configurator
command:
- /script.sh
envFrom:
- configMapRef:
name: kubeflow-m2m-oidc-configurator-envs
volumeMounts:
- mountPath: /script.sh
name: script
subPath: script.sh
resources: {}
volumes:
- name: script
configMap:
name: kubeflow-m2m-oidc-configurator-script
defaultMode: 0777
items:
- key: script.sh
path: script.sh
@@ -0,0 +1,24 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- cronjob.kubeflow-m2m-oidc-configurator.yaml
- rbac.yaml
configMapGenerator:
- name: kubeflow-m2m-oidc-configurator-script
namespace: istio-system
files:
- script.sh=script.sh
- name: kubeflow-m2m-oidc-configurator-envs
namespace: istio-system
literals:
- ISTIO_ROOT_NAMESPACE=istio-system
- REQUEST_AUTHENTICATION_NAME=m2m-token-issuer
- KUBERNETES_API_SERVER_URL=https://kubernetes.default.svc
- name: oauth2-proxy-parameters
behavior: merge
literals:
- ALLOW_SELF_SIGNED_ISSUER=true
@@ -0,0 +1,35 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubeflow-m2m-oidc-configurator
namespace: istio-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubeflow-m2m-oidc-configurator
namespace: istio-system
rules:
- apiGroups:
- security.istio.io
resources:
- requestauthentications
verbs:
- get
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubeflow-m2m-oidc-configurator
namespace: istio-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubeflow-m2m-oidc-configurator
subjects:
- kind: ServiceAccount
name: kubeflow-m2m-oidc-configurator
namespace: istio-system
@@ -0,0 +1,143 @@
#!/bin/sh
set -e
KUBERNETES_API_SERVER_URL="${KUBERNETES_API_SERVER_URL:-https://kubernetes.default.svc}"
ISTIO_ROOT_NAMESPACE="${ISTIO_ROOT_NAMESPACE:-istio-system}"
REQUEST_AUTHENTICATION_NAME="${REQUEST_AUTHENTICATION_NAME:-m2m-token-issuer}"
RESOURCE_URL="\
${KUBERNETES_API_SERVER_URL}\
/apis/security.istio.io/v1/namespaces/\
${ISTIO_ROOT_NAMESPACE}\
/requestauthentications/\
${REQUEST_AUTHENTICATION_NAME}"
wait_for_resource_ready() {
while true; do
response="$(
curl -s -o /dev/null \
--url "${RESOURCE_URL}" \
-w "%{http_code}" \
--header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \
--insecure
)"
if [ "${response}" = "200" ]; then
break
fi
sleep 5
done
}
get_request_authentication_obj() {
curl -s --request GET \
--url "${RESOURCE_URL}" \
--header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \
--insecure
}
get_issuer_url_from_obj() {
obj="${1}"
echo "${obj}" | awk -F'"' '/"issuer":/ { print $4 }'
}
get_current_escaped_jwks_from_obj() {
obj="${1}"
echo "${obj}" | awk -F'"' '/"jwks":/' | sed -n 's/^.*"jwks": "\(.*\)".*$/\1/p'
}
get_jwks_uri() {
issuer_url="${1}"
curl -s --request GET \
--url "${issuer_url}/.well-known/openid-configuration" \
--header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \
--insecure |
grep -o '"jwks_uri":"https:\/\/[^"]\+"' |
sed 's/"jwks_uri":"\(.*\)"/\1/'
}
get_jwks_from_uri() {
jwks_uri="${1}"
curl -s --request GET \
--url "${jwks_uri}" \
--header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \
--insecure
}
# Format JWKS in a way that can be accepted in resource patch.
parse_escaped_jwks() {
jwks="${1}"
echo "${jwks}" | sed 's/"/\\"/g'
}
are_jwks_equal() {
jwks1="${1}"
jwks2="${2}"
test "$(echo "${jwks1}" | base64 -w0)" = "$(echo "${jwks2}" | base64 -w0)"
}
patch_request_authentication_with_escaped_jwks() {
jwks_escaped="${1}"
curl -s --request PATCH \
--url "${RESOURCE_URL}" \
--header "Content-Type: application/json-patch+json" \
--header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \
-d '[{ "op": "add", "path": "/spec/jwtRules/0/jwks", "value": "'"${jwks_escaped}"'" }]' \
--insecure
echo
}
patch_request_authentication_with_jwks_if_required() {
echo "Getting RequestAuthentication object."
REQUEST_AUTHENTICATION_OBJ="$(get_request_authentication_obj)"
ISSUER_URL="$(get_issuer_url_from_obj "${REQUEST_AUTHENTICATION_OBJ}")"
echo "Issuer Url in RequestAuthentication: ${ISSUER_URL}"
CURRENT_JWKS_ESCAPED="$(get_current_escaped_jwks_from_obj "${REQUEST_AUTHENTICATION_OBJ}")"
printf "Current Jwks (escaped):\n%s\n" "${CURRENT_JWKS_ESCAPED}"
JWKS_URI="$(get_jwks_uri "${ISSUER_URL}")"
echo "Jwks Uri from Well Known OpenID Configuration: ${JWKS_URI}"
JWKS="$(get_jwks_from_uri "${JWKS_URI}")"
JWKS_ESCAPED="$(parse_escaped_jwks "${JWKS}")"
printf "JWKS from Well Known OpenID Configuration (escaped): \n%s\n" "${JWKS_ESCAPED}"
if are_jwks_equal "${JWKS_ESCAPED}" "${CURRENT_JWKS_ESCAPED}"; then
echo "JWKS in RequestAuthentication ${REQUEST_AUTHENTICATION_NAME} is configured correctly."
else
echo "JWKS in RequestAuthentication ${REQUEST_AUTHENTICATION_NAME} needs to be configured."
patch_request_authentication_with_escaped_jwks "${JWKS_ESCAPED}"
fi
}
verify_jwks_in_request_authentication() {
REQUEST_AUTHENTICATION_OBJ="$(get_request_authentication_obj)"
ISSUER_URL="$(get_issuer_url_from_obj "${REQUEST_AUTHENTICATION_OBJ}")"
CURRENT_JWKS_ESCAPED="$(get_current_escaped_jwks_from_obj "${REQUEST_AUTHENTICATION_OBJ}")"
JWKS_URI="$(get_jwks_uri "${ISSUER_URL}")"
JWKS="$(get_jwks_from_uri "${JWKS_URI}")"
JWKS_ESCAPED="$(parse_escaped_jwks "${JWKS}")"
if ! are_jwks_equal "${JWKS_ESCAPED}" "${CURRENT_JWKS_ESCAPED}"; then
echo "JWKS not properly configured, exit with error code 1"
exit 1
fi
}
main() {
echo "Wait until resource RequestAuthentication ${REQUEST_AUTHENTICATION_NAME} in namespace ${ISTIO_ROOT_NAMESPACE} is ready."
wait_for_resource_ready
echo "Resource RequestAuthentication ${REQUEST_AUTHENTICATION_NAME} in namespace ${ISTIO_ROOT_NAMESPACE} is ready."
echo "Patch RequestAuthentication with JWKS if required."
patch_request_authentication_with_jwks_if_required
echo "Wait 5 seconds before verifying RequestAuthentication JWKS configuration."
sleep 5
echo "Verify if RequestAuthentication is properly configured with JWKS..."
verify_jwks_in_request_authentication
echo "RequestAuthentication is properly configured with JWKS."
}
main
@@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/cm.enable-oauth2-proxy.yaml
- path: patches/deployment.jwt-refresh-interval.yaml
@@ -0,0 +1,36 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
namespace: istio-system
data:
mesh: |-
accessLogFile: /dev/stdout
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing: {}
enablePrometheusMerge: true
rootNamespace: istio-system
tcpKeepalive:
interval: 5s
probes: 3
time: 10s
trustDomain: cluster.local
extensionProviders:
- envoyExtAuthzHttp:
headersToDownstreamOnDeny:
- content-type
- set-cookie
headersToUpstreamOnAllow:
- authorization
- path
- x-auth-request-email
- x-auth-request-groups
- x-auth-request-user
includeRequestHeadersInCheck:
- authorization
- cookie
service: oauth2-proxy.oauth2-proxy.svc.cluster.local
port: 80
name: oauth2-proxy
@@ -0,0 +1,18 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: istiod
namespace: istio-system
spec:
template:
spec:
containers:
- name: discovery
env:
# Istio will refresh the JWT Public Keys provided with
# RequestAuthentication by default every 20 minutes. For deployment
# from scratch this is not ideal because there is a high chance that
# the istiod will be available before dex is available, triggering
# Istio mechanism to use a placeholder jwt until refreshed.
- name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL
value: "1m"
@@ -0,0 +1,23 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
- to:
- operation:
notPaths:
- /favicon*
- /webcomponentsjs*
- /vendor.bundle.js
- /app.bundle.js
- /dashboard_lib.bundle.js
- /assets*
- /app.css
@@ -0,0 +1,14 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
- {}
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
# If you're running Kubeflow behind CloudFlare, use
# authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml
# instead of
# authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
@@ -0,0 +1,19 @@
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: dex-jwt
namespace: istio-system
spec:
jwtRules:
# forwardOriginalToken must be set to true so the authorization header will
# be passed between Kubeflow Components and Istio can configure the
# Kubeflow Auth Headers based on this request authorization header.
- forwardOriginalToken: true
issuer: http://dex.auth.svc.cluster.local:5556/dex
# These 5 lines provides integration of istio/oauth2-proxy with
# Kubeflow custom auth headers.
outputClaimToHeaders:
- header: kubeflow-userid
claim: email
- header: kubeflow-groups
claim: groups
@@ -0,0 +1,19 @@
# Kubernetes M2M Authentication with Istio and RequestAuthentication
## Overview
This kustomize component enables M2M (Machine-to-Machine) authentication in Kubernetes, using
Istio and the `RequestAuthentication` object. It configures Istio to trust JWTs (JSON Web Tokens)
in Authorization Bearer tokens when the JWT issuer matches the one in `RequestAuthentication`. The
default setup uses Kubernetes' self-served OIDC issuer with self-signed certificates.
In Kubernetes clusters managed by platform providers, the OIDC issuer is usually managed by the
provider and served behind publicly trusted certificates. In these cases, it's advisable to use
the platform-managed Kubernetes OIDC issuer in the `RequestAuthentication` for seamless integration
and authentication compliance with the platform's security standards.
For scenarios where the OIDC issuer is served behind self-signed certificates, the kustomize
overlay using this component should include the `common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer`
component. This additional configuration is necessary to handle the self-signed nature of the
certificates. This setup is the default in the Kustomize overlay defined in `common/oidc-client/oauth2-proxy/overlays/m2m-self-signed`,
which is tailored for environments with self-signed OIDC issuers.
@@ -0,0 +1,29 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- requestauthentication.yaml
configMapGenerator:
- name: istio-m2m-params
envs:
- m2m.env
replacements:
- source:
kind: ConfigMap
version: v1
name: istio-m2m-params
fieldPath: data.M2M_TOKEN_ISSUER
targets:
- fieldPaths:
- spec.jwtRules.0.issuer
select:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: m2m-token-issuer
namespace: istio-system
configurations:
- params.yaml
@@ -0,0 +1 @@
M2M_TOKEN_ISSUER=issuer
@@ -0,0 +1,21 @@
apiVersion: v1
data:
M2M_TOKEN_ISSUER: issuer
kind: ConfigMap
metadata:
name: istio-m2m-params-gfd5m8bd92
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: m2m-token-issuer
namespace: istio-system
spec:
jwtRules:
- forwardOriginalToken: true
issuer: issuer
outputClaimToHeaders:
- claim: sub
header: x-auth-request-user
- claim: sub
header: kubeflow-userid
@@ -0,0 +1,3 @@
varReference:
- path: spec/jwtRules/issuer
kind: RequestAuthentication
@@ -0,0 +1,14 @@
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: m2m-token-issuer
namespace: istio-system
spec:
jwtRules:
- forwardOriginalToken: true
issuer: M2M_TOKEN_ISSUER_PLACEHOLDER
outputClaimToHeaders:
- claim: sub
header: x-auth-request-user
- claim: email
header: kubeflow-userid
@@ -0,0 +1,19 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/istio-m2m
- ../../components/istio-external-auth
- ../../components/allow-unauthenticated-issuer-discovery
- ../../components/configure-self-signed-kubernetes-oidc-issuer
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
- ALLOW_SELF_SIGNED_ISSUER=true
- ENABLE_M2M_TOKENS=true
- EXTRA_JWT_ISSUERS=https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local
@@ -0,0 +1,27 @@
# Kustomize Overlay for M2M Authentication Integration with Istio
## Overview
This kustomize overlay facilitates the integration of M2M (Machine-to-Machine) authentication
with Istio in a Kubernetes environment. It includes components designed to enable effective
authentication within the service mesh.
## Components
The overlay consists of:
1. **`istio-m2m`**: Configures Istio to trust JWTs in M2M communication, essential for enabling
M2M authentication within the service mesh.
2. **`component-overwrite-m2m-token-issuer`**: This component is used when the OIDC issuer used
by Kubernetes is external to the cluster. It allows for the modification of the default
issuer URL in the M2M authentication setup to match the externally defined OIDC issuer. This
adjustment is necessary due to kustomize's handling of config map generators, requiring a
separate component for proper configuration merging.
## Usage Scenario
- **External OIDC Issuer Integration**: In cases where the OIDC Issuer integrated with Kubernetes
is defined outside the cluster, `component-overwrite-m2m-token-issuer` is used to update the
issuer URL for M2M authentication. This scenario is common in setups where an external OIDC
issuer is preferred over the default Kubernetes self-served OIDC issuer.
@@ -0,0 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
- name: istio-m2m-params
behavior: merge
literals:
- M2M_TOKEN_ISSUER=https://oidc.eks.region.amazonaws.com/id/1234abcd
@@ -0,0 +1,18 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/istio-m2m
- ../../components/istio-external-auth
- component-overwrite-m2m-token-issuer
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
- FORCE_HTTPS=true
- ENABLE_M2M_TOKENS=true
- EXTRA_JWT_ISSUERS=https://oidc.eks.region.amazonaws.com/id/1234abcd=https://kubernetes.default.svc