Change chart directory structure

This commit is contained in:
wbsong111
2026-01-19 16:11:44 +09:00
parent 0f2284bf35
commit 0436749932
5948 changed files with 119 additions and 79 deletions
@@ -0,0 +1,3 @@
approvers:
- axel7083
- kromanow94
@@ -0,0 +1,308 @@
# Kubeflow Authentication using Oauth2 Proxy
## Istio envoyExtAuthzHttp
This is Istio's recommended approach for External Authorization[^2]. It is not limited to the use
of `oauth2-proxy`[^3] alone. This method is an industry standard, meeting all of Kubeflow's
current and foreseeable authentication needs.
## Kubeflow Pipelines User and M2M Authentication and Authorization
The Kubeflow Pipelines component relies on the built-in kubernetes functionalities to authenticate and authorize
user requests, specifically the TokenReviews[^4] and SubjectAccessReview[^5].
The best way to describe how it works is to explain with an example. Lets analyze the flow
when a client calls the API to list the KF Pipeline runs:
1. api-server starts endpoints in:
https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L95
Focusing on the pipelines run service:
1. Register Run Service:
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L116
2. proto RPC definition of ListRunsV1
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/api/v1beta1/run.proto#L80
3. code definition of ListRunsV1
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L226
4. ListRunsV1 calls internal method `listRuns`
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L188
5. `listRuns` calls internal method `canAccessRun` which itself calls `s.resourceManager.IsAuthorized`
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L637
6. `ResourceManager.IsAuthorized` first tries to authenticate over every available authenticator, which are the `TokenReviewAuthenticator` and `HTTPHeaderAuthenticator`
* here the user identity is either the user email provided directly in the `kubeflow-userid` header or the user identity obtained from provided token
* https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/resource/resource_manager.go#L1667
7. `TokenReviewAuthenticator.GetUserIdentity` gets the token from `Authorization` header and calls the K8s Auth `authv1.TokenReview` with given token which in return provides `userInfo := review.Status.User`. `GetUserIdentity` return `userInfo.Username` which at this point is the `system:serviceaccount:default:default`.
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/auth/authenticator_token_review.go#L53
8. Next in `ResourceManager.IsAuthorized` a SubjectAccessReview is created with `r.subjectAccessReviewClient.Create` with arguments specifying RBAC verbs provided in code definition of `RunServer.listRuns`. If the user (sa) is not authorized, an error is thrown
* https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/resource/resource_manager.go#L1703
* if the identity was obtained from token (service account), the `rolebinding.rbac.authorization.k8s.io/default-editor` provides the RBAC permission
* if the identity was obtained from header (user), the `rolebinding.rbac.authorization.k8s.io/user-example-com` or similar provides the RBAC permission
2. User calls api to list pipeline runs as unauthorized service account.
* This can be done by running Pod with curl in `default` namespace:
```bash
$ kubectl -n default run -ti --rm curl --image curlimages/curl --command -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
```
3. User calls api to list pipeline runs as authorized service account.
* This can be done by running Pod with curl in `kubeflow-user-example-com` namespace specifying correct service account:
```bash
$ kubectl -n kubeflow-user-example-com run -ti --rm curl --image curlimages/curl --command --overrides='{"spec": {"serviceAccountName": "default-editor"}}' -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{} # empty response which is fine because no pipeline runs exist
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{} # empty response which is fine because no pipeline runs exist
```
### Authentication and Authorization analysis diagram for Kubeflow Pipelines
![Kubeflow Auth Diagram](./components/kubeflow_auth_diagram.svg)
### Change the default authentication from "Dex + Oauth2-proxy" to "Oauth2-proxy" only
The authentication in Kubeflow evolved over time and we dropped envoyfilters and oidc-authservice in favor of RequestAuthentication and Oauth2-proxy in Kubeflow 1.9.
![auth-flow](components/oauth2-flow.svg)
You can adjust OAuth2 Proxy to directly connect to your own IDP(Identity Provider) suchg as GCP, [AWS](https://docs.aws.amazon.com/cognito/latest/developerguide/federation-endpoints-oauth-grants.html), Azure etc:
1. Create an application on your IdP (purple line)
2. Change your [OAuth2 Proxy issuer](https://github.com/kubeflow/manifests/blob/35539f162ea7fafc8c5035d8df0d8d8cf5a9d327/common/oauth2-proxy/base/oauth2-proxy-config.yaml#L10) to your IdP. Of course never ever directly, but with kustomize overlays and components.
Here is an example of patching oauth2-proxy to connect directly to Azure IDP and skip Dex.
This is enterprise integration so feel free to hire consultants or pay for commercial distributions if you need more help.
For example Azure returns rather large headers compared to other IDPs, so maybe you need to annotate the nginx-ingress to support that.
```
# based on https://github.com/kubeflow/manifests/blob/master/common/oauth2-proxy/base/oauth2_proxy.cfg
# and https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/azure/
apiVersion: v1
kind: ConfigMap
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
data:
oauth2_proxy.cfg: |
provider = "oidc"
oidc_issuer_url = "https://login.microsoftonline.com/$MY_TENANT/v2.0"
scope = "openid email offline_access" # removed groups and profile to decrease the size and added offline_access to enable refresh tokens
email_domains = [ "*" ]
# serve a static HTTP 200 upstream on for authentication success
# we are using oauth2-proxy as an ExtAuthz to "check" each request, not pass it on
upstreams = [ "static://200" ]
# skip authentication for these paths
skip_auth_routes = [
"^/dex/",
]
# requests to paths matching these regex patterns will receive a 401 Unauthorized response
# when not authenticated, instead of being redirected to the login page with a 302,
# this prevents background requests being redirected to the login page,
# and the accumulation of CSRF cookies
api_routes = [
# Generic
# NOTE: included because most background requests contain these paths
"/api/",
"/apis/",
# Kubeflow Pipelines
# NOTE: included because KFP UI makes MANY background requests to these paths but because they are
# not `application/json` requests, oauth2-proxy will redirect them to the login page
"^/ml_metadata",
]
skip_provider_button = true
set_authorization_header = true
set_xauthrequest = true
cookie_name = "oauth2_proxy_kubeflow"
cookie_expire = "24h"
cookie_refresh = "59m" # This improves the user experience a lot
redirect_url = "https://$MY_PUBLIC_KUBEFLOW_DOMAIN/oauth2/callback"
relative_redirect_url = false
```
3. In the istio-system namespace is a RequestAuthentication resource. You need to change its issuer to your own IdP, or even better create an additional one.
```
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: azure-aad-requestauthentication
namespace: istio-system
spec:
# we only apply to the ingress-gateway because:
# - there is no need to verify the same tokens at each sidecar
# - having no selector will apply to the RequestAuthentication to ALL
# Pods in the mesh, even ones which are not part of Kubeflow
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- issuer: https://login.microsoftonline.com/$MY_TENANT/v2.0
# `forwardOriginalToken` is not strictly required to be true.
# there are pros and cons to each value:
# - true: the original token is forwarded to the destination service
# which raises the risk of the token leaking
# - false: the original token is stripped from the request
# which will prevent the destination service from
# verifying the token (possibly with its own RequestAuthentication)
forwardOriginalToken: true
# This will unpack the JWTs issued by Dex or other IDPs into the expected headers.
# It is applied to BOTH the m2m tokens from outside the cluster (which skip
# oauth2-proxy because they already have a dex JWT), AND user requests which were
# authenticated by oauth2-proxy (which injected a dex JWT).
outputClaimToHeaders:
- header: kubeflow-userid
claim: email
- header: kubeflow-groups
claim: groups
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
fromHeaders:
- name: Authorization
prefix: "Bearer "
```
You can also add more RequestAuthentication to support other issuers as for example for M2M access from github actions as explained in the root level Readme.md.
This feature is useful when you need to integrate Kubeflow with your current CI/CD platform (GitHub Actions, Jenkins) via machine-to-machine authentication.
The following is an example for obtaining and using a JWT token From your IDP with Python, but you can also just take a look at our CI/CD test that uses simple Kubernetes serviceaccount tokens to access KFP, Jupyterlabs etc. from GitHub Actions.
```
import requests
token_url = "https://your-idp.com/oauth/token"
client_id = "YOUR_CLIENT_ID"
client_secret = "YOUR_CLIENT_SECRET"
username = "YOUR_USERNAME"
password = "YOUR_PASSWORD"
# request header
headers = {
"Content-Type": "application/x-www-form-urlencoded"
}
data = {
"grant_type": "password",
"client_id": client_id,
"client_secret": client_secret,
"username": username,
"password": password,
"scope": "openid profile email" #change your scope
}
response = requests.post(token_url, headers=headers, data=data)
TOKEN = response.json()['access_token']
```
```
import kfp
kubeflow_host="https://your_host"
pipeline_host = kubeflow_host + "/pipeline"
client = kfp.Client(host=pipeline_host, existing_token=TOKEN)
print(client.list_runs(namespace="your-profile-name"))
```
## Known Issues:
Some openidc providers such as Azure provide too large JWTs / Cookies that exceed the limit of most GRPC and gunicorn web application deployments in Kubeflow.
If removing the groups claim in oauth2-proxy is not enough then you can add an envrionment variable to all web applications
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: kserve-models-web-app
namespace: kubeflow
spec:
template:
spec:
containers:
- name: kserve-models-web-app # repeat for all other *-web-app-(deployment)
env:
- name: GUNICORN_CMD_ARGS
value: --limit-request-field_size 32000
```
and modify the KFP GRPC server via
```
- path: patches/metadata-grpc-virtualservice-patch.yaml
target:
kind: VirtualService
name: metadata-grpc
namespace: kubeflow
# patches/metadata-grpc-virtualservice-patch.yaml
# Remove the oauth2-proxy cookie that violates the maximum metadata size for a GRPC request
- op: add
path: /spec/http/0/route/0/headers
value:
request:
remove:
- Cookie
```
to fix `received initial metadata size exceeds limit`.
## Kubeflow Notebooks User and M2M Authentication and Authorization
The underlying mechanism is the same as in Kubeflow Pipelines.
Similarly, to explain how it works, let's analyze the code step by step, starting from the api route definition
for listing notebooks:
* list notebooks api route definition
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L53
* this calls `crud_backend/api/notebook.py::list_notebooks`
* `crud_backend/api/notebook.py::list_notebooks` calls `authz.ensure_authorized`
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/api/notebook.py#L24
* `crud_backend/authz.py::ensure_authorized` calls `crud_backend/authn.py::get_username`
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authz.py#L101
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authn.py#L12
* `crud_backend/authn.py::get_username` gets the user id from userid header (email or sa in format `system:serviceaccount:kubeflowusernamespace:default-editor`)
* `crud_backend/authz.py::ensure_authorized` calls `crud_backend/authz.py::is_authorized`
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authz.py#L46
* this calls `create_subject_access_review` which uses the same mechanism as pipelines with `r.subjectAccessReviewClient.Create`
## KServe Authentication
The analysis of KServe auth capabilities suggests that while it's possible to limit access to only authenticated agents,
there might be some improvements required to enable access only to authorized agents.
This is based on the following:
1. KServe Controller Manager patch integrating kube-rbac-proxy[^6].
This suggests the kserve **might** use the same mechanism based on
`SubjectAccessReviews`. Having a look at the kubeflow/manifests I see it's
not enabled.
2. Search through the docs and code:
* https://github.com/kserve/kserve/tree/v0.12.0/docs/samples/istio-dex
* https://github.com/kserve/kserve/tree/v0.12.0/docs/samples/gcp-iap
The docs above mention that while it's possible to enable authentication,
authorization is more complicated and probably we need to add
`AuthorizationPolicy`
> create an [Istio AuthorizationPolicy](https://istio.io/latest/docs/reference/config/security/authorization-policy/) to grant access to the pods or disable it
Most probably some work is needed to enable authorized access to kserve models.
## Links
[^1]: [Envoy Filter](https://istio.io/latest/docs/reference/config/networking/envoy-filter/)
[^2]: [External Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
[^3]: [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy)
[^4]: [Kubernetes TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/)
[^5]: [Kubernetes SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v3/)
[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy)
@@ -0,0 +1,24 @@
# oauth2-proxy
## `oauth2-proxy` Deployment
This deployment of `oauth2-proxy` has been configured to align closely with the official
`oauth2-proxy` Helm installation. This approach facilitates easier integration with any
existing `oauth2-proxy` deployments that may already be present on the cluster.
### Upgrading `oauth2-proxy`
The `oauth2-proxy` component is designed for easy upgrading, thanks to its foundation on the
official `oauth2-proxy` Helm chart. The use of the standard Helm chart simplifies the upgrade
process, closely following the upgrades of the official `oauth2-proxy` releases.
### Stateless Nature of `oauth2-proxy`
`oauth2-proxy` operates as a stateless application. This statelessness simplifies many
aspects of its operation, particularly upgrades, as there are no concerns about complex state
management or data migration. Additionally, while `oauth2-proxy` is integrated into the
Kubernetes environment, this integration is limited to running the application, thereby
minimizing the impact on Kubernetes infrastructure during upgrades.
These characteristics make the upgrade process for `oauth2-proxy` more predictable and
manageable in Kubernetes environments.
@@ -0,0 +1,100 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
labels:
app: oauth2-proxy
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: oauth2-proxy
template:
metadata:
labels:
app.kubernetes.io/name: oauth2-proxy
spec:
volumes:
- name: oauth2-proxy-config
configMap:
name: oauth2-proxy
- name: oauth2-proxy-theme
configMap:
name: oauth2-proxy-theme
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:latest
args:
- --http-address=0.0.0.0:4180
- --config=/etc/oauth2_proxy/oauth2_proxy.cfg
ports:
- name: http
containerPort: 4180
protocol: TCP
- name: metrics
containerPort: 44180
protocol: TCP
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
- name: OAUTH2_PROXY_COOKIE_SECURE
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: FORCE_HTTPS
- name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: ALLOW_SELF_SIGNED_ISSUER
- name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: ENABLE_M2M_TOKENS
- name: OAUTH2_PROXY_EXTRA_JWT_ISSUERS
valueFrom:
configMapKeyRef:
name: oauth2-proxy-parameters
key: EXTRA_JWT_ISSUERS
securityContext:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
capabilities:
drop:
- ALL
volumeMounts:
- name: oauth2-proxy-config
mountPath: /etc/oauth2_proxy/oauth2_proxy.cfg
subPath: oauth2_proxy.cfg
readOnly: true
- name: oauth2-proxy-theme
mountPath: /custom-theme/kubeflow-logo.svg
subPath: kubeflow-logo.svg
readOnly: true
livenessProbe:
httpGet:
path: /ping
port: http
scheme: HTTP
readinessProbe:
httpGet:
path: /ready
port: http
scheme: HTTP
resources: {}
@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="256" height="256" version="1.1" viewBox="0 0 256 256" xmlns="http://www.w3.org/2000/svg">
<g transform="matrix(.85002 0 0 .85002 10.306 11.309)" data-name="Layer 2">
<g data-name="Layer 1">
<path d="m95.9 62.15 4.1 102.1 73.75-94.12a6.79 6.79 0 0 1 9.6-1.11l46 36.92-15-65.61z" fill="#4279f4"/>
<path d="m102.55 182.98h65.42l-40.17-32.23z" fill="#0028aa"/>
<path d="m180.18 83.92-44 56.14 46.88 37.61 44.47-55.76z" fill="#014bd1"/>
<path d="m83.56 52.3 0.01-0.01 38.69-48.52-62.39 30.05-15.41 67.51z" fill="#bedcff"/>
<path d="m45.32 122.05 41.44 51.96-3.95-98.98z" fill="#6ca1ff"/>
<path d="m202.31 28.73-59.66-28.73-37.13 46.56z" fill="#a1c3ff"/>
<path d="m1.6 272v-44.78h5.74v23.41l20.48-23.41h6.4l-17.39 19.7 19 25.07h-6.73l-15.92-20.8-5.84 6.65v14.16zm40.02-9.79v-22.21h5.43v22.39a4.67 4.67 0 0 0 2.35 4.19 11 11 0 0 0 11 0 4.69 4.69 0 0 0 2.33-4.19v-22.39h5.43v22.19a9.08 9.08 0 0 1-4.1 7.87 16.2 16.2 0 0 1-18.37 0 9.07 9.07 0 0 1-4.07-7.85zm35.84 9.79v-48h5.43v16.81a29.29 29.29 0 0 1 9.32-1.73 13.1 13.1 0 0 1 6.2 1.41 10.71 10.71 0 0 1 4.18 3.74 18.07 18.07 0 0 1 2.23 5.06 21.26 21.26 0 0 1 0.73 5.58q0 8.43-4.38 12.79t-13.82 4.34zm5.43-4.87h4.55q6.77 0 9.72-2.95t3-9.51a14.21 14.21 0 0 0-2-7.52 6.55 6.55 0 0 0-6-3.22 24.73 24.73 0 0 0-9.25 1.54zm29.47-11.19q0-7.71 4.09-12.3a13.75 13.75 0 0 1 10.8-4.59q13.35 0 13.36 18.86h-22.82a12.3 12.3 0 0 0 2.9 7.07q2.59 3.11 7.9 3.1a24.92 24.92 0 0 0 10.55-2v5a27.74 27.74 0 0 1-9.86 1.87 19.83 19.83 0 0 1-7.7-1.37 13.31 13.31 0 0 1-5.28-3.76 16.21 16.21 0 0 1-3-5.38 20.84 20.84 0 0 1-0.94-6.5zm5.62-2.12h17.26a14.91 14.91 0 0 0-2.37-7.12 6.44 6.44 0 0 0-5.62-2.78 8.2 8.2 0 0 0-6.21 2.72 12.07 12.07 0 0 0-3.04 7.18z" fill="#4279f4" stroke="#4279f4" stroke-miterlimit="10" stroke-width="3.2"/>
<path d="m147.32 244.89v-4.89h5v-7.59a8.14 8.14 0 0 1 2.31-6.05 7.79 7.79 0 0 1 5.69-2.28h7.86v4.92h-5c-2.21 0-3.67 0.45-4.37 1.34s-1.06 2.55-1.06 5v4.66h8.46v4.87h-8.46v27.13h-5.44v-27.1zm27.94 27.11v-48h5.43v48zm19.15-3.95a17.86 17.86 0 1 1 12.33 4.9 16.57 16.57 0 0 1-12.33-4.9zm3.84-20.65a13.16 13.16 0 0 0 0 17.2 12.07 12.07 0 0 0 17 0 13.09 13.09 0 0 0 0-17.2 12.07 12.07 0 0 0-17 0zm30.2-7.4h5.75l7.3 25.32 7.43-25.32h5.36l7.34 25.34 7.37-25.34h5.74l-10.04 32h-6.12l-6.83-24.58-6.75 24.58h-6.47z" fill="#0028aa" stroke="#0028aa" stroke-miterlimit="10" stroke-width="3.2"/>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.4 KiB

@@ -0,0 +1,95 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: oauth2-proxy
resources:
- namespace.yaml
- deployment.yaml
- serviceaccount.yaml
- service.yaml
- virtualservice.yaml
secretGenerator:
- name: oauth2-proxy
type: Opaque
literals:
- client-id=kubeflow-oidc-authservice
- client-secret=pUBnBOY80SnXgjibTYM9ZWNzY2xreNGQok
- cookie-secret=7d16fee92f8d11b8940b081b3f8b8acb
configMapGenerator:
- name: oauth2-proxy
files:
- oauth2_proxy.cfg
- name: oauth2-proxy-theme
files:
- kubeflow-logo.svg
- name: oauth2-proxy-parameters
literals:
# This will configure oauth2-proxy option --cookie-secure which can force
# auth redirect with redirect_uri parameter using https.
- FORCE_HTTPS=false
# If Kubernetes is managed by kind, vCluster, minikube or similar tool,
# most probably the Kubernetes OIDC Issuer will be server in-cluster
# behind self-signed certs. This option will configure
# --ssl-insecure-skip-verify which will accept self-signed-certificates.
- ALLOW_SELF_SIGNED_ISSUER=true
# This will configure oauth2-proxy option --skip-jwt-bearer-tokens which
# will pass the requests with 'Authorization' header with Bearer Token
# matching --extra-jwt-issuers.
- ENABLE_M2M_TOKENS=true
# Bearer tokens issued by these OIDC Issuers will be verified against the
# Issuer and accepted on success.
# The format is exactly like in --extra-jwt-issuers.
# If extra jwt issuers are meant to be used with m2m bearer tokens,
# each issuer has to be added with RequestAuthentication so Istio can
# verify, trust and use the JWT. See
# 'common/oauth2-proxy/components/istio-m2m' for details.
# Examples:
# - EXTRA_JWT_ISSUERS=https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local
# - EXTRA_JWT_ISSUERS=https://oidc.eks.region.amazonaws.com/id/1234abcd=https://kubernetes.default.svc
- EXTRA_JWT_ISSUERS=
replacements:
- source:
version: v1
kind: Service
name: oauth2-proxy
fieldPath: metadata.name
targets:
- fieldPaths:
- spec.http.0.route.0.destination.host
options:
delimiter: .
select:
version: v1alpha3
group: networking.istio.io
kind: VirtualService
name: oauth2-proxy
- source:
fieldPath: metadata.namespace
kind: Service
name: oauth2-proxy
version: v1
targets:
- fieldPaths:
- spec.http.0.route.0.destination.host
options:
delimiter: .
index: 1
select:
version: v1alpha3
group: networking.istio.io
kind: VirtualService
name: oauth2-proxy
images:
- name: quay.io/oauth2-proxy/oauth2-proxy
newName: quay.io/oauth2-proxy/oauth2-proxy
newTag: v7.7.1
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
@@ -0,0 +1,79 @@
provider = "oidc"
oidc_issuer_url = "http://dex.auth.svc.cluster.local:5556/dex"
scope = "profile email groups openid"
email_domains = [ "*" ]
# serve a static HTTP 200 upstream on for authentication success
# we are using oauth2-proxy as an ExtAuthz to "check" each request, not pass it on
upstreams = [ "static://200" ]
# skip authentication for these paths
skip_auth_routes = [
"^/dex/",
]
# requests to paths matching these regex patterns will receive a 401 Unauthorized response
# when not authenticated, instead of being redirected to the login page with a 302,
# this prevents background requests being redirected to the login page,
# and the accumulation of CSRF cookies
api_routes = [
# Generic
# NOTE: included because most background requests contain these paths
"/api/",
"/apis/",
# Kubeflow Pipelines
# NOTE: included because KFP UI makes MANY background requests to these paths but because they are
# not `application/json` requests, oauth2-proxy will redirect them to the login page
"^/ml_metadata",
]
# OIDC Discovery has to be skipped and login url has to be provided directly
# in order to enable relative auth redirect. Using OIDC Discovery would set
# the redirect location to http://dex.auth.svc.cluster.local:5556 in the example
# installation. This address is usually not available through the Web Browser.
# If you have a setup where dex has it's url as other than the in-cluster
# service, this is optional.
skip_oidc_discovery = true
login_url = "/dex/auth"
redeem_url = "http://dex.auth.svc.cluster.local:5556/dex/token"
oidc_jwks_url = "http://dex.auth.svc.cluster.local:5556/dex/keys"
# if `false`, a sign-in page is displayed before starting the login flow
# prevents background requests starting their own login flow on token expiry,
# which can lead to many CSRF cookies, potentially exceeding the cookie limit
skip_provider_button = false
# style the sign-in page
provider_display_name = "Dex"
custom_sign_in_logo = "/custom-theme/kubeflow-logo.svg"
banner = "-"
footer = "-"
# oauth2-proxy sends "force" by default, which causes dex to always prompt for login
# https://github.com/dexidp/dex/pull/3086
prompt = "none"
# set Authorization Bearer response header. This is needed in order to
# forward the Authorization Bearer token to Istio and enable authorization
# based on JWT.
set_authorization_header = true
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
# may have other services that use standard auth headers.
set_xauthrequest = true
cookie_name = "oauth2_proxy_kubeflow"
# Dex default cookie expiration is 24h.
# If set to 168h (default oauth2-proxy), Istio will not be able to use the JWT after 24h,
# but oauth2-proxy will still consider the cookie valid.
# It's possible to configure the JWT Refresh Token to enable longer login session.
cookie_expire = "24h"
cookie_refresh = 0
code_challenge_method = "S256"
redirect_url = "/oauth2/callback"
relative_redirect_url = true
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: oauth2-proxy
ports:
- port: 80
name: http
targetPort: http
publishNotReadyAddresses: true
@@ -0,0 +1,4 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: oauth2-proxy
@@ -0,0 +1,18 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: oauth2-proxy
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /oauth2/
route:
- destination:
host: OAUTH2_PROXY_SERVICE.OAUTH2_PROXY_NAMESPACE.svc.cluster.local
port:
number: 80
@@ -0,0 +1,216 @@
# Kubeflow with oauth2-proxy and envoyExtAuthzHttp
For a quick install, see [Example installation](#example-installation).
## Description
Kubeflow authorization operates using custom authentication headers:
* `kubeflow-userid`: Contains the user's email address.
* `kubeflow-groups`: Holds a comma-separated list of user groups.
* Note: The functionality for `kubeflow-groups` is not fully operational at this time.
This feature was implemented using a custom, minimalistic authorization tool from Arrikto
named `oidc-authservice`. This tool was integrated into Istio using `EnvoyFilter`.
The adoption of `envoyExtAuthzHttp` for authentication offers several advantages:
* **Simplified Authorization Process**: `envoyExtAuthzHttp` extension streamlines adding
external authorization to the Envoy proxy within an Istio service mesh. It allows for
declarative policy configuration, reducing the complexity associated with direct
`EnvoyFilter` modifications.
* **Standardization**: Istio recommends `envoyExtAuthzHttp` for its standardized approach.
This promotes consistency across Istio deployments and simplifies understanding and
maintenance of the authorization logic.
* **Separate Policy Management**: Authorization policies are defined in distinct
configuration resources, not directly in `EnvoyFilter`. This separation eases policy
management and facilitates updates or replacements of authorization logic without
altering `EnvoyFilter` configurations.
* **Seamless Integration with Istio**: `envoyExtAuthzHttp` harmonizes with Istio features
like `AuthorizationPolicy` and `VirtualService`, enabling smoother integration within
Istio's ecosystem and taking advantage of its architecture and capabilities.
* **Future-Proofing**: There is a high likelihood that Kubeflow's internal authentication
decisions will transition to relying directly on JWTs instead of custom auth headers.
Additional information on the benefits of moving away from `EnvoyFilter` can be found in these
resources:
* [Istio GitHub Issue #27790](https://github.com/istio/istio/issues/27790)
* [Istio Documentation on Authorization with Custom Authentication](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
While `envoyExtAuthzHttp` could potentially integrate with `oidc-authservice`, `oauth2-proxy`
emerges as a more advanced authentication proxy. It boasts broader community support and is
widely used in the industry, including in the official Istio documentation on [External
Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom).
For more details on the `oauth2-proxy`, refer to the [official documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/behaviour).
## Available Components
Below is a list of the available Kustomize Components with brief descriptions. Click on each for more details.
* **[allow-unauthenticated-issuer-discovery](./allow-unauthenticated-issuer-discovery.md)** -
Creates a ClusterRoleBinding for anonymous access to Kubernetes OIDC
discovery.
* **[central-dashboard](./central-dashboard.md)** - Configures the central
dashboard to use oauth2-proxy logout URL.
* **[istio-external-auth](./istio-external-auth.md)** - Modifies Istio
configuration to define oauth2-proxy as external authentication middleware via
envoyExtAuthzHttp extension provider. Adds RequestAuthentication to trust Dex
as IdP and AuthorizationPolicies to delegate authentication to oauth2-proxy.
* **[istio-m2m](./istio-m2m.md)** - Creates RequestAuthentication for Istio to
trust the OIDC Issuer specified in parameters. This allows the generation of
JWTs for authenticating requests, typically as Bearer Tokens in the
Authorization header. By default, the OIDC Issuer is the in-cluster Kubernetes
OIDC.
## CloudFlare
CloudFlare requires that certain static, standard web browser assets are accessible without
user authentication. This is crucial because CloudFlare aims to cache these assets for
enhanced performance. If these assets necessitate user authentication, CloudFlare robots
will be redirected to the authentication page, potentially causing access issues with the
Kubeflow instance behind CloudFlare.
This issue can be resolved by defining a set of assets in the Istio `AuthorizationPolicy`
that do not require authentication. An example `AuthorizationPolicy` for this purpose is
provided in the file `authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml`.
## Explaining the Auth Routine
1. Istio is configured with the `envoyExtAuthzHttp` extension provider pointing to the
`oauth2-proxy` service. This configuration enables the use of this extension in
`AuthorizationPolicy` for adding external authorization to the service mesh.
2. The Istio service mesh has an `AuthorizationPolicy` named `istio-ingressgateway-oauth2-proxy`
in the `istio-system` (Istio root) namespace. This policy is set with `action: CUSTOM` and
specifies the `oauth2-proxy` provider. Consequently, every request to the
`istio-ingressgateway` must pass through the external authorization service, `oauth2-proxy`.
3. `oauth2-proxy` decides based on the cookie named `oauth2_proxy_kubeflow`. If this cookie
is absent, expired, or invalid, `oauth2-proxy` redirects to the configured OIDC provider,
typically `dex`. The authentication redirect includes:
* A `redirect_uri` for redirecting the user post-authentication,
* A code challenge to guard against interception and replay attacks,
* A state parameter to validate the authorization response's authenticity and
ensure it originates from the initial request.
Post-authentication, a cookie is set in the user's browser, used by Istio and `oauth2-proxy`
for authorizing requests.
`oauth2-proxy` also checks for an `Authorization` header with a JWT bearer token. It trusts
JWTs issued by the configured OIDC provider (default `dex`) and can be configured to trust
additional JWTs. If a valid JWT is present, `oauth2-proxy` forwards the request to Istio
along with the authorization header.
The key objective here is to supply an `Authorization` header with a JWT Bearer Token,
later used in `RequestAuthentication`. This step configures Istio to trust the JWT, parse
its claims to include user email and groups in custom Kubeflow authorization headers, and
make routing and authorization decisions based on these JWT claims.
4. With this setup, Istio always receives the JWT in requests, enabling authorization
decisions based on JWT claims.
## Using HTTPS
`oauth2-proxy` is initially set up with an `http` endpoint, secured by the Istio Service Mesh.
As a result, `oauth2-proxy` may default to assuming that the authentication redirect URI should
also use `http`. To enforce the use of `https`, modify the variable `FORCE_HTTPS` in
`kustomization.yaml` to `true`. This adjustment leverages the `oauth2-proxy` configuration
option `--cookie-secure`, ensuring redirection occurs with `https`.
## Istio JWT Public Key Refresh Interval
In the initial setup of Kubeflow, it's common for `istiod` to become available before `dex`.
Istio's `RequestAuthentication` is configured to retrieve the JWT Public Key from the Issuer
URL. If the Issuer is not yet operational, placeholder keys are set, which can render the
setup nonfunctional until Istio can access the correct JWT Public Key. To address this,
`istiod` is configured with the environment variable `PILOT_JWT_PUB_KEY_REFRESH_INTERVAL="1m"`.
This setting ensures the JWT Public Key is refreshed every minute, rather than the default
20 minutes.
Without this configuration, users may encounter the following Istio error:
```
Jwks doesn't have key to match kid or alg from Jwt
```
## Issues with This Setup
While not an inherent issue with Istio or `oauth2-proxy`, the current Kubeflow configuration
automatically redirects to a URL specified in the logout response body's `afterLogoutURL` key.
This behavior stems from custom integration with the `oidc-authservice` component. While
`oauth2-proxy` is capable of redirecting to the base Kubeflow page, this custom setup results
in users being logged out (with the authentication cookie removed) but not redirected back to
the Kubeflow Home Page.
Details of this custom integration are available at:
* [oidc-authservice server.go](https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509)
* [Kubeflow logout-button.js](https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50)
To log in again, users must manually refresh the page.
## Example Installation
To install Kubeflow configured to use `oauth2-proxy` with Istio's `envoyExtAuthzHttp` extension,
make the following changes to the `example/kustomization.yaml` file:
* use `oauth2-proxy` overlay for istio-install
```
# from
- ../common/istio-1-24/istio-install/base
# to
- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
```
* change `OIDC Authservice` to `oauth2-proxy for OIDC` and use overlay for m2m
bearer tokens with self-signed in-cluster issuer
```
# from
- ../common//oidc-authservice/base
# to
- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
```
* change Dex overlay
```
# from
- ../common/dex/overlays/istio
# to
- ../common/dex/overlays/oauth2-proxy
* change Central Dashboard overlay to use oauth2-proxy for logout
```
# from
- ../apps/centraldashboard/upstream/overlays/kserve
# to
- ../apps/centraldashboard/manuel-patches/overlays/oauth2-proxy
```
All those changes combined can be done with this single command:
```diff
$ git apply <<EOF
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
index c1a85789..4a50440c 100644
--- a/example/kustomization.yaml
+++ b/example/kustomization.yaml
@@ -38,11 +38,11 @@ resources:
# Istio
- ../common/istio-1-24/istio-crds/base
- ../common/istio-1-24/istio-namespace/base
-- ../common/istio-1-24/istio-install/base
-# OIDC Authservice
-- ../common//oidc-authservice/base
+- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
+# oauth2-proxy for OIDC
+- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
# Dex
-- ../common/dex/overlays/istio
+- ../common/dex/overlays/oauth2-proxy
# KNative
- ../common/knative/knative-serving/overlays/gateways
- ../common/knative/knative-eventing/base
@@ -60,7 +60,7 @@ resources:
# Katib
- ../apps/katib/upstream/installs/katib-with-kubeflow
# Central Dashboard
-- ../apps/centraldashboard/upstream/overlays/kserve
+- ../apps/centraldashboard/overlays
# Admission Webhook
- ../apps/admission-webhook/upstream/overlays/cert-manager
# Jupyter Web App
EOF
```
@@ -0,0 +1,43 @@
# Unauthenticated Issuer Discovery
If you are using Kubernetes with tools like kind, vCluster, minikube, or similar solutions for local
development, it's highly likely that the Kubernetes OIDC Issuer operates within the cluster and is
secured with self-signed certificates.
To facilitate the use of m2m tokens with `oauth2-proxy` and Istio, both tools must perform OIDC
Connect Discovery on the Token Issuer. Kubernetes offers OIDC Discovery functionality at a specific
URI:
```
https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
```
Access to this endpoint is blocked by default due to RBAC policies, but it can be enabled by
creating a `ClusterRoleBinding`. This binding associates the predefined `ClusterRole`
`system:service-account-issuer-discovery` with the `system:unauthenticated` group. The
configuration for this is detailed in the resource file `clusterrolebinding.unauthenticated-oidc-viewer.yaml`.
Once this step is completed, the endpoint can be accessed to reveal the Issuer URL:
```bash
$ curl -k https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
# Example output in kind:
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://172.18.0.5:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
# Example output in vCluster:
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://1.2.3.4:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
# Example output in AWS EKS:
{"issuer":"https://oidc.eks.region.amazonaws.com/id/123abc","jwks_uri":"https://ip-1-2-3-4.eu-central-1.compute.internal:443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
```
If you're operating in a `vCluster`, access to the endpoint specified under `jwks_uri` (for example,
`https://1.2.3.4:6443/openid/v1/jwks`) is managed separately, and merely creating the previously
mentioned `ClusterRoleBinding` is insufficient. To circumvent this limitation, you can configure the
`kube-apiserver` to allow anonymous authentication by setting `--anonymous-auth=true`. This is
achieved by appending `--kube-apiserver-arg=anonymous-auth=true` to the list of arguments in the
Helm Chart Values file:
```yaml
vcluster:
extraArgs:
- --kube-apiserver-arg=anonymous-auth=true
```
@@ -0,0 +1,12 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: unauthenticated-oidc-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:unauthenticated
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- clusterrolebinding.unauthenticated-oidc-viewer.yaml
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/deployment.logout-url.yaml
@@ -0,0 +1,17 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: centraldashboard
namespace: kubeflow
spec:
template:
spec:
containers:
- name: centraldashboard
env:
# This successfully logs out the user but the user is not redirected to
# the home page.
# https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50
# Please refresh the page after logging out.
- name: LOGOUT_URL
value: /oauth2/sign_out
@@ -0,0 +1,88 @@
# the default JWKS endpoint for Kind is not public, and uses a self-signed certificate
# for the Istio RequestAuthentication to trust it, we need to proxy the cluster's JWKS endpoint over HTTP
apiVersion: v1
kind: ServiceAccount
metadata:
name: cluster-jwks-proxy
namespace: istio-system
labels:
app.kubernetes.io/name: cluster-jwks-proxy
---
apiVersion: v1
kind: Service
metadata:
name: cluster-jwks-proxy
namespace: istio-system
labels:
app.kubernetes.io/name: cluster-jwks-proxy
spec:
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/name: cluster-jwks-proxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-jwks-proxy
namespace: istio-system
labels:
app.kubernetes.io/name: cluster-jwks-proxy
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: cluster-jwks-proxy
template:
metadata:
labels:
app.kubernetes.io/name: cluster-jwks-proxy
spec:
serviceAccountName: cluster-jwks-proxy
containers:
- name: kubectl-proxy
image: docker.io/bitnami/kubectl
securityContext:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
capabilities:
drop:
- ALL
ports:
- name: http
containerPort: 8080
startupProbe:
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
httpGet:
path: /openid/v1/jwks
port: http
livenessProbe:
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
httpGet:
path: /openid/v1/jwks
port: http
readinessProbe:
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 5
httpGet:
path: /openid/v1/jwks
port: http
args:
- proxy
- --address=0.0.0.0
- --port=8080
## accept all hosts (default is local only)
- --accept-hosts=.*
## only accept requests to '/openid/v1/jwks' and '/.well-known/openid-configuration'
- --accept-paths=^(?:/openid/v1/jwks)|(?:/.well-known/openid-configuration)$
## reject all methods except 'GET'
- --reject-methods=^(POST|PUT|PATCH|DELETE|HEAD|OPTIONS|CONNECT|TRACE)$
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- cluster-jwks-proxy.yaml
images:
- name: docker.io/bitnami/kubectl
newName: docker.io/bitnami/kubectl
newTag: 1.30.4
@@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/cm.enable-oauth2-proxy.yaml
- path: patches/deployment.jwt-refresh-interval.yaml
@@ -0,0 +1,36 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: istio
namespace: istio-system
data:
mesh: |-
accessLogFile: /dev/stdout
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing: {}
enablePrometheusMerge: true
rootNamespace: istio-system
tcpKeepalive:
interval: 5s
probes: 3
time: 10s
trustDomain: cluster.local
extensionProviders:
- envoyExtAuthzHttp:
headersToDownstreamOnDeny:
- content-type
- set-cookie
headersToUpstreamOnAllow:
- authorization
- path
- x-auth-request-email
- x-auth-request-groups
- x-auth-request-user
includeRequestHeadersInCheck:
- authorization
- cookie
service: oauth2-proxy.oauth2-proxy.svc.cluster.local
port: 80
name: oauth2-proxy
@@ -0,0 +1,18 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: istiod
namespace: istio-system
spec:
template:
spec:
containers:
- name: discovery
env:
# Istio will refresh the JWT Public Keys provided with
# RequestAuthentication by default every 20 minutes. For deployment
# from scratch this is not ideal because there is a high chance that
# the istiod will be available before dex is available, triggering
# Istio mechanism to use a placeholder jwt until refreshed.
- name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL
value: "1m"
@@ -0,0 +1,34 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
- when:
- key: request.headers[authorization]
notValues: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
# Exclude paths which are safe to cache by Cloudflare.
- /favicon*
- /webcomponentsjs*
- /vendor.bundle.js
- /app.bundle.js
- /dashboard_lib.bundle.js
- /assets*
- /app.css
@@ -0,0 +1,25 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
- when:
- key: request.headers[authorization]
notValues: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
@@ -0,0 +1,33 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-require-jwt
namespace: istio-system
spec:
action: DENY
selector:
matchLabels:
app: istio-ingressgateway
rules:
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
# because oauth2-proxy injects a Dex JWT into the request.
- from:
- source:
notRequestPrincipals: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
# Exclude paths which are safe to cache by Cloudflare.
- /favicon*
- /webcomponentsjs*
- /vendor.bundle.js
- /app.bundle.js
- /dashboard_lib.bundle.js
- /assets*
- /app.css
@@ -0,0 +1,24 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-require-jwt
namespace: istio-system
spec:
action: DENY
selector:
matchLabels:
app: istio-ingressgateway
rules:
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
# because oauth2-proxy injects a Dex JWT into the request.
- from:
- source:
notRequestPrincipals: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
- authorizationpolicy.istio-ingressgateway-require-jwt.yaml
- requestauthentication.dex-jwt.yaml
# If want to enable caching for some paths (e.g. when using Cloudflare),
# use the following AuthorizationPolicies instead of the default ones.
#- authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml
#- authorizationpolicy.istio-ingressgateway-require-jwt.cloudflare.yaml
@@ -0,0 +1,44 @@
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: dex-jwt
namespace: istio-system
spec:
# we only apply to the ingress-gateway because:
# - there is no need to verify the same tokens at each sidecar
# - having no selector will apply to the RequestAuthentication to ALL
# Pods in the mesh, even ones which are not part of Kubeflow
# - some Kubeflow services accept direct connections with Kubernetes JWTs,
# and we don't want to require that users configure Istio to verify Kubernetes JWTs
# as there is no method to do this which works on all distributions.
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- issuer: http://dex.auth.svc.cluster.local:5556/dex
# `forwardOriginalToken` is not strictly required to be true.
# there are pros and cons to each value:
# - true: the original token is forwarded to the destination service
# which raises the risk of the token leaking
# - false: the original token is stripped from the request
# which will prevent the destination service from
# verifying the token (possibly with its own RequestAuthentication)
forwardOriginalToken: true
# This will unpack the JWTs issued by dex into the expected headers.
# It is applied to BOTH the m2m tokens from outside the cluster (which skip
# oauth2-proxy because they already have a dex JWT), AND user requests which were
# authenticated by oauth2-proxy (which injected a dex JWT).
outputClaimToHeaders:
- header: kubeflow-userid
claim: email
- header: kubeflow-groups
claim: groups
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
fromHeaders:
- name: Authorization
prefix: "Bearer "
@@ -0,0 +1,13 @@
# Kubernetes M2M Authentication with Istio and RequestAuthentication
## Overview
This kustomize component enables M2M (Machine-to-Machine) authentication in Kubernetes, using
Istio and the `RequestAuthentication` object. It configures Istio to trust JWTs (JSON Web Tokens)
in Authorization Bearer tokens when the JWT issuer matches the one in `RequestAuthentication`. The
default setup uses Kubernetes' self-served OIDC issuer with self-signed certificates.
In Kubernetes clusters managed by platform providers, the OIDC issuer is usually managed by the
provider and served behind publicly trusted certificates. In these cases, it's advisable to use
the platform-managed Kubernetes OIDC issuer in the `RequestAuthentication` for seamless integration
and authentication compliance with the platform's security standards.
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- requestauthentication.yaml
@@ -0,0 +1,42 @@
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: m2m-token-issuer
namespace: istio-system
spec:
# we only apply to the ingress-gateway because:
# - there is no need to verify the same tokens at each sidecar
# - having no selector will apply the RequestAuthentication to ALL
# Pods in the mesh, even ones which are not part of Kubeflow
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- # The `issuer` must be replaced with a Kustomize patch.
issuer: PATCH_ME
# `forwardOriginalToken` is not strictly required to be true.
# there are pros and cons to each value:
# - true: the original token is forwarded to the destination service
# which raises the risk of the token leaking
# - false: the original token is stripped from the request
# which will prevent the destination service from
# verifying the token (possibly with its own RequestAuthentication)
forwardOriginalToken: true
# This will unpack the JWTs issued by Kubernetes into the expected headers.
outputClaimToHeaders:
- header: kubeflow-userid
claim: sub
- # NOTE: K8S SA Tokens (e.g. those created with `kubectl create token`) do not contain a `groups` claim,
# but we need to ensure that any groups header provided by the request is overwritten to avoid
# users passing a valid JWT with `kubeflow-groups` header to impersonate other users.
header: kubeflow-groups
claim: groups
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
fromHeaders:
- name: Authorization
prefix: "Bearer "
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 636 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 23 KiB

@@ -0,0 +1,31 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/istio-external-auth
- ../../components/istio-m2m
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
# Configs for oauth2-proxy
- FORCE_HTTPS=true # sets `secure` flag on cookies, requires HTTPS on the gateway
patches:
# patch the 'm2m-token-issuer' RequestAuthentication with correct `issuer`
# NOTE: we are using kustomize components, so we can't use the outer `configMapGenerator` to
# patch the inner one, so we are stuck with using a `patch` instead
- patch: |-
- op: replace
path: /spec/jwtRules/0/issuer
value: https://oidc.eks.AWS_REGION.amazonaws.com/id/CLUSTER_ID
target:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: m2m-token-issuer
namespace: istio-system
@@ -0,0 +1,57 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/cluster-jwks-proxy
- ../../components/istio-external-auth
- ../../components/istio-m2m
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
# Configs for oauth2-proxy
- ALLOW_SELF_SIGNED_ISSUER=true
patches:
# patch the 'm2m-token-issuer' RequestAuthentication with correct `issuer` and `jwksUri`
# NOTE: we are using kustomize components, so we can't use the outer `configMapGenerator` to
# patch the inner one, so we are stuck with using a `patch` instead
- patch: |-
- op: replace
path: /spec/jwtRules/0/issuer
value: https://kubernetes.default.svc.cluster.local
- op: replace
path: /spec/jwtRules/0/jwksUri
value: http://cluster-jwks-proxy.istio-system.svc.cluster.local/openid/v1/jwks
target:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: m2m-token-issuer
namespace: istio-system
- patch: |-
- op: add
path: /spec/jwtRules/-
value:
issuer: "https://kubernetes.default.svc"
jwksUri: "http://cluster-jwks-proxy.istio-system.svc.cluster.local/openid/v1/jwks"
forwardOriginalToken: true
outputClaimToHeaders:
- header: kubeflow-userid
claim: sub
- header: kubeflow-groups
claim: groups
fromHeaders:
- name: Authorization
prefix: "Bearer "
target:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: m2m-token-issuer
namespace: istio-system
@@ -0,0 +1,15 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/istio-external-auth
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
# Configs for oauth2-proxy
- ALLOW_SELF_SIGNED_ISSUER=true
@@ -0,0 +1,28 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-oauth2-proxy
namespace: istio-system
spec:
action: CUSTOM
provider:
name: oauth2-proxy
selector:
matchLabels:
app: istio-ingressgateway
rules:
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
- when:
- key: request.headers[authorization]
notValues: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
- /v1*
- /v2*
- /openai*
@@ -0,0 +1,27 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-require-jwt
namespace: istio-system
spec:
action: DENY
selector:
matchLabels:
app: istio-ingressgateway
rules:
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
# because oauth2-proxy injects a Dex JWT into the request.
- from:
- source:
notRequestPrincipals: ["*"]
to:
- operation:
notPaths:
# Exclude dex paths, otherwise users won't be able to log in.
- /dex/*
- /dex/**
- /oauth2/*
- /v1*
- /v2*
- /openai*
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
- authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
- authorizationpolicy.istio-ingressgateway-require-jwt.yaml
- requestauthentication.keycloak-jwt.yaml
# If want to enable caching for some paths (e.g. when using Cloudflare),
# use the following AuthorizationPolicies instead of the default ones.
#- authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml
#- authorizationpolicy.istio-ingressgateway-require-jwt.cloudflare.yaml
@@ -0,0 +1,25 @@
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: keycloak-jwt
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- # The `issuer` must be replaced with a Kustomize patch.
issuer: PATCH_ME
jwksUri: PATCH_ME
forwardOriginalToken: true
outputClaimToHeaders:
- header: kubeflow-userid
claim: email
- header: kubeflow-groups
claim: groups
- header: x-auth-request-user
claim: sub
fromHeaders:
- name: Authorization
prefix: "Bearer "
@@ -0,0 +1,63 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ./istio-keycloak-auth
configMapGenerator:
- name: oauth2-proxy-parameters
behavior: merge
literals:
# Configs for oauth2-proxy
- ALLOW_SELF_SIGNED_ISSUER=true
- name: istio-m2m-params
envs:
- m2m.env
replacements:
- source:
kind: ConfigMap
version: v1
name: istio-m2m-params
fieldPath: data.M2M_ISSUER
targets:
- select:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: keycloak-jwt
namespace: istio-system
fieldPaths:
- spec.jwtRules.0.issuer
- source:
kind: ConfigMap
version: v1
name: istio-m2m-params
fieldPath: data.M2M_JWKS
targets:
- select:
group: security.istio.io
version: v1beta1
kind: RequestAuthentication
name: keycloak-jwt
namespace: istio-system
fieldPaths:
- spec.jwtRules.0.jwksUri
secretGenerator:
- name: oauth2-proxy
behavior: merge
type: Opaque
envs:
- secrets.env
patches:
- target:
kind: ConfigMap
name: oauth2-proxy
path: patch-oauth2-proxy-config.yaml
@@ -0,0 +1,2 @@
M2M_ISSUER=$OIDC_ISSUER_URL
M2M_JWKS=$OIDC_JWKS_URL
@@ -0,0 +1,56 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: oauth2-proxy
labels:
app: oauth2-proxy
data:
oauth2_proxy.cfg: |
provider = "keycloak-oidc"
oidc_issuer_url = "$OIDC_ISSUER_URL"
scope = "profile email roles openid"
upstreams = "static://200"
email_domains = [ "*" ]
insecure_oidc_allow_unverified_email = "true"
# ---
# OIDC Discovery has to be skipped and login url has to be provided directly
# in order to enable relative auth redirect.
# Turning On OIDC Discovery would set the auth redirect location as the dex
# Issuer URL which is http://dex.auth.svc.cluster.local:5556 in the default,
# example installation. This address is usuallynot available through the Web
# Browser. If you have a setup where dex has it's url as other than the
# in-cluster service, this is optional.
# ---
# Go to dex login page directly instead of showing the oauth2-proxy login
# page.
skip_provider_button = true
# ---
# Set Authorization Bearer response header. This is needed in order to
# forward the Authorization Bearer token to Istio and enable authorization
# based on JWT.
set_authorization_header = true
pass_access_token = true
pass_authorization_header = true
# ---
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
# may have other services that use standard auth headers.
set_xauthrequest = true
# ---
cookie_name = "oauth2_proxy_kubeflow"
# ---
# Dex default cookie expiration is 24h. If set to 168h (default oauth2-proxy),
# Istio will not be able to use the JWT after 24h but oauth2-proxy will still
# consider the cookie valid.
# It's possible to configure the JWT Refresh Token to enable longer login
# session.
cookie_expire = "24h"
cookie_refresh = "5m"
# ---
code_challenge_method = "S256"
# ---
redirect_url = "$REDIRECT_URL"
relative_redirect_url = true
binaryData: {}
@@ -0,0 +1,3 @@
client-id=$CLIENT-ID
client-secret=$CLIENT-SECRET
cookie-secret=$COOKIE-SECRET