Change chart directory structure
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
approvers:
|
||||
- axel7083
|
||||
- kromanow94
|
||||
@@ -0,0 +1,308 @@
|
||||
# Kubeflow Authentication using Oauth2 Proxy
|
||||
|
||||
## Istio envoyExtAuthzHttp
|
||||
|
||||
This is Istio's recommended approach for External Authorization[^2]. It is not limited to the use
|
||||
of `oauth2-proxy`[^3] alone. This method is an industry standard, meeting all of Kubeflow's
|
||||
current and foreseeable authentication needs.
|
||||
|
||||
## Kubeflow Pipelines User and M2M Authentication and Authorization
|
||||
|
||||
The Kubeflow Pipelines component relies on the built-in kubernetes functionalities to authenticate and authorize
|
||||
user requests, specifically the TokenReviews[^4] and SubjectAccessReview[^5].
|
||||
|
||||
The best way to describe how it works is to explain with an example. Lets analyze the flow
|
||||
when a client calls the API to list the KF Pipeline runs:
|
||||
|
||||
1. api-server starts endpoints in:
|
||||
|
||||
https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L95
|
||||
|
||||
Focusing on the pipelines run service:
|
||||
|
||||
1. Register Run Service:
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L116
|
||||
2. proto RPC definition of ListRunsV1
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/api/v1beta1/run.proto#L80
|
||||
3. code definition of ListRunsV1
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L226
|
||||
4. ListRunsV1 calls internal method `listRuns`
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L188
|
||||
5. `listRuns` calls internal method `canAccessRun` which itself calls `s.resourceManager.IsAuthorized`
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/server/run_server.go#L637
|
||||
6. `ResourceManager.IsAuthorized` first tries to authenticate over every available authenticator, which are the `TokenReviewAuthenticator` and `HTTPHeaderAuthenticator`
|
||||
* here the user identity is either the user email provided directly in the `kubeflow-userid` header or the user identity obtained from provided token
|
||||
* https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/resource/resource_manager.go#L1667
|
||||
7. `TokenReviewAuthenticator.GetUserIdentity` gets the token from `Authorization` header and calls the K8s Auth `authv1.TokenReview` with given token which in return provides `userInfo := review.Status.User`. `GetUserIdentity` return `userInfo.Username` which at this point is the `system:serviceaccount:default:default`.
|
||||
* https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/auth/authenticator_token_review.go#L53
|
||||
8. Next in `ResourceManager.IsAuthorized` a SubjectAccessReview is created with `r.subjectAccessReviewClient.Create` with arguments specifying RBAC verbs provided in code definition of `RunServer.listRuns`. If the user (sa) is not authorized, an error is thrown
|
||||
* https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/resource/resource_manager.go#L1703
|
||||
* if the identity was obtained from token (service account), the `rolebinding.rbac.authorization.k8s.io/default-editor` provides the RBAC permission
|
||||
* if the identity was obtained from header (user), the `rolebinding.rbac.authorization.k8s.io/user-example-com` or similar provides the RBAC permission
|
||||
2. User calls api to list pipeline runs as unauthorized service account.
|
||||
|
||||
* This can be done by running Pod with curl in `default` namespace:
|
||||
```bash
|
||||
$ kubectl -n default run -ti --rm curl --image curlimages/curl --command -- sh
|
||||
# v1beta1
|
||||
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
|
||||
{"error":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
|
||||
# v2beta1
|
||||
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
|
||||
{"error":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason: (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
|
||||
```
|
||||
3. User calls api to list pipeline runs as authorized service account.
|
||||
|
||||
* This can be done by running Pod with curl in `kubeflow-user-example-com` namespace specifying correct service account:
|
||||
```bash
|
||||
$ kubectl -n kubeflow-user-example-com run -ti --rm curl --image curlimages/curl --command --overrides='{"spec": {"serviceAccountName": "default-editor"}}' -- sh
|
||||
# v1beta1
|
||||
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
|
||||
{} # empty response which is fine because no pipeline runs exist
|
||||
# v2beta1
|
||||
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
|
||||
{} # empty response which is fine because no pipeline runs exist
|
||||
```
|
||||
|
||||
### Authentication and Authorization analysis diagram for Kubeflow Pipelines
|
||||

|
||||
|
||||
### Change the default authentication from "Dex + Oauth2-proxy" to "Oauth2-proxy" only
|
||||
|
||||
The authentication in Kubeflow evolved over time and we dropped envoyfilters and oidc-authservice in favor of RequestAuthentication and Oauth2-proxy in Kubeflow 1.9.
|
||||

|
||||
|
||||
You can adjust OAuth2 Proxy to directly connect to your own IDP(Identity Provider) suchg as GCP, [AWS](https://docs.aws.amazon.com/cognito/latest/developerguide/federation-endpoints-oauth-grants.html), Azure etc:
|
||||
|
||||
1. Create an application on your IdP (purple line)
|
||||
2. Change your [OAuth2 Proxy issuer](https://github.com/kubeflow/manifests/blob/35539f162ea7fafc8c5035d8df0d8d8cf5a9d327/common/oauth2-proxy/base/oauth2-proxy-config.yaml#L10) to your IdP. Of course never ever directly, but with kustomize overlays and components.
|
||||
|
||||
Here is an example of patching oauth2-proxy to connect directly to Azure IDP and skip Dex.
|
||||
This is enterprise integration so feel free to hire consultants or pay for commercial distributions if you need more help.
|
||||
For example Azure returns rather large headers compared to other IDPs, so maybe you need to annotate the nginx-ingress to support that.
|
||||
|
||||
```
|
||||
# based on https://github.com/kubeflow/manifests/blob/master/common/oauth2-proxy/base/oauth2_proxy.cfg
|
||||
# and https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/azure/
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: oauth2-proxy
|
||||
data:
|
||||
oauth2_proxy.cfg: |
|
||||
provider = "oidc"
|
||||
oidc_issuer_url = "https://login.microsoftonline.com/$MY_TENANT/v2.0"
|
||||
scope = "openid email offline_access" # removed groups and profile to decrease the size and added offline_access to enable refresh tokens
|
||||
email_domains = [ "*" ]
|
||||
|
||||
# serve a static HTTP 200 upstream on for authentication success
|
||||
# we are using oauth2-proxy as an ExtAuthz to "check" each request, not pass it on
|
||||
upstreams = [ "static://200" ]
|
||||
|
||||
# skip authentication for these paths
|
||||
skip_auth_routes = [
|
||||
"^/dex/",
|
||||
]
|
||||
|
||||
# requests to paths matching these regex patterns will receive a 401 Unauthorized response
|
||||
# when not authenticated, instead of being redirected to the login page with a 302,
|
||||
# this prevents background requests being redirected to the login page,
|
||||
# and the accumulation of CSRF cookies
|
||||
api_routes = [
|
||||
# Generic
|
||||
# NOTE: included because most background requests contain these paths
|
||||
"/api/",
|
||||
"/apis/",
|
||||
|
||||
# Kubeflow Pipelines
|
||||
# NOTE: included because KFP UI makes MANY background requests to these paths but because they are
|
||||
# not `application/json` requests, oauth2-proxy will redirect them to the login page
|
||||
"^/ml_metadata",
|
||||
]
|
||||
|
||||
skip_provider_button = true
|
||||
set_authorization_header = true
|
||||
set_xauthrequest = true
|
||||
cookie_name = "oauth2_proxy_kubeflow"
|
||||
cookie_expire = "24h"
|
||||
cookie_refresh = "59m" # This improves the user experience a lot
|
||||
redirect_url = "https://$MY_PUBLIC_KUBEFLOW_DOMAIN/oauth2/callback"
|
||||
relative_redirect_url = false
|
||||
```
|
||||
|
||||
3. In the istio-system namespace is a RequestAuthentication resource. You need to change its issuer to your own IdP, or even better create an additional one.
|
||||
|
||||
```
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: RequestAuthentication
|
||||
metadata:
|
||||
name: azure-aad-requestauthentication
|
||||
namespace: istio-system
|
||||
spec:
|
||||
# we only apply to the ingress-gateway because:
|
||||
# - there is no need to verify the same tokens at each sidecar
|
||||
# - having no selector will apply to the RequestAuthentication to ALL
|
||||
# Pods in the mesh, even ones which are not part of Kubeflow
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
|
||||
jwtRules:
|
||||
- issuer: https://login.microsoftonline.com/$MY_TENANT/v2.0
|
||||
|
||||
# `forwardOriginalToken` is not strictly required to be true.
|
||||
# there are pros and cons to each value:
|
||||
# - true: the original token is forwarded to the destination service
|
||||
# which raises the risk of the token leaking
|
||||
# - false: the original token is stripped from the request
|
||||
# which will prevent the destination service from
|
||||
# verifying the token (possibly with its own RequestAuthentication)
|
||||
forwardOriginalToken: true
|
||||
|
||||
# This will unpack the JWTs issued by Dex or other IDPs into the expected headers.
|
||||
# It is applied to BOTH the m2m tokens from outside the cluster (which skip
|
||||
# oauth2-proxy because they already have a dex JWT), AND user requests which were
|
||||
# authenticated by oauth2-proxy (which injected a dex JWT).
|
||||
outputClaimToHeaders:
|
||||
- header: kubeflow-userid
|
||||
claim: email
|
||||
- header: kubeflow-groups
|
||||
claim: groups
|
||||
|
||||
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
|
||||
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
|
||||
fromHeaders:
|
||||
- name: Authorization
|
||||
prefix: "Bearer "
|
||||
```
|
||||
|
||||
You can also add more RequestAuthentication to support other issuers as for example for M2M access from github actions as explained in the root level Readme.md.
|
||||
This feature is useful when you need to integrate Kubeflow with your current CI/CD platform (GitHub Actions, Jenkins) via machine-to-machine authentication.
|
||||
The following is an example for obtaining and using a JWT token From your IDP with Python, but you can also just take a look at our CI/CD test that uses simple Kubernetes serviceaccount tokens to access KFP, Jupyterlabs etc. from GitHub Actions.
|
||||
|
||||
```
|
||||
import requests
|
||||
token_url = "https://your-idp.com/oauth/token"
|
||||
client_id = "YOUR_CLIENT_ID"
|
||||
client_secret = "YOUR_CLIENT_SECRET"
|
||||
username = "YOUR_USERNAME"
|
||||
password = "YOUR_PASSWORD"
|
||||
# request header
|
||||
headers = {
|
||||
"Content-Type": "application/x-www-form-urlencoded"
|
||||
}
|
||||
data = {
|
||||
"grant_type": "password",
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"username": username,
|
||||
"password": password,
|
||||
"scope": "openid profile email" #change your scope
|
||||
}
|
||||
response = requests.post(token_url, headers=headers, data=data)
|
||||
TOKEN = response.json()['access_token']
|
||||
```
|
||||
|
||||
```
|
||||
import kfp
|
||||
kubeflow_host="https://your_host"
|
||||
pipeline_host = kubeflow_host + "/pipeline"
|
||||
client = kfp.Client(host=pipeline_host, existing_token=TOKEN)
|
||||
print(client.list_runs(namespace="your-profile-name"))
|
||||
```
|
||||
|
||||
## Known Issues:
|
||||
|
||||
Some openidc providers such as Azure provide too large JWTs / Cookies that exceed the limit of most GRPC and gunicorn web application deployments in Kubeflow.
|
||||
If removing the groups claim in oauth2-proxy is not enough then you can add an envrionment variable to all web applications
|
||||
|
||||
```
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: kserve-models-web-app
|
||||
namespace: kubeflow
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: kserve-models-web-app # repeat for all other *-web-app-(deployment)
|
||||
env:
|
||||
- name: GUNICORN_CMD_ARGS
|
||||
value: --limit-request-field_size 32000
|
||||
```
|
||||
|
||||
and modify the KFP GRPC server via
|
||||
```
|
||||
- path: patches/metadata-grpc-virtualservice-patch.yaml
|
||||
target:
|
||||
kind: VirtualService
|
||||
name: metadata-grpc
|
||||
namespace: kubeflow
|
||||
|
||||
# patches/metadata-grpc-virtualservice-patch.yaml
|
||||
# Remove the oauth2-proxy cookie that violates the maximum metadata size for a GRPC request
|
||||
- op: add
|
||||
path: /spec/http/0/route/0/headers
|
||||
value:
|
||||
request:
|
||||
remove:
|
||||
- Cookie
|
||||
```
|
||||
|
||||
to fix `received initial metadata size exceeds limit`.
|
||||
|
||||
## Kubeflow Notebooks User and M2M Authentication and Authorization
|
||||
|
||||
The underlying mechanism is the same as in Kubeflow Pipelines.
|
||||
|
||||
Similarly, to explain how it works, let's analyze the code step by step, starting from the api route definition
|
||||
for listing notebooks:
|
||||
|
||||
* list notebooks api route definition
|
||||
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L53
|
||||
* this calls `crud_backend/api/notebook.py::list_notebooks`
|
||||
* `crud_backend/api/notebook.py::list_notebooks` calls `authz.ensure_authorized`
|
||||
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/api/notebook.py#L24
|
||||
* `crud_backend/authz.py::ensure_authorized` calls `crud_backend/authn.py::get_username`
|
||||
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authz.py#L101
|
||||
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authn.py#L12
|
||||
* `crud_backend/authn.py::get_username` gets the user id from userid header (email or sa in format `system:serviceaccount:kubeflowusernamespace:default-editor`)
|
||||
* `crud_backend/authz.py::ensure_authorized` calls `crud_backend/authz.py::is_authorized`
|
||||
* https://github.com/kubeflow/kubeflow/blob/v1.8.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/authz.py#L46
|
||||
* this calls `create_subject_access_review` which uses the same mechanism as pipelines with `r.subjectAccessReviewClient.Create`
|
||||
|
||||
## KServe Authentication
|
||||
|
||||
The analysis of KServe auth capabilities suggests that while it's possible to limit access to only authenticated agents,
|
||||
there might be some improvements required to enable access only to authorized agents.
|
||||
|
||||
This is based on the following:
|
||||
|
||||
1. KServe Controller Manager patch integrating kube-rbac-proxy[^6].
|
||||
|
||||
This suggests the kserve **might** use the same mechanism based on
|
||||
`SubjectAccessReviews`. Having a look at the kubeflow/manifests I see it's
|
||||
not enabled.
|
||||
2. Search through the docs and code:
|
||||
|
||||
* https://github.com/kserve/kserve/tree/v0.12.0/docs/samples/istio-dex
|
||||
* https://github.com/kserve/kserve/tree/v0.12.0/docs/samples/gcp-iap
|
||||
|
||||
The docs above mention that while it's possible to enable authentication,
|
||||
authorization is more complicated and probably we need to add
|
||||
`AuthorizationPolicy`
|
||||
|
||||
> create an [Istio AuthorizationPolicy](https://istio.io/latest/docs/reference/config/security/authorization-policy/) to grant access to the pods or disable it
|
||||
|
||||
Most probably some work is needed to enable authorized access to kserve models.
|
||||
|
||||
## Links
|
||||
|
||||
[^1]: [Envoy Filter](https://istio.io/latest/docs/reference/config/networking/envoy-filter/)
|
||||
[^2]: [External Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
|
||||
[^3]: [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy)
|
||||
[^4]: [Kubernetes TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/)
|
||||
[^5]: [Kubernetes SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v3/)
|
||||
[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy)
|
||||
@@ -0,0 +1,24 @@
|
||||
# oauth2-proxy
|
||||
|
||||
## `oauth2-proxy` Deployment
|
||||
|
||||
This deployment of `oauth2-proxy` has been configured to align closely with the official
|
||||
`oauth2-proxy` Helm installation. This approach facilitates easier integration with any
|
||||
existing `oauth2-proxy` deployments that may already be present on the cluster.
|
||||
|
||||
### Upgrading `oauth2-proxy`
|
||||
|
||||
The `oauth2-proxy` component is designed for easy upgrading, thanks to its foundation on the
|
||||
official `oauth2-proxy` Helm chart. The use of the standard Helm chart simplifies the upgrade
|
||||
process, closely following the upgrades of the official `oauth2-proxy` releases.
|
||||
|
||||
### Stateless Nature of `oauth2-proxy`
|
||||
|
||||
`oauth2-proxy` operates as a stateless application. This statelessness simplifies many
|
||||
aspects of its operation, particularly upgrades, as there are no concerns about complex state
|
||||
management or data migration. Additionally, while `oauth2-proxy` is integrated into the
|
||||
Kubernetes environment, this integration is limited to running the application, thereby
|
||||
minimizing the impact on Kubernetes infrastructure during upgrades.
|
||||
|
||||
These characteristics make the upgrade process for `oauth2-proxy` more predictable and
|
||||
manageable in Kubernetes environments.
|
||||
@@ -0,0 +1,100 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
spec:
|
||||
volumes:
|
||||
- name: oauth2-proxy-config
|
||||
configMap:
|
||||
name: oauth2-proxy
|
||||
- name: oauth2-proxy-theme
|
||||
configMap:
|
||||
name: oauth2-proxy-theme
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:latest
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --config=/etc/oauth2_proxy/oauth2_proxy.cfg
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
- name: metrics
|
||||
containerPort: 44180
|
||||
protocol: TCP
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: cookie-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECURE
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: oauth2-proxy-parameters
|
||||
key: FORCE_HTTPS
|
||||
- name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: oauth2-proxy-parameters
|
||||
key: ALLOW_SELF_SIGNED_ISSUER
|
||||
- name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: oauth2-proxy-parameters
|
||||
key: ENABLE_M2M_TOKENS
|
||||
- name: OAUTH2_PROXY_EXTRA_JWT_ISSUERS
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: oauth2-proxy-parameters
|
||||
key: EXTRA_JWT_ISSUERS
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
runAsNonRoot: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
volumeMounts:
|
||||
- name: oauth2-proxy-config
|
||||
mountPath: /etc/oauth2_proxy/oauth2_proxy.cfg
|
||||
subPath: oauth2_proxy.cfg
|
||||
readOnly: true
|
||||
- name: oauth2-proxy-theme
|
||||
mountPath: /custom-theme/kubeflow-logo.svg
|
||||
subPath: kubeflow-logo.svg
|
||||
readOnly: true
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
scheme: HTTP
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: http
|
||||
scheme: HTTP
|
||||
resources: {}
|
||||
@@ -0,0 +1,15 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg width="256" height="256" version="1.1" viewBox="0 0 256 256" xmlns="http://www.w3.org/2000/svg">
|
||||
<g transform="matrix(.85002 0 0 .85002 10.306 11.309)" data-name="Layer 2">
|
||||
<g data-name="Layer 1">
|
||||
<path d="m95.9 62.15 4.1 102.1 73.75-94.12a6.79 6.79 0 0 1 9.6-1.11l46 36.92-15-65.61z" fill="#4279f4"/>
|
||||
<path d="m102.55 182.98h65.42l-40.17-32.23z" fill="#0028aa"/>
|
||||
<path d="m180.18 83.92-44 56.14 46.88 37.61 44.47-55.76z" fill="#014bd1"/>
|
||||
<path d="m83.56 52.3 0.01-0.01 38.69-48.52-62.39 30.05-15.41 67.51z" fill="#bedcff"/>
|
||||
<path d="m45.32 122.05 41.44 51.96-3.95-98.98z" fill="#6ca1ff"/>
|
||||
<path d="m202.31 28.73-59.66-28.73-37.13 46.56z" fill="#a1c3ff"/>
|
||||
<path d="m1.6 272v-44.78h5.74v23.41l20.48-23.41h6.4l-17.39 19.7 19 25.07h-6.73l-15.92-20.8-5.84 6.65v14.16zm40.02-9.79v-22.21h5.43v22.39a4.67 4.67 0 0 0 2.35 4.19 11 11 0 0 0 11 0 4.69 4.69 0 0 0 2.33-4.19v-22.39h5.43v22.19a9.08 9.08 0 0 1-4.1 7.87 16.2 16.2 0 0 1-18.37 0 9.07 9.07 0 0 1-4.07-7.85zm35.84 9.79v-48h5.43v16.81a29.29 29.29 0 0 1 9.32-1.73 13.1 13.1 0 0 1 6.2 1.41 10.71 10.71 0 0 1 4.18 3.74 18.07 18.07 0 0 1 2.23 5.06 21.26 21.26 0 0 1 0.73 5.58q0 8.43-4.38 12.79t-13.82 4.34zm5.43-4.87h4.55q6.77 0 9.72-2.95t3-9.51a14.21 14.21 0 0 0-2-7.52 6.55 6.55 0 0 0-6-3.22 24.73 24.73 0 0 0-9.25 1.54zm29.47-11.19q0-7.71 4.09-12.3a13.75 13.75 0 0 1 10.8-4.59q13.35 0 13.36 18.86h-22.82a12.3 12.3 0 0 0 2.9 7.07q2.59 3.11 7.9 3.1a24.92 24.92 0 0 0 10.55-2v5a27.74 27.74 0 0 1-9.86 1.87 19.83 19.83 0 0 1-7.7-1.37 13.31 13.31 0 0 1-5.28-3.76 16.21 16.21 0 0 1-3-5.38 20.84 20.84 0 0 1-0.94-6.5zm5.62-2.12h17.26a14.91 14.91 0 0 0-2.37-7.12 6.44 6.44 0 0 0-5.62-2.78 8.2 8.2 0 0 0-6.21 2.72 12.07 12.07 0 0 0-3.04 7.18z" fill="#4279f4" stroke="#4279f4" stroke-miterlimit="10" stroke-width="3.2"/>
|
||||
<path d="m147.32 244.89v-4.89h5v-7.59a8.14 8.14 0 0 1 2.31-6.05 7.79 7.79 0 0 1 5.69-2.28h7.86v4.92h-5c-2.21 0-3.67 0.45-4.37 1.34s-1.06 2.55-1.06 5v4.66h8.46v4.87h-8.46v27.13h-5.44v-27.1zm27.94 27.11v-48h5.43v48zm19.15-3.95a17.86 17.86 0 1 1 12.33 4.9 16.57 16.57 0 0 1-12.33-4.9zm3.84-20.65a13.16 13.16 0 0 0 0 17.2 12.07 12.07 0 0 0 17 0 13.09 13.09 0 0 0 0-17.2 12.07 12.07 0 0 0-17 0zm30.2-7.4h5.75l7.3 25.32 7.43-25.32h5.36l7.34 25.34 7.37-25.34h5.74l-10.04 32h-6.12l-6.83-24.58-6.75 24.58h-6.47z" fill="#0028aa" stroke="#0028aa" stroke-miterlimit="10" stroke-width="3.2"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
@@ -0,0 +1,95 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: oauth2-proxy
|
||||
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- deployment.yaml
|
||||
- serviceaccount.yaml
|
||||
- service.yaml
|
||||
- virtualservice.yaml
|
||||
|
||||
secretGenerator:
|
||||
- name: oauth2-proxy
|
||||
type: Opaque
|
||||
literals:
|
||||
- client-id=kubeflow-oidc-authservice
|
||||
- client-secret=pUBnBOY80SnXgjibTYM9ZWNzY2xreNGQok
|
||||
- cookie-secret=7d16fee92f8d11b8940b081b3f8b8acb
|
||||
|
||||
configMapGenerator:
|
||||
- name: oauth2-proxy
|
||||
files:
|
||||
- oauth2_proxy.cfg
|
||||
|
||||
- name: oauth2-proxy-theme
|
||||
files:
|
||||
- kubeflow-logo.svg
|
||||
|
||||
- name: oauth2-proxy-parameters
|
||||
literals:
|
||||
# This will configure oauth2-proxy option --cookie-secure which can force
|
||||
# auth redirect with redirect_uri parameter using https.
|
||||
- FORCE_HTTPS=false
|
||||
|
||||
# If Kubernetes is managed by kind, vCluster, minikube or similar tool,
|
||||
# most probably the Kubernetes OIDC Issuer will be server in-cluster
|
||||
# behind self-signed certs. This option will configure
|
||||
# --ssl-insecure-skip-verify which will accept self-signed-certificates.
|
||||
- ALLOW_SELF_SIGNED_ISSUER=true
|
||||
|
||||
# This will configure oauth2-proxy option --skip-jwt-bearer-tokens which
|
||||
# will pass the requests with 'Authorization' header with Bearer Token
|
||||
# matching --extra-jwt-issuers.
|
||||
- ENABLE_M2M_TOKENS=true
|
||||
|
||||
# Bearer tokens issued by these OIDC Issuers will be verified against the
|
||||
# Issuer and accepted on success.
|
||||
# The format is exactly like in --extra-jwt-issuers.
|
||||
# If extra jwt issuers are meant to be used with m2m bearer tokens,
|
||||
# each issuer has to be added with RequestAuthentication so Istio can
|
||||
# verify, trust and use the JWT. See
|
||||
# 'common/oauth2-proxy/components/istio-m2m' for details.
|
||||
# Examples:
|
||||
# - EXTRA_JWT_ISSUERS=https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local
|
||||
# - EXTRA_JWT_ISSUERS=https://oidc.eks.region.amazonaws.com/id/1234abcd=https://kubernetes.default.svc
|
||||
- EXTRA_JWT_ISSUERS=
|
||||
|
||||
replacements:
|
||||
- source:
|
||||
version: v1
|
||||
kind: Service
|
||||
name: oauth2-proxy
|
||||
fieldPath: metadata.name
|
||||
targets:
|
||||
- fieldPaths:
|
||||
- spec.http.0.route.0.destination.host
|
||||
options:
|
||||
delimiter: .
|
||||
select:
|
||||
version: v1alpha3
|
||||
group: networking.istio.io
|
||||
kind: VirtualService
|
||||
name: oauth2-proxy
|
||||
- source:
|
||||
fieldPath: metadata.namespace
|
||||
kind: Service
|
||||
name: oauth2-proxy
|
||||
version: v1
|
||||
targets:
|
||||
- fieldPaths:
|
||||
- spec.http.0.route.0.destination.host
|
||||
options:
|
||||
delimiter: .
|
||||
index: 1
|
||||
select:
|
||||
version: v1alpha3
|
||||
group: networking.istio.io
|
||||
kind: VirtualService
|
||||
name: oauth2-proxy
|
||||
|
||||
images:
|
||||
- name: quay.io/oauth2-proxy/oauth2-proxy
|
||||
newName: quay.io/oauth2-proxy/oauth2-proxy
|
||||
newTag: v7.7.1
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
@@ -0,0 +1,79 @@
|
||||
provider = "oidc"
|
||||
oidc_issuer_url = "http://dex.auth.svc.cluster.local:5556/dex"
|
||||
scope = "profile email groups openid"
|
||||
email_domains = [ "*" ]
|
||||
|
||||
# serve a static HTTP 200 upstream on for authentication success
|
||||
# we are using oauth2-proxy as an ExtAuthz to "check" each request, not pass it on
|
||||
upstreams = [ "static://200" ]
|
||||
|
||||
# skip authentication for these paths
|
||||
skip_auth_routes = [
|
||||
"^/dex/",
|
||||
]
|
||||
|
||||
# requests to paths matching these regex patterns will receive a 401 Unauthorized response
|
||||
# when not authenticated, instead of being redirected to the login page with a 302,
|
||||
# this prevents background requests being redirected to the login page,
|
||||
# and the accumulation of CSRF cookies
|
||||
api_routes = [
|
||||
# Generic
|
||||
# NOTE: included because most background requests contain these paths
|
||||
"/api/",
|
||||
"/apis/",
|
||||
|
||||
# Kubeflow Pipelines
|
||||
# NOTE: included because KFP UI makes MANY background requests to these paths but because they are
|
||||
# not `application/json` requests, oauth2-proxy will redirect them to the login page
|
||||
"^/ml_metadata",
|
||||
]
|
||||
|
||||
# OIDC Discovery has to be skipped and login url has to be provided directly
|
||||
# in order to enable relative auth redirect. Using OIDC Discovery would set
|
||||
# the redirect location to http://dex.auth.svc.cluster.local:5556 in the example
|
||||
# installation. This address is usually not available through the Web Browser.
|
||||
# If you have a setup where dex has it's url as other than the in-cluster
|
||||
# service, this is optional.
|
||||
skip_oidc_discovery = true
|
||||
login_url = "/dex/auth"
|
||||
redeem_url = "http://dex.auth.svc.cluster.local:5556/dex/token"
|
||||
oidc_jwks_url = "http://dex.auth.svc.cluster.local:5556/dex/keys"
|
||||
|
||||
# if `false`, a sign-in page is displayed before starting the login flow
|
||||
# prevents background requests starting their own login flow on token expiry,
|
||||
# which can lead to many CSRF cookies, potentially exceeding the cookie limit
|
||||
skip_provider_button = false
|
||||
|
||||
# style the sign-in page
|
||||
provider_display_name = "Dex"
|
||||
custom_sign_in_logo = "/custom-theme/kubeflow-logo.svg"
|
||||
banner = "-"
|
||||
footer = "-"
|
||||
|
||||
# oauth2-proxy sends "force" by default, which causes dex to always prompt for login
|
||||
# https://github.com/dexidp/dex/pull/3086
|
||||
prompt = "none"
|
||||
|
||||
# set Authorization Bearer response header. This is needed in order to
|
||||
# forward the Authorization Bearer token to Istio and enable authorization
|
||||
# based on JWT.
|
||||
set_authorization_header = true
|
||||
|
||||
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
|
||||
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
|
||||
# may have other services that use standard auth headers.
|
||||
set_xauthrequest = true
|
||||
|
||||
cookie_name = "oauth2_proxy_kubeflow"
|
||||
|
||||
# Dex default cookie expiration is 24h.
|
||||
# If set to 168h (default oauth2-proxy), Istio will not be able to use the JWT after 24h,
|
||||
# but oauth2-proxy will still consider the cookie valid.
|
||||
# It's possible to configure the JWT Refresh Token to enable longer login session.
|
||||
cookie_expire = "24h"
|
||||
cookie_refresh = 0
|
||||
|
||||
code_challenge_method = "S256"
|
||||
|
||||
redirect_url = "/oauth2/callback"
|
||||
relative_redirect_url = true
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: oauth2-proxy
|
||||
ports:
|
||||
- port: 80
|
||||
name: http
|
||||
targetPort: http
|
||||
publishNotReadyAddresses: true
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
@@ -0,0 +1,18 @@
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
spec:
|
||||
gateways:
|
||||
- kubeflow/kubeflow-gateway
|
||||
hosts:
|
||||
- '*'
|
||||
http:
|
||||
- match:
|
||||
- uri:
|
||||
prefix: /oauth2/
|
||||
route:
|
||||
- destination:
|
||||
host: OAUTH2_PROXY_SERVICE.OAUTH2_PROXY_NAMESPACE.svc.cluster.local
|
||||
port:
|
||||
number: 80
|
||||
@@ -0,0 +1,216 @@
|
||||
# Kubeflow with oauth2-proxy and envoyExtAuthzHttp
|
||||
|
||||
For a quick install, see [Example installation](#example-installation).
|
||||
|
||||
|
||||
## Description
|
||||
|
||||
Kubeflow authorization operates using custom authentication headers:
|
||||
* `kubeflow-userid`: Contains the user's email address.
|
||||
* `kubeflow-groups`: Holds a comma-separated list of user groups.
|
||||
* Note: The functionality for `kubeflow-groups` is not fully operational at this time.
|
||||
|
||||
This feature was implemented using a custom, minimalistic authorization tool from Arrikto
|
||||
named `oidc-authservice`. This tool was integrated into Istio using `EnvoyFilter`.
|
||||
|
||||
The adoption of `envoyExtAuthzHttp` for authentication offers several advantages:
|
||||
* **Simplified Authorization Process**: `envoyExtAuthzHttp` extension streamlines adding
|
||||
external authorization to the Envoy proxy within an Istio service mesh. It allows for
|
||||
declarative policy configuration, reducing the complexity associated with direct
|
||||
`EnvoyFilter` modifications.
|
||||
* **Standardization**: Istio recommends `envoyExtAuthzHttp` for its standardized approach.
|
||||
This promotes consistency across Istio deployments and simplifies understanding and
|
||||
maintenance of the authorization logic.
|
||||
* **Separate Policy Management**: Authorization policies are defined in distinct
|
||||
configuration resources, not directly in `EnvoyFilter`. This separation eases policy
|
||||
management and facilitates updates or replacements of authorization logic without
|
||||
altering `EnvoyFilter` configurations.
|
||||
* **Seamless Integration with Istio**: `envoyExtAuthzHttp` harmonizes with Istio features
|
||||
like `AuthorizationPolicy` and `VirtualService`, enabling smoother integration within
|
||||
Istio's ecosystem and taking advantage of its architecture and capabilities.
|
||||
* **Future-Proofing**: There is a high likelihood that Kubeflow's internal authentication
|
||||
decisions will transition to relying directly on JWTs instead of custom auth headers.
|
||||
|
||||
Additional information on the benefits of moving away from `EnvoyFilter` can be found in these
|
||||
resources:
|
||||
* [Istio GitHub Issue #27790](https://github.com/istio/istio/issues/27790)
|
||||
* [Istio Documentation on Authorization with Custom Authentication](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
|
||||
|
||||
While `envoyExtAuthzHttp` could potentially integrate with `oidc-authservice`, `oauth2-proxy`
|
||||
emerges as a more advanced authentication proxy. It boasts broader community support and is
|
||||
widely used in the industry, including in the official Istio documentation on [External
|
||||
Authorization](https://istio.io/latest/docs/tasks/security/authorization/authz-custom).
|
||||
|
||||
For more details on the `oauth2-proxy`, refer to the [official documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/behaviour).
|
||||
|
||||
## Available Components
|
||||
|
||||
Below is a list of the available Kustomize Components with brief descriptions. Click on each for more details.
|
||||
|
||||
* **[allow-unauthenticated-issuer-discovery](./allow-unauthenticated-issuer-discovery.md)** -
|
||||
Creates a ClusterRoleBinding for anonymous access to Kubernetes OIDC
|
||||
discovery.
|
||||
|
||||
* **[central-dashboard](./central-dashboard.md)** - Configures the central
|
||||
dashboard to use oauth2-proxy logout URL.
|
||||
|
||||
* **[istio-external-auth](./istio-external-auth.md)** - Modifies Istio
|
||||
configuration to define oauth2-proxy as external authentication middleware via
|
||||
envoyExtAuthzHttp extension provider. Adds RequestAuthentication to trust Dex
|
||||
as IdP and AuthorizationPolicies to delegate authentication to oauth2-proxy.
|
||||
|
||||
* **[istio-m2m](./istio-m2m.md)** - Creates RequestAuthentication for Istio to
|
||||
trust the OIDC Issuer specified in parameters. This allows the generation of
|
||||
JWTs for authenticating requests, typically as Bearer Tokens in the
|
||||
Authorization header. By default, the OIDC Issuer is the in-cluster Kubernetes
|
||||
OIDC.
|
||||
|
||||
## CloudFlare
|
||||
|
||||
CloudFlare requires that certain static, standard web browser assets are accessible without
|
||||
user authentication. This is crucial because CloudFlare aims to cache these assets for
|
||||
enhanced performance. If these assets necessitate user authentication, CloudFlare robots
|
||||
will be redirected to the authentication page, potentially causing access issues with the
|
||||
Kubeflow instance behind CloudFlare.
|
||||
|
||||
This issue can be resolved by defining a set of assets in the Istio `AuthorizationPolicy`
|
||||
that do not require authentication. An example `AuthorizationPolicy` for this purpose is
|
||||
provided in the file `authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml`.
|
||||
|
||||
## Explaining the Auth Routine
|
||||
|
||||
1. Istio is configured with the `envoyExtAuthzHttp` extension provider pointing to the
|
||||
`oauth2-proxy` service. This configuration enables the use of this extension in
|
||||
`AuthorizationPolicy` for adding external authorization to the service mesh.
|
||||
2. The Istio service mesh has an `AuthorizationPolicy` named `istio-ingressgateway-oauth2-proxy`
|
||||
in the `istio-system` (Istio root) namespace. This policy is set with `action: CUSTOM` and
|
||||
specifies the `oauth2-proxy` provider. Consequently, every request to the
|
||||
`istio-ingressgateway` must pass through the external authorization service, `oauth2-proxy`.
|
||||
3. `oauth2-proxy` decides based on the cookie named `oauth2_proxy_kubeflow`. If this cookie
|
||||
is absent, expired, or invalid, `oauth2-proxy` redirects to the configured OIDC provider,
|
||||
typically `dex`. The authentication redirect includes:
|
||||
* A `redirect_uri` for redirecting the user post-authentication,
|
||||
* A code challenge to guard against interception and replay attacks,
|
||||
* A state parameter to validate the authorization response's authenticity and
|
||||
ensure it originates from the initial request.
|
||||
|
||||
Post-authentication, a cookie is set in the user's browser, used by Istio and `oauth2-proxy`
|
||||
for authorizing requests.
|
||||
|
||||
`oauth2-proxy` also checks for an `Authorization` header with a JWT bearer token. It trusts
|
||||
JWTs issued by the configured OIDC provider (default `dex`) and can be configured to trust
|
||||
additional JWTs. If a valid JWT is present, `oauth2-proxy` forwards the request to Istio
|
||||
along with the authorization header.
|
||||
|
||||
The key objective here is to supply an `Authorization` header with a JWT Bearer Token,
|
||||
later used in `RequestAuthentication`. This step configures Istio to trust the JWT, parse
|
||||
its claims to include user email and groups in custom Kubeflow authorization headers, and
|
||||
make routing and authorization decisions based on these JWT claims.
|
||||
4. With this setup, Istio always receives the JWT in requests, enabling authorization
|
||||
decisions based on JWT claims.
|
||||
|
||||
## Using HTTPS
|
||||
|
||||
`oauth2-proxy` is initially set up with an `http` endpoint, secured by the Istio Service Mesh.
|
||||
As a result, `oauth2-proxy` may default to assuming that the authentication redirect URI should
|
||||
also use `http`. To enforce the use of `https`, modify the variable `FORCE_HTTPS` in
|
||||
`kustomization.yaml` to `true`. This adjustment leverages the `oauth2-proxy` configuration
|
||||
option `--cookie-secure`, ensuring redirection occurs with `https`.
|
||||
|
||||
## Istio JWT Public Key Refresh Interval
|
||||
|
||||
In the initial setup of Kubeflow, it's common for `istiod` to become available before `dex`.
|
||||
Istio's `RequestAuthentication` is configured to retrieve the JWT Public Key from the Issuer
|
||||
URL. If the Issuer is not yet operational, placeholder keys are set, which can render the
|
||||
setup nonfunctional until Istio can access the correct JWT Public Key. To address this,
|
||||
`istiod` is configured with the environment variable `PILOT_JWT_PUB_KEY_REFRESH_INTERVAL="1m"`.
|
||||
This setting ensures the JWT Public Key is refreshed every minute, rather than the default
|
||||
20 minutes.
|
||||
|
||||
Without this configuration, users may encounter the following Istio error:
|
||||
```
|
||||
Jwks doesn't have key to match kid or alg from Jwt
|
||||
```
|
||||
|
||||
## Issues with This Setup
|
||||
|
||||
While not an inherent issue with Istio or `oauth2-proxy`, the current Kubeflow configuration
|
||||
automatically redirects to a URL specified in the logout response body's `afterLogoutURL` key.
|
||||
This behavior stems from custom integration with the `oidc-authservice` component. While
|
||||
`oauth2-proxy` is capable of redirecting to the base Kubeflow page, this custom setup results
|
||||
in users being logged out (with the authentication cookie removed) but not redirected back to
|
||||
the Kubeflow Home Page.
|
||||
|
||||
Details of this custom integration are available at:
|
||||
* [oidc-authservice server.go](https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509)
|
||||
* [Kubeflow logout-button.js](https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50)
|
||||
|
||||
To log in again, users must manually refresh the page.
|
||||
|
||||
## Example Installation
|
||||
|
||||
To install Kubeflow configured to use `oauth2-proxy` with Istio's `envoyExtAuthzHttp` extension,
|
||||
make the following changes to the `example/kustomization.yaml` file:
|
||||
* use `oauth2-proxy` overlay for istio-install
|
||||
```
|
||||
# from
|
||||
- ../common/istio-1-24/istio-install/base
|
||||
# to
|
||||
- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
|
||||
```
|
||||
* change `OIDC Authservice` to `oauth2-proxy for OIDC` and use overlay for m2m
|
||||
bearer tokens with self-signed in-cluster issuer
|
||||
```
|
||||
# from
|
||||
- ../common//oidc-authservice/base
|
||||
# to
|
||||
- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
|
||||
```
|
||||
* change Dex overlay
|
||||
```
|
||||
# from
|
||||
- ../common/dex/overlays/istio
|
||||
# to
|
||||
- ../common/dex/overlays/oauth2-proxy
|
||||
* change Central Dashboard overlay to use oauth2-proxy for logout
|
||||
```
|
||||
# from
|
||||
- ../apps/centraldashboard/upstream/overlays/kserve
|
||||
# to
|
||||
- ../apps/centraldashboard/manuel-patches/overlays/oauth2-proxy
|
||||
```
|
||||
|
||||
All those changes combined can be done with this single command:
|
||||
```diff
|
||||
$ git apply <<EOF
|
||||
diff --git a/example/kustomization.yaml b/example/kustomization.yaml
|
||||
index c1a85789..4a50440c 100644
|
||||
--- a/example/kustomization.yaml
|
||||
+++ b/example/kustomization.yaml
|
||||
@@ -38,11 +38,11 @@ resources:
|
||||
# Istio
|
||||
- ../common/istio-1-24/istio-crds/base
|
||||
- ../common/istio-1-24/istio-namespace/base
|
||||
-- ../common/istio-1-24/istio-install/base
|
||||
-# OIDC Authservice
|
||||
-- ../common//oidc-authservice/base
|
||||
+- ../common/istio-1-24/istio-install/overlays/oauth2-proxy
|
||||
+# oauth2-proxy for OIDC
|
||||
+- ../common/oauth2-proxy/overlays/m2m-dex-and-kind
|
||||
# Dex
|
||||
-- ../common/dex/overlays/istio
|
||||
+- ../common/dex/overlays/oauth2-proxy
|
||||
# KNative
|
||||
- ../common/knative/knative-serving/overlays/gateways
|
||||
- ../common/knative/knative-eventing/base
|
||||
@@ -60,7 +60,7 @@ resources:
|
||||
# Katib
|
||||
- ../apps/katib/upstream/installs/katib-with-kubeflow
|
||||
# Central Dashboard
|
||||
-- ../apps/centraldashboard/upstream/overlays/kserve
|
||||
+- ../apps/centraldashboard/overlays
|
||||
# Admission Webhook
|
||||
- ../apps/admission-webhook/upstream/overlays/cert-manager
|
||||
# Jupyter Web App
|
||||
EOF
|
||||
```
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Unauthenticated Issuer Discovery
|
||||
|
||||
If you are using Kubernetes with tools like kind, vCluster, minikube, or similar solutions for local
|
||||
development, it's highly likely that the Kubernetes OIDC Issuer operates within the cluster and is
|
||||
secured with self-signed certificates.
|
||||
|
||||
To facilitate the use of m2m tokens with `oauth2-proxy` and Istio, both tools must perform OIDC
|
||||
Connect Discovery on the Token Issuer. Kubernetes offers OIDC Discovery functionality at a specific
|
||||
URI:
|
||||
```
|
||||
https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
|
||||
```
|
||||
|
||||
Access to this endpoint is blocked by default due to RBAC policies, but it can be enabled by
|
||||
creating a `ClusterRoleBinding`. This binding associates the predefined `ClusterRole`
|
||||
`system:service-account-issuer-discovery` with the `system:unauthenticated` group. The
|
||||
configuration for this is detailed in the resource file `clusterrolebinding.unauthenticated-oidc-viewer.yaml`.
|
||||
|
||||
Once this step is completed, the endpoint can be accessed to reveal the Issuer URL:
|
||||
```bash
|
||||
$ curl -k https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration
|
||||
|
||||
# Example output in kind:
|
||||
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://172.18.0.5:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
|
||||
|
||||
# Example output in vCluster:
|
||||
{"issuer":"https://kubernetes.default.svc.cluster.local","jwks_uri":"https://1.2.3.4:6443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
|
||||
|
||||
# Example output in AWS EKS:
|
||||
{"issuer":"https://oidc.eks.region.amazonaws.com/id/123abc","jwks_uri":"https://ip-1-2-3-4.eu-central-1.compute.internal:443/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
|
||||
```
|
||||
|
||||
If you're operating in a `vCluster`, access to the endpoint specified under `jwks_uri` (for example,
|
||||
`https://1.2.3.4:6443/openid/v1/jwks`) is managed separately, and merely creating the previously
|
||||
mentioned `ClusterRoleBinding` is insufficient. To circumvent this limitation, you can configure the
|
||||
`kube-apiserver` to allow anonymous authentication by setting `--anonymous-auth=true`. This is
|
||||
achieved by appending `--kube-apiserver-arg=anonymous-auth=true` to the list of arguments in the
|
||||
Helm Chart Values file:
|
||||
```yaml
|
||||
vcluster:
|
||||
extraArgs:
|
||||
- --kube-apiserver-arg=anonymous-auth=true
|
||||
```
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: unauthenticated-oidc-viewer
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:service-account-issuer-discovery
|
||||
subjects:
|
||||
- apiGroup: rbac.authorization.k8s.io
|
||||
kind: Group
|
||||
name: system:unauthenticated
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
resources:
|
||||
- clusterrolebinding.unauthenticated-oidc-viewer.yaml
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
patches:
|
||||
- path: patches/deployment.logout-url.yaml
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: centraldashboard
|
||||
namespace: kubeflow
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: centraldashboard
|
||||
env:
|
||||
# This successfully logs out the user but the user is not redirected to
|
||||
# the home page.
|
||||
# https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50
|
||||
# Please refresh the page after logging out.
|
||||
- name: LOGOUT_URL
|
||||
value: /oauth2/sign_out
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
# the default JWKS endpoint for Kind is not public, and uses a self-signed certificate
|
||||
# for the Istio RequestAuthentication to trust it, we need to proxy the cluster's JWKS endpoint over HTTP
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: cluster-jwks-proxy
|
||||
namespace: istio-system
|
||||
labels:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: cluster-jwks-proxy
|
||||
namespace: istio-system
|
||||
labels:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: http
|
||||
selector:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cluster-jwks-proxy
|
||||
namespace: istio-system
|
||||
labels:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: cluster-jwks-proxy
|
||||
spec:
|
||||
serviceAccountName: cluster-jwks-proxy
|
||||
containers:
|
||||
- name: kubectl-proxy
|
||||
image: docker.io/bitnami/kubectl
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
runAsNonRoot: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
startupProbe:
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 2
|
||||
httpGet:
|
||||
path: /openid/v1/jwks
|
||||
port: http
|
||||
livenessProbe:
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
httpGet:
|
||||
path: /openid/v1/jwks
|
||||
port: http
|
||||
readinessProbe:
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 5
|
||||
httpGet:
|
||||
path: /openid/v1/jwks
|
||||
port: http
|
||||
args:
|
||||
- proxy
|
||||
- --address=0.0.0.0
|
||||
- --port=8080
|
||||
## accept all hosts (default is local only)
|
||||
- --accept-hosts=.*
|
||||
## only accept requests to '/openid/v1/jwks' and '/.well-known/openid-configuration'
|
||||
- --accept-paths=^(?:/openid/v1/jwks)|(?:/.well-known/openid-configuration)$
|
||||
## reject all methods except 'GET'
|
||||
- --reject-methods=^(POST|PUT|PATCH|DELETE|HEAD|OPTIONS|CONNECT|TRACE)$
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
resources:
|
||||
- cluster-jwks-proxy.yaml
|
||||
|
||||
images:
|
||||
- name: docker.io/bitnami/kubectl
|
||||
newName: docker.io/bitnami/kubectl
|
||||
newTag: 1.30.4
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
patches:
|
||||
- path: patches/cm.enable-oauth2-proxy.yaml
|
||||
- path: patches/deployment.jwt-refresh-interval.yaml
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: istio
|
||||
namespace: istio-system
|
||||
data:
|
||||
mesh: |-
|
||||
accessLogFile: /dev/stdout
|
||||
defaultConfig:
|
||||
discoveryAddress: istiod.istio-system.svc:15012
|
||||
proxyMetadata: {}
|
||||
tracing: {}
|
||||
enablePrometheusMerge: true
|
||||
rootNamespace: istio-system
|
||||
tcpKeepalive:
|
||||
interval: 5s
|
||||
probes: 3
|
||||
time: 10s
|
||||
trustDomain: cluster.local
|
||||
extensionProviders:
|
||||
- envoyExtAuthzHttp:
|
||||
headersToDownstreamOnDeny:
|
||||
- content-type
|
||||
- set-cookie
|
||||
headersToUpstreamOnAllow:
|
||||
- authorization
|
||||
- path
|
||||
- x-auth-request-email
|
||||
- x-auth-request-groups
|
||||
- x-auth-request-user
|
||||
includeRequestHeadersInCheck:
|
||||
- authorization
|
||||
- cookie
|
||||
service: oauth2-proxy.oauth2-proxy.svc.cluster.local
|
||||
port: 80
|
||||
name: oauth2-proxy
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: istiod
|
||||
namespace: istio-system
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: discovery
|
||||
env:
|
||||
# Istio will refresh the JWT Public Keys provided with
|
||||
# RequestAuthentication by default every 20 minutes. For deployment
|
||||
# from scratch this is not ideal because there is a high chance that
|
||||
# the istiod will be available before dex is available, triggering
|
||||
# Istio mechanism to use a placeholder jwt until refreshed.
|
||||
- name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL
|
||||
value: "1m"
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-oauth2-proxy
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: CUSTOM
|
||||
provider:
|
||||
name: oauth2-proxy
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
|
||||
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
|
||||
- when:
|
||||
- key: request.headers[authorization]
|
||||
notValues: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
|
||||
# Exclude paths which are safe to cache by Cloudflare.
|
||||
- /favicon*
|
||||
- /webcomponentsjs*
|
||||
- /vendor.bundle.js
|
||||
- /app.bundle.js
|
||||
- /dashboard_lib.bundle.js
|
||||
- /assets*
|
||||
- /app.css
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-oauth2-proxy
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: CUSTOM
|
||||
provider:
|
||||
name: oauth2-proxy
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
|
||||
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
|
||||
- when:
|
||||
- key: request.headers[authorization]
|
||||
notValues: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-require-jwt
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: DENY
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
|
||||
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
|
||||
# because oauth2-proxy injects a Dex JWT into the request.
|
||||
- from:
|
||||
- source:
|
||||
notRequestPrincipals: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
|
||||
# Exclude paths which are safe to cache by Cloudflare.
|
||||
- /favicon*
|
||||
- /webcomponentsjs*
|
||||
- /vendor.bundle.js
|
||||
- /app.bundle.js
|
||||
- /dashboard_lib.bundle.js
|
||||
- /assets*
|
||||
- /app.css
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-require-jwt
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: DENY
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
|
||||
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
|
||||
# because oauth2-proxy injects a Dex JWT into the request.
|
||||
- from:
|
||||
- source:
|
||||
notRequestPrincipals: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
resources:
|
||||
- authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
|
||||
- authorizationpolicy.istio-ingressgateway-require-jwt.yaml
|
||||
- requestauthentication.dex-jwt.yaml
|
||||
|
||||
# If want to enable caching for some paths (e.g. when using Cloudflare),
|
||||
# use the following AuthorizationPolicies instead of the default ones.
|
||||
#- authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml
|
||||
#- authorizationpolicy.istio-ingressgateway-require-jwt.cloudflare.yaml
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: RequestAuthentication
|
||||
metadata:
|
||||
name: dex-jwt
|
||||
namespace: istio-system
|
||||
spec:
|
||||
# we only apply to the ingress-gateway because:
|
||||
# - there is no need to verify the same tokens at each sidecar
|
||||
# - having no selector will apply to the RequestAuthentication to ALL
|
||||
# Pods in the mesh, even ones which are not part of Kubeflow
|
||||
# - some Kubeflow services accept direct connections with Kubernetes JWTs,
|
||||
# and we don't want to require that users configure Istio to verify Kubernetes JWTs
|
||||
# as there is no method to do this which works on all distributions.
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
|
||||
jwtRules:
|
||||
- issuer: http://dex.auth.svc.cluster.local:5556/dex
|
||||
|
||||
# `forwardOriginalToken` is not strictly required to be true.
|
||||
# there are pros and cons to each value:
|
||||
# - true: the original token is forwarded to the destination service
|
||||
# which raises the risk of the token leaking
|
||||
# - false: the original token is stripped from the request
|
||||
# which will prevent the destination service from
|
||||
# verifying the token (possibly with its own RequestAuthentication)
|
||||
forwardOriginalToken: true
|
||||
|
||||
# This will unpack the JWTs issued by dex into the expected headers.
|
||||
# It is applied to BOTH the m2m tokens from outside the cluster (which skip
|
||||
# oauth2-proxy because they already have a dex JWT), AND user requests which were
|
||||
# authenticated by oauth2-proxy (which injected a dex JWT).
|
||||
outputClaimToHeaders:
|
||||
- header: kubeflow-userid
|
||||
claim: email
|
||||
- header: kubeflow-groups
|
||||
claim: groups
|
||||
|
||||
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
|
||||
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
|
||||
fromHeaders:
|
||||
- name: Authorization
|
||||
prefix: "Bearer "
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Kubernetes M2M Authentication with Istio and RequestAuthentication
|
||||
|
||||
## Overview
|
||||
|
||||
This kustomize component enables M2M (Machine-to-Machine) authentication in Kubernetes, using
|
||||
Istio and the `RequestAuthentication` object. It configures Istio to trust JWTs (JSON Web Tokens)
|
||||
in Authorization Bearer tokens when the JWT issuer matches the one in `RequestAuthentication`. The
|
||||
default setup uses Kubernetes' self-served OIDC issuer with self-signed certificates.
|
||||
|
||||
In Kubernetes clusters managed by platform providers, the OIDC issuer is usually managed by the
|
||||
provider and served behind publicly trusted certificates. In these cases, it's advisable to use
|
||||
the platform-managed Kubernetes OIDC issuer in the `RequestAuthentication` for seamless integration
|
||||
and authentication compliance with the platform's security standards.
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
resources:
|
||||
- requestauthentication.yaml
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: RequestAuthentication
|
||||
metadata:
|
||||
name: m2m-token-issuer
|
||||
namespace: istio-system
|
||||
spec:
|
||||
# we only apply to the ingress-gateway because:
|
||||
# - there is no need to verify the same tokens at each sidecar
|
||||
# - having no selector will apply the RequestAuthentication to ALL
|
||||
# Pods in the mesh, even ones which are not part of Kubeflow
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
|
||||
jwtRules:
|
||||
- # The `issuer` must be replaced with a Kustomize patch.
|
||||
issuer: PATCH_ME
|
||||
|
||||
# `forwardOriginalToken` is not strictly required to be true.
|
||||
# there are pros and cons to each value:
|
||||
# - true: the original token is forwarded to the destination service
|
||||
# which raises the risk of the token leaking
|
||||
# - false: the original token is stripped from the request
|
||||
# which will prevent the destination service from
|
||||
# verifying the token (possibly with its own RequestAuthentication)
|
||||
forwardOriginalToken: true
|
||||
|
||||
# This will unpack the JWTs issued by Kubernetes into the expected headers.
|
||||
outputClaimToHeaders:
|
||||
- header: kubeflow-userid
|
||||
claim: sub
|
||||
- # NOTE: K8S SA Tokens (e.g. those created with `kubectl create token`) do not contain a `groups` claim,
|
||||
# but we need to ensure that any groups header provided by the request is overwritten to avoid
|
||||
# users passing a valid JWT with `kubeflow-groups` header to impersonate other users.
|
||||
header: kubeflow-groups
|
||||
claim: groups
|
||||
|
||||
# We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
|
||||
# This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
|
||||
fromHeaders:
|
||||
- name: Authorization
|
||||
prefix: "Bearer "
|
||||
+1
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 636 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 23 KiB |
+31
@@ -0,0 +1,31 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
|
||||
components:
|
||||
- ../../components/istio-external-auth
|
||||
- ../../components/istio-m2m
|
||||
|
||||
configMapGenerator:
|
||||
- name: oauth2-proxy-parameters
|
||||
behavior: merge
|
||||
literals:
|
||||
# Configs for oauth2-proxy
|
||||
- FORCE_HTTPS=true # sets `secure` flag on cookies, requires HTTPS on the gateway
|
||||
|
||||
patches:
|
||||
# patch the 'm2m-token-issuer' RequestAuthentication with correct `issuer`
|
||||
# NOTE: we are using kustomize components, so we can't use the outer `configMapGenerator` to
|
||||
# patch the inner one, so we are stuck with using a `patch` instead
|
||||
- patch: |-
|
||||
- op: replace
|
||||
path: /spec/jwtRules/0/issuer
|
||||
value: https://oidc.eks.AWS_REGION.amazonaws.com/id/CLUSTER_ID
|
||||
target:
|
||||
group: security.istio.io
|
||||
version: v1beta1
|
||||
kind: RequestAuthentication
|
||||
name: m2m-token-issuer
|
||||
namespace: istio-system
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
|
||||
components:
|
||||
- ../../components/cluster-jwks-proxy
|
||||
- ../../components/istio-external-auth
|
||||
- ../../components/istio-m2m
|
||||
|
||||
configMapGenerator:
|
||||
- name: oauth2-proxy-parameters
|
||||
behavior: merge
|
||||
literals:
|
||||
# Configs for oauth2-proxy
|
||||
- ALLOW_SELF_SIGNED_ISSUER=true
|
||||
|
||||
patches:
|
||||
# patch the 'm2m-token-issuer' RequestAuthentication with correct `issuer` and `jwksUri`
|
||||
# NOTE: we are using kustomize components, so we can't use the outer `configMapGenerator` to
|
||||
# patch the inner one, so we are stuck with using a `patch` instead
|
||||
- patch: |-
|
||||
- op: replace
|
||||
path: /spec/jwtRules/0/issuer
|
||||
value: https://kubernetes.default.svc.cluster.local
|
||||
- op: replace
|
||||
path: /spec/jwtRules/0/jwksUri
|
||||
value: http://cluster-jwks-proxy.istio-system.svc.cluster.local/openid/v1/jwks
|
||||
target:
|
||||
group: security.istio.io
|
||||
version: v1beta1
|
||||
kind: RequestAuthentication
|
||||
name: m2m-token-issuer
|
||||
namespace: istio-system
|
||||
|
||||
- patch: |-
|
||||
- op: add
|
||||
path: /spec/jwtRules/-
|
||||
value:
|
||||
issuer: "https://kubernetes.default.svc"
|
||||
jwksUri: "http://cluster-jwks-proxy.istio-system.svc.cluster.local/openid/v1/jwks"
|
||||
forwardOriginalToken: true
|
||||
outputClaimToHeaders:
|
||||
- header: kubeflow-userid
|
||||
claim: sub
|
||||
- header: kubeflow-groups
|
||||
claim: groups
|
||||
fromHeaders:
|
||||
- name: Authorization
|
||||
prefix: "Bearer "
|
||||
target:
|
||||
group: security.istio.io
|
||||
version: v1beta1
|
||||
kind: RequestAuthentication
|
||||
name: m2m-token-issuer
|
||||
namespace: istio-system
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
|
||||
components:
|
||||
- ../../components/istio-external-auth
|
||||
|
||||
configMapGenerator:
|
||||
- name: oauth2-proxy-parameters
|
||||
behavior: merge
|
||||
literals:
|
||||
# Configs for oauth2-proxy
|
||||
- ALLOW_SELF_SIGNED_ISSUER=true
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-oauth2-proxy
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: CUSTOM
|
||||
provider:
|
||||
name: oauth2-proxy
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# We ONLY authenticate requests that DON'T have an `Authorization` header using oauth2-proxy.
|
||||
# This is because we use RequestAuthentication to authenticate requests with an `Authorization` header.
|
||||
- when:
|
||||
- key: request.headers[authorization]
|
||||
notValues: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
- /v1*
|
||||
- /v2*
|
||||
- /openai*
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: AuthorizationPolicy
|
||||
metadata:
|
||||
name: istio-ingressgateway-require-jwt
|
||||
namespace: istio-system
|
||||
spec:
|
||||
action: DENY
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
rules:
|
||||
# Deny requests that don't have a verified JWT (from a RequestAuthentication)
|
||||
# Note, even user requests that have been authenticated by oauth2-proxy will have a JWT,
|
||||
# because oauth2-proxy injects a Dex JWT into the request.
|
||||
- from:
|
||||
- source:
|
||||
notRequestPrincipals: ["*"]
|
||||
to:
|
||||
- operation:
|
||||
notPaths:
|
||||
# Exclude dex paths, otherwise users won't be able to log in.
|
||||
- /dex/*
|
||||
- /dex/**
|
||||
- /oauth2/*
|
||||
- /v1*
|
||||
- /v2*
|
||||
- /openai*
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1alpha1
|
||||
kind: Component
|
||||
|
||||
resources:
|
||||
- authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml
|
||||
- authorizationpolicy.istio-ingressgateway-require-jwt.yaml
|
||||
- requestauthentication.keycloak-jwt.yaml
|
||||
|
||||
# If want to enable caching for some paths (e.g. when using Cloudflare),
|
||||
# use the following AuthorizationPolicies instead of the default ones.
|
||||
#- authorizationpolicy.istio-ingressgateway-oauth2-proxy.cloudflare.yaml
|
||||
#- authorizationpolicy.istio-ingressgateway-require-jwt.cloudflare.yaml
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: RequestAuthentication
|
||||
metadata:
|
||||
name: keycloak-jwt
|
||||
namespace: istio-system
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: istio-ingressgateway
|
||||
|
||||
jwtRules:
|
||||
- # The `issuer` must be replaced with a Kustomize patch.
|
||||
issuer: PATCH_ME
|
||||
jwksUri: PATCH_ME
|
||||
forwardOriginalToken: true
|
||||
outputClaimToHeaders:
|
||||
- header: kubeflow-userid
|
||||
claim: email
|
||||
- header: kubeflow-groups
|
||||
claim: groups
|
||||
- header: x-auth-request-user
|
||||
claim: sub
|
||||
fromHeaders:
|
||||
- name: Authorization
|
||||
prefix: "Bearer "
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
|
||||
components:
|
||||
- ./istio-keycloak-auth
|
||||
|
||||
configMapGenerator:
|
||||
- name: oauth2-proxy-parameters
|
||||
behavior: merge
|
||||
literals:
|
||||
# Configs for oauth2-proxy
|
||||
- ALLOW_SELF_SIGNED_ISSUER=true
|
||||
- name: istio-m2m-params
|
||||
envs:
|
||||
- m2m.env
|
||||
|
||||
replacements:
|
||||
- source:
|
||||
kind: ConfigMap
|
||||
version: v1
|
||||
name: istio-m2m-params
|
||||
fieldPath: data.M2M_ISSUER
|
||||
targets:
|
||||
- select:
|
||||
group: security.istio.io
|
||||
version: v1beta1
|
||||
kind: RequestAuthentication
|
||||
name: keycloak-jwt
|
||||
namespace: istio-system
|
||||
fieldPaths:
|
||||
- spec.jwtRules.0.issuer
|
||||
|
||||
- source:
|
||||
kind: ConfigMap
|
||||
version: v1
|
||||
name: istio-m2m-params
|
||||
fieldPath: data.M2M_JWKS
|
||||
targets:
|
||||
- select:
|
||||
group: security.istio.io
|
||||
version: v1beta1
|
||||
kind: RequestAuthentication
|
||||
name: keycloak-jwt
|
||||
namespace: istio-system
|
||||
fieldPaths:
|
||||
- spec.jwtRules.0.jwksUri
|
||||
|
||||
|
||||
secretGenerator:
|
||||
- name: oauth2-proxy
|
||||
behavior: merge
|
||||
type: Opaque
|
||||
envs:
|
||||
- secrets.env
|
||||
|
||||
patches:
|
||||
- target:
|
||||
kind: ConfigMap
|
||||
name: oauth2-proxy
|
||||
path: patch-oauth2-proxy-config.yaml
|
||||
@@ -0,0 +1,2 @@
|
||||
M2M_ISSUER=$OIDC_ISSUER_URL
|
||||
M2M_JWKS=$OIDC_JWKS_URL
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
data:
|
||||
oauth2_proxy.cfg: |
|
||||
provider = "keycloak-oidc"
|
||||
oidc_issuer_url = "$OIDC_ISSUER_URL"
|
||||
scope = "profile email roles openid"
|
||||
upstreams = "static://200"
|
||||
email_domains = [ "*" ]
|
||||
insecure_oidc_allow_unverified_email = "true"
|
||||
# ---
|
||||
# OIDC Discovery has to be skipped and login url has to be provided directly
|
||||
# in order to enable relative auth redirect.
|
||||
# Turning On OIDC Discovery would set the auth redirect location as the dex
|
||||
# Issuer URL which is http://dex.auth.svc.cluster.local:5556 in the default,
|
||||
# example installation. This address is usuallynot available through the Web
|
||||
# Browser. If you have a setup where dex has it's url as other than the
|
||||
# in-cluster service, this is optional.
|
||||
# ---
|
||||
# Go to dex login page directly instead of showing the oauth2-proxy login
|
||||
# page.
|
||||
skip_provider_button = true
|
||||
# ---
|
||||
# Set Authorization Bearer response header. This is needed in order to
|
||||
# forward the Authorization Bearer token to Istio and enable authorization
|
||||
# based on JWT.
|
||||
set_authorization_header = true
|
||||
pass_access_token = true
|
||||
pass_authorization_header = true
|
||||
|
||||
# ---
|
||||
# set X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email and
|
||||
# X-Auth-Request-Preferred-Username. This is optional for Kubeflow but you
|
||||
# may have other services that use standard auth headers.
|
||||
set_xauthrequest = true
|
||||
# ---
|
||||
cookie_name = "oauth2_proxy_kubeflow"
|
||||
# ---
|
||||
# Dex default cookie expiration is 24h. If set to 168h (default oauth2-proxy),
|
||||
# Istio will not be able to use the JWT after 24h but oauth2-proxy will still
|
||||
# consider the cookie valid.
|
||||
# It's possible to configure the JWT Refresh Token to enable longer login
|
||||
# session.
|
||||
cookie_expire = "24h"
|
||||
cookie_refresh = "5m"
|
||||
# ---
|
||||
code_challenge_method = "S256"
|
||||
# ---
|
||||
redirect_url = "$REDIRECT_URL"
|
||||
relative_redirect_url = true
|
||||
|
||||
binaryData: {}
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
client-id=$CLIENT-ID
|
||||
client-secret=$CLIENT-SECRET
|
||||
cookie-secret=$COOKIE-SECRET
|
||||
Reference in New Issue
Block a user