dip/service-catalog
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

update kubeflow dip-catalog

Browse Source
This commit is contained in:
ChanghoWoo
2025-01-13 02:31:27 +00:00
parent 1dc1181a03
commit 5451f16d72
1959 changed files with 602337 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
charts/kubeflow/proposals/20200913-rootlessKubeflow.md
+50
View File
@@ -0,0 +1,50 @@
# Rootless Kubeflow
Authors: Julius von Kohout (@juliusvonkohout)
### Goals
We want to run Kubeflow as rootless as possible according to CNCF/Kubernetes best practices.
Most enterprise environments will require this as well.
### Implementation details
The main steps are adding an additional profile for istio-cni and later ambient mesh, updating the documentation and manifest generation process.
Only istio-cni or istio ambient mesh can run rootless as explained here https://istio.io/latest/docs/setup/additional-setup/cni/.
Istio-cni will still need a deamonset in kube-system, but that is completly isolated from user workloads.
The ambient mesh should get rid of this as well and also has the benefit of removing the istio initcontainers and sidecars altogether.
Then adding the baseline and restricted PSS as kustomize component to `/contrib` and extending the profile controller to annotate user namespaces with configurable PSS labels.
We want to use a staged approach.
#### First Stage:
1. Implement Istio 1.17.5 and use it by default, because 1.17. is what we have planned to use for Kubeflow 1.8.
2. Implement istio-cni (`--set components.cni.enabled=true --set components.cni.namespace=kube-system`) as second option.
3. Add simple tests similar to `tests/gh-actions/install_istio.sh` and `tests/gh-actions/install_knative.sh` for istio-cni and support both rootfull and rootless istio at the same time and give users one release to test
#### Second stage:
4. Add pod security standards (https://kubernetes.io/docs/concepts/security/pod-security-standards/) `base/restricted` to `manifests/contrib`
5. Enforce PSS baseline (Adavanced users can still build OCI containers via Podman and buildah, but not Docker in Docker). The baseline PSS should works with any istio. If not we will move this item to the third stage after istio-cni or the ambient mesh is the default.
7. Enable Warnings for violations of restricted PSS
8. Add tests to make sure that the PSS are used and tested in the CICD
9. Optionally Enforce PSS restricted (this is where minor corner cases are affected)
#### Third stage:
9. Upgrade Istio to 1.19 to make the ambient mesh available
10. Add istio-ambient as an option to the next Kubeflow release.
#### Fourth stage:
11. Use the ambient service mesh by default in Kubeflow 1.10.
### Non-Goals
This does not cover Application level CVEs, only cluster level security.
### Does this break any existing functionality?
So far not. Only PSS restricted may block the security-wise dangerous Docker in Docker.
This is a rarely used feature from the KFP SDK.
With PSS baseline you can still build OCI images with Podman for example.
We should replace Docker with the cli compatible podman in the KFP SDK https://kubeflow-pipelines.readthedocs.io/en/1.8.22/source/kfp.containers.html?highlight=kfp.containers.build_image_from_working_dir#kfp.containers.build_image_from_working_dir.
### Does this fix/solve any outstanding issues?
This proposal enables Kubeflow to implement parts of Kubernetes best practices and improve the usage in enterprise and regulated environments.
The progress is tracked in https://github.com/kubeflow/manifests/issues/2528
charts/kubeflow/proposals/20220926-contrib-component-guidelines.md
+71
View File
@@ -0,0 +1,71 @@
# Guidelines for /contrib Components
**Authors**: Kimonas Sotirchos kimwnasptd@arrikto.com
The motivation behind this proposal is to fully document expectations and
requirements that components under `/contrib` should satisfy. This will make it
more clear how to use a component, how it integrates with Kubeflow, the problems
it tries to solve as well as dependency versions.
## Goals
* Document requirements that components under `/contrib` should satisfy
* Introduce a process for deprecating unmaintained components
## Non-Goals
* Get into the discussion of which components are considered "Kubeflow" components
* The assumption until now is that components under the Kubeflow GitHub Org live
under the `/apps` dir, and all others under `/contrib`
* Provide a migration plan for components to move out from `/contrib`
* Modify the [`example/kustomization.yaml`](https://github.com/kubeflow/manifests/blob/master/example/kustomization.yaml) with new components
## Proposal
### Component Requirements
Components living under `/contrib` should satisfy some strict requirements to
ensure they are always usable by end-users and contain complementary documentation.
These are the requirements for all components under `/contrib`:
1. There must be a `README.md` file that documents:
* Instructions on how someone can install the component in a Kubeflow cluster
* Since Kubeflow manifests have standardized on [Kustomize](https://kustomize.io/)
we expect all manifests to be a kustomize packages
* How to use the component as part of Kubeflow (examples)
* The problems it tries to solve and the value it brings
* Links to the official documentation of the component
2. There must be an OWNERS file with at least 2 users
3. The component must work with the latest version of Kubeflow, and its
dependencies
4. There must be an `UPGRADE.md` file that documents any instructions users need
to follow when applying manifests of a newer version
5. There needs to be sufficient work on testing
* There must be a script file [python, bash etc] that vefiries the component
is working as expected. This can be something very simple, like submitting a
CustomResource and waiting for it to become Ready
* The maintainers will need to work with the leads of Manifests WG to ensure
there's some basic automation in place that will be running the above script(s)
6. All pods must run with the [baseline PSS set](https://kubernetes.io/docs/concepts/security/pod-security-standards/#baseline)
7. [stretch goal] All pods must run according to the offical Kubernetes
[podsecuritystandards restricted set](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted).
This means explicitly no root containers and dropping all capabilities in your pods securitycontexts.
8. [stretch goal] Running with Istio sidecars
9. [stretch goal] Namespace isolation
At this point we don't want to provide too much of a strict structure for the
README. Developers are free to expose any other information in the README that
they find fit, as long as the above info is exposed.
### Deprecation plan
The proposed criteria for deciding that a component should be deprecated are:
1. The component can not be installed in the minimum K8s version supported by Kubeflow
2. The manifests are not working as expected and result in undeployable Pods
3. The documented examples do not work as expected
If a component meets all the above criteria then it will initially be marked as
UNMAINTAINED/OUT-OF-DATE, in the component's README. Then if a whole Kubeflow release
cycle concludes and the component is still in UNMAINTAINED/OUT-OF-DATE phase and
without any feedback from the OWNERS, it will be removed.
charts/kubeflow/proposals/20230323-end-to-end-testing.md
+395
View File
@@ -0,0 +1,395 @@
# End-to-end Testing
**Authors**: Dominik Fleischmann ([@domFleischmann](https://github.com/domFleischmann)), Kimonas
Sotirchos ([@kimwnasptd](https://github.com/kimwnasptd)), and Anna Jung ([@annajung](https://github.com/annajung))
## Background
Previously, the Kubeflow community leveraged prow optional-test-infra for e2e testing with credit from AWS. After the
optional test infrastructure deprecation notice, all WGs moved their test to GitHub Actions as a temporary solution. Due
to resource constraints of GitHub-hosted runners, the Kubeflow community stopped supporting e2e tests as part of the
migration. In partnership with Amazon, a new AWS account has been created with sponsored credits. With the new AWS
account, the Kubeflow community is no longer limited by resource constraints posed by GitHub Actions. To enable the e2e
test for the Manifest repo, this doc proposes a design to set up the infrastructure needed to run the necessary tests.
References
- [Optional Test Infra Deprecation Notice](https://github.com/kubeflow/testing/issues/993)
- [Alternative solution to removal of test on optional-test-infra](https://github.com/kubeflow/testing/issues/1006)
## Goal
Enable the e2e testing for the Manifest repo and leverage it to shorten the manifest testing phase of the Kubeflow
release cycle and to increase quality of the Kubeflow release by ensuring Kubeflow components and dependencies work
correctly together.
## Proposal
After some initial conversations, it has been agreed to create integration tests based on GitHub Actions, which will
spawn an EC2 instance with enough resources to deploy the complete Kubeflow solution and run some end-to-end testing.
## Implementation
Below lists steps the GitHub actions will perform to complete end-to-end testing
- [Create Credentials required by the AWS](#create-credentials-required-by-the-aws)
- [Create an EC2 instance](#create-an-ec2-instance)
- [Install a Kubernetes on the instance](#install-a-kubernetes-on-the-instance)
- [Deploy Kubeflow](#deploy-kubeflow)
- [Run tests](#run-tests)
- [Log and report errors](#log-and-report-errors)
- [Clean up](#clean-up)
### Create credentials required by the AWS
To leverage AWS, two credentials are required:
- `AWS_ACCESS_KEY_ID`: Specifies an AWS access key associated with an IAM user or role.
- `AWS_SECRET_ACCESS_KEY`: Specifies the secret key associated with the access key. This is essentially the "password"
for the access key.
Both credentials needs to
be [stored as secrets on GitHub](https://docs.github.com/en/actions/security-guides/encrypted-secrets)
and will be accessed in a workflow as environment variables.
```shell
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
```
### Create an EC2 instance
Access the AWS credentials (stored as GH Secrets) and create an EC2 instance
Using [juju](https://juju.is/) as an orchestration, configure AWS credentials and deploy an EC2 instance with the
following configurations
- Image: Ubuntu Server (latest)
- Type: t3a.xlarge
- Root disk: 80G
- Region: us-east-1 (default)
#### Why juju?
Juju allows easy configuration to various cloud providers. In the future, if there comes a reason to shift to another
infrastructure provider, it would allow us to pivot quickly.
While juju provides more capability, the proposal is to use the tool as config management and a medium to deploy and
connect with EC2 instances.
**Note**: Using GitHub Secrets to store AWS credentials will not allow any forked repositories to access the secrets.
### Install a Kubernetes on the Instance
Install Kubernetes on the EC2 instance where Kubeflow will be deployed and tested
To install Kubernetes, we explored two options and propose to use **KinD**
- [Microk8s](#microk8s)
- [KinD](#kind)
#### KinD
Using KinD, install Kubernetes with the existing KinD configuration managed by the Manifest WG.
```shell
# Install dependencies - docker
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common tar
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
apt-cache policy docker-ce
sudo apt install -y docker-ce
sudo systemctl status docker
sudo usermod -a -G docker ubuntu
# Install dependencies - kubectl
sudo curl -L "https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl
sudo chmod +x /usr/local/bin/kubectl
kubectl version --short --client
# Install KinD
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
# Deploy kubernetes using KinD
cd manifests
kind create cluster --config ./tests/gh-actions/kind-cluster.yaml
```
##### Why KinD?
While many tools can be leveraged to deploy Kubernetes, Manifest WG already leverages KinD to run both core and contrib
component tests. By reusing the tool, we can leverage the existing KinD configuration and keep the similarity between
component and e2e testing.
**Note**: KinD is a subproject of Kubernetes but does not automatically release with a new Kubernetes version and does
not follow the Kubernetes release cadence. More details can be found at
[kind/issue#197](https://github.com/kubernetes-sigs/kind/issues/197).
### Deploy Kubeflow
Deploy Kubeflow, in the same manner, the manifests WG documents.
Copy the manifest repo to the AWS instance and use Kustomize to run the Kubeflow installation. After Kustomize
installation is complete, verify all pods are running.
Manifest installation may result in an infinite while loop; therefore, a time limit of 45mins should be set to ensure
installation exits when a problem occurs with Kubeflow installation.
### Run Tests
Execute integration tests to verify the correct functioning of different features using python scripts and jupyter
notebooks.
As the first iteration, test the Kubeflow integration using the
existing [e2e mnist python script](https://github.com/kubeflow/manifests/tree/master/tests/e2e)
and [e2e mnist notebook](https://github.com/kubeflow/pipelines/blob/master/samples/contrib/kubeflow-e2e-mnist/kubeflow-e2e-mnist.ipynb)
.
- [Python script](#python-script)
- [Jupyter notebook](#jupyter-notebook)
Both python and notebook tests the following:
- Kfp and Katib SDK packages (compatibility with other python packages)
- Creation and execution of a pipeline from a user namespace
- Creation and execution of hyperparameter running with Katib from a user namespace
- Creation and execution of distributive training with TFJob from a user namespace
- Creation and execution of inference using KServe from a user namespace
**Note**: The mnist notebook does not test the Kubeflow Notebook resources. In the future, additional verification and
tests should be added to cover various Kubeflow components and features.
#### Python script
Step to run e2e python script from the workflow:
1. Convert e2e mnist notebook to a python script (
reuse [mnist.py](https://github.com/kubeflow/manifests/blob/master/tests/e2e/mnist.py))
2. Run mnist python script outside of the cluster (
reuse [runner.sh](https://github.com/kubeflow/manifests/blob/master/tests/e2e/runner.sh))
#### Jupyter notebook
Step to run e2e notebook from the workflow:
1. Get e2e mnist notebook
1. To run the existing e2e mnist notebook, modification needs to be made in the last step to wait for the triggered
run to finish running before executing. Changes proposed are defined below and a pull request will need to be
made in the future to avoid copying mnist notebook into the manifest directory.
```shell
import numpy as np
import time
from PIL import Image
import requests
# Pipeline Run should be succeeded.
run_status = kfp_client.get_run(run_id=run_id).run.status
if run_status == None:
print("Waiting for the Run {} to start".format(run_id))
time.sleep(60)
run_status = kfp_client.get_run(run_id=run_id).run.status
while run_status == "Running":
print("Run {} is in progress".format(run_id))
time.sleep(60)
run_status = kfp_client.get_run(run_id=run_id).run.status
if run_status == "Succeeded":
print("Run {} has Succeeded\n".format(run_id))
# Specify the image URL here.
image_url = "https://raw.githubusercontent.com/kubeflow/katib/master/examples/v1beta1/kubeflow-pipelines/images/9.bmp"
image = Image.open(requests.get(image_url, stream=True).raw)
data = np.array(image.convert('L').resize((28, 28))).astype(float).reshape(-1, 28, 28, 1)
data_formatted = np.array2string(data, separator=",", formatter={"float": lambda x: "%.1f" % x})
json_request = '{{ "instances" : {} }}'.format(data_formatted)
# Specify the prediction URL. If you are runing this notebook outside of Kubernetes cluster, you should set the Cluster IP.
url = "http://{}-predictor-default.{}.svc.cluster.local/v1/models/{}:predict".format(name, namespace, name)
time.sleep(60)
response = requests.post(url, data=json_request)
print("Prediction for the image")
display(image)
print(response.json())
else:
raise Exception("Run {} failed with status {}\n".format(run_id, kfp_client.get_run(run_id=run_id).run.status))
```
2. Move the mnist notebook into the cluster
```shell
kubectl -n kubeflow-user-example-com create configmap <configmap name> --from-file kubeflow-e2e-mnist.ipynb
```
3. Create a PodDefault to allow access to Kubeflow pipelines
```shell
apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
name: access-ml-pipeline
namespace: kubeflow-user-example-com
spec:
desc: Allow access to Kubeflow Pipelines
selector:
matchLabels:
access-ml-pipeline: "true"
env:
- ## this environment variable is automatically read by `kfp.Client()`
## this is the default value, but we show it here for clarity
name: KF_PIPELINES_SA_TOKEN_PATH
value: /var/run/secrets/kubeflow/pipelines/token
volumes:
- name: volume-kf-pipeline-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 7200
## defined by the `TOKEN_REVIEW_AUDIENCE` environment variable on the `ml-pipeline` deployment
audience: pipelines.kubeflow.org
volumeMounts:
- mountPath: /var/run/secrets/kubeflow/pipelines
name: volume-kf-pipeline-token
readOnly: true
```
4. Run the notebook programmatically using a Kubernetes resource Job or Notebook
```shell
apiVersion: batch/v1
kind: Job
metadata:
name: test-notebook-job
namespace: kubeflow-user-example-com
spec:
backoffLimit: 1
activeDeadlineSeconds: 1200
template:
metadata:
labels:
access-ml-pipeline: "true"
spec:
restartPolicy: Never
initContainers:
- name: copy-notebook
image: busybox
command: ['sh', '-c', 'cp /scripts/* /etc/kubeflow-e2e/']
volumeMounts:
- name: e2e-test
mountPath: /scripts
- name: kubeflow-e2e
mountPath: /etc/kubeflow-e2e
containers:
- image: kubeflownotebookswg/jupyter-scipy:v1.6.1
imagePullPolicy: IfNotPresent
name: execute-notebook
command:
- /bin/sh
- -c
- |
jupyter nbconvert --to notebook --execute /etc/kubeflow-e2e/kubeflow-e2e-mnist.ipynb;
x=$(echo $?); curl -fsI -X POST http://localhost:15020/quitquitquit && exit $x;
volumeMounts:
- name: kubeflow-e2e
mountPath: /etc/kubeflow-e2e
serviceAccountName: default-editor
volumes:
- name: e2e-test
configMap:
name: e2e-test
- name: kubeflow-e2e
emptyDir: {}
```
5. Verify Job succeeded or failed
```shell
kubectl -n kubeflow-user-example-com wait --for=condition=complete --timeout=1200s job/test-notebook-job
```
### Log and Report Errors
Report logs generated in the EC2 instance back to GitHub actions for users.
For failures in the workflow steps, generate inspect logs, pod logs, and describe logs. Copy the generated logs back to
the GitHub Actions system and use [actions/upload-artifact@v2](https://github.com/actions/upload-artifact)
to allow users to access the logs when necessary.
**Note**: As default, artifacts are retained for 90 days. The number of retention days is configurable.
### Clean Up
Regardless of the success or failure of the workflow, at the end of the workflow, the EC2 instance is deleted to ensure
there are no resources left behind.
## Debugging
To debug any failed step of the GitHub Actions
workflow, [debugging with ssh](https://github.com/marketplace/actions/debugging-with-ssh)
or other similar tools can be used to ssh into the GitHub system. In the GitHub system, juju can be used to connect to
an AWS EC2 instance.
**Notes**:
- GitHub secrets are limited to the Manifest repo and do not cascade to forked repositories. To debug, users must set up
their own AWS secrets.
- To debug the AWS EC2 instance without ssh into the GitHub system, you must have access to AWS credentials. Access to
AWS credentials is limited to [Manifest WG approvers](https://github.com/kubeflow/manifests/blob/master/OWNERS).
## Proof of Concept Workflow
The POC code
is [available](https://github.com/DomFleischmann/manifests/blob/aj-dev/.github/workflows/aws_e2e_tests.yaml)
with examples of both [successful](https://github.com/DomFleischmann/manifests/actions/runs/4118561167/jobs/7111228604)
and [failed](https://github.com/DomFleischmann/manifests/actions/runs/4119052861) runs.
The proposed end-to-end workflow has been tested with the following Kubernetes and Kubeflow versions
- 1.22 Kubernetes and [1.6.1 Kubeflow release](https://github.com/kubeflow/manifests/releases/tag/v1.6.1) (microk8s)
- 1.24 Kubernetes and main branch of the manifest
repo ([last commit](https://github.com/DomFleischmann/manifests/commit/8e5714171f1fd5b00f59f436e9ab8cb45a0f30e3)) (
microk8s)
- 1.25 Kuberentes and main branch of the manifest
repo ([last commit](https://github.com/DomFleischmann/manifests/commit/8e5714171f1fd5b00f59f436e9ab8cb45a0f30e3)) (
kind)
### Alternative solutions considered
#### Prow
While there are some existing tests with Prow, those tests were discarded due to them not having been updated in 2 years
and there being a high amount of complexity in these tests. After some investigation, the Manifests Working Group
decided that it would be more work adapting those tests to the current state of manifests than starting from scratch
with lower complexity.
#### Self-hosted runners
Self-hosted runners are not recommended with public repositories due to security concerns with how it behaves on a pull
request made by a forked repository.
#### MicroK8s
Instead of KinD, [microk8s](https://microk8s.io/) was considered as an alternative to install Kubernetes.
Below shows the steps required in the workflow to install microk8s and to install Kubernetes using microk8s. During
the Kubernetes installation, you must enable [dns](https://microk8s.io/docs/addon-dns),
[storage](https://microk8s.io/docs/addon-hostpath-storage), [ingress](https://microk8s.io/docs/addon-ingress),
[loadbalancer](https://microk8s.io/docs/addon-metallb), and [rbac](https://microk8s.io/docs/multi-user).
```shell
# Install microk8s
sudo snap install microk8s --classic --channel ${{ matrix.microk8s }}
sudo apt update
sudo usermod -a -G microk8s ubuntu
# Install dependencies - kubectl
sudo snap alias microk8s.kubectl kubectl
# Deploy kubernetes using microk8s
sudo snap install microk8s --classic --channel 1.24/stable
microk8s enable dns hostpath-storage ingress metallb:10.64.140.43-10.64.140.49 rbac
```
**Note**: microk8s requires IP address pool when enabling dns, address pool of 10.64.140.43-10.64.140.49 is an arbitrary
decision.
charts/kubeflow/proposals/20240606-jwt-handling.md
+136
View File
@@ -0,0 +1,136 @@
# Standardise JWTs Usage
**Authors**: Kimonas Sotirchos @kimwnasptd
## Requirements
- OIDC logic, via [Istio external authoriser](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/), should be adding `id_token` to http requests in `Authorization: Bearer <token>` headers
## Scope
This proposal aims to standardise how the Kubeflow backends should be handling the user information (name of user, groups they belong to), living in JWTs and http headers.
This proposal takes as a requirement that users should be able to use K8s Tokens as `Authorization: Bearer <token>` headers in their request from inside the cluster. Note that the issuer of the tokens should not be relevant to the Kubeflow applications. It'll be up to the service-mesh to verify them, and be able to work with multiple issuers (i.e. Dex, K8s etc). The applications will only care about the user/groups information from the tokens.
### In
- Which component should be validating JWTs (id-tokens from OIDC or K8s ServiceAccount tokens)
- Define where the backends should expect to find user related information
- Define how different token issuers (i.e. Dex, K8s etc) should be handled
### Out
- Proposing code changes to existing components
## Current State
As of Kubeflow 1.8 the user information has been injected into requests as the `kubeflow-userid` header, from the AuthService (replaced by `oauth2-proxy`). For this approach to be secure there are the following patterns that Kubeflow follows:
1. Backends in the `kubeflow` namespace that need to know the user identity rely on `kubeflow-userid` headers in http requests.
2. The AuthService adds the `kubeflow-userid` header to all authenticated requests.
3. Only requests from the Istio IngressGateway are trusted to have the `kubeflow-userid` header
1. Backends that are not exposed to user namespaces (i.e. jupyter-web-app) are only reachable via the Istio IngressGateway.
2. The KFP backend [explicitly drops requests](https://github.com/kubeflow/manifests/blob/96ce068e16b2a707464471bddc0d2a58e403d1fc/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml#L37) from user namespaces if they have this header
3. In-cluster Pods that want to talk to `kubeflow` workloads, which understands identity, are [using a K8s ServiceAccount Token](https://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kubeflow-subfrom-inside-clustersub)
### Limitations
- Not able to express `AuthorizationPolicies` for group header in Istio
- Limited possibility to use custom JWT claims as a source of information about the authenticated user
To accommodate the above limitations and improve the authentication and authorization flow in terms of security, maintenance and flexibility
we propose to add the JWT to the `Authorization` header, so it can be digested by Istio and have the user details securely
injected into the `Authorization` headers. This will also enable us to define policies in the future for better handling of groups.
https://istio.io/latest/docs/tasks/security/authorization/authz-jwt/
But the above creates the following topics that require an agreement on how to handle them:
1. There will be `id_tokens` from different issuers (i.e. from Dex, K8s) that the platform will need to handle
2. Information of user is both in `kubeflow-userid` and in `id_token` of http request, for Kubeflow components to deduce the identity from
3. It's not clear if backends should be validating the JWT (i.e. KFP right now validates ServiceAccount tokens [`1`](https://github.com/kubeflow/pipelines/blob/2.2.0/backend/src/apiserver/auth/authenticator_token_review.go#L47-L58) [`2`](https://github.com/kubeflow/pipelines/blob/2.2.0/backend/src/apiserver/resource/resource_manager.go#L1698-L1699) )
## Specification
The goal of this proposal is to provide a uniform way for all backends to handle identity tokens and to specify
which levels of the stack are responsible for which parts.
This spec proposes to standardise on the following high level agreement, for new backends:
1. Requests hitting `kubeflow` apps, which expect user identity in requests, should have a JWT in `Authorization: Bearer <token>` header
2. The service-mesh is responsible for validating the JWTs
1. The service-mesh must drop (401) a request if the JWT is invalid (`RequestAuthentication`)
2. The service-mesh must drop (403) a request if the JWT is not present, and the application expects requests to have a user identity (`AuthorizationPolicy`)
3. The backends are not responsible for validating the JWTs or their existence
4. The service-mesh must expose user and groups to `kubeflow-userid` and `kubeflow-groups` headers, after validating JWTs
1. if the mesh will not override these headers, then users could forge requests and impersonate other users by setting the header and any valid token
1. for User to Machine traffic, the `email` claim from `id_token` should be used by default but it should also allow parameterization to allow using different claims. This is doable via `RequestAuthentication` with object per issuer
2. in case of K8s ServiceAccount tokens the `sub` claim will be used. `sub` claim format is `system:serviceaccount:<sa-namespace>:<sa-name>`
3. in case of Dex tokens it would be `kimonas@email.com`
5. The backends will use the information from the headers and not deal with JWTs
1. `SubjectAccessReviews` should be made for the `user` and `groups` that were exposed from the headers, independently of the issuer of the token
2. [`SubjectAccessReview API`](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) uses the `user`,
`groups`, `resource` and `verb` details to verify the access against K8s RBAC, which allows defining authorization to specific actions based on
K8s standard RBAC implementation
With the above implementation we move all the logic of handling the JWTs to the service-mesh and leave only the business logic to the apps. This also means that the apps don't care about token "types" (i.e. dex, k8s tokens etc) and only have to look at the corresponding headers.
This proposal aims to put more focus on keeping and validating `id_tokens` but also bridging to the existing functionality of the backends, to avoid extensive changes.
### Implementation
The technical details for the above proposal translate to the following
1. Common Kubeflow manifests, for all components, for configuring Istio for supporting multiple issuers ([Dex](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml) and [K8s-m2m](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml)), via `RequestAuthentication` objects
2. `AuthorizationPolicy` objects of components, for allowing access from Istio IngressGateway, will need to be extended for also requiring a JWT
3. Backends that need to be accessible from other user-namespaces will need to have an `AuthorizationPolicy` that allows any request, only if it has a JWT
4. Backends don't need any logic for validating the JWTs and their existence
6. `RequestAuthentication` objects, per issuer, should expose the corresponding token claims to the `kubeflow-userid` and `kubeflow-groups` headers
7. Backends only need to care about `kubeflow-userid` and `kubeflow-groups` headers
#### Requiring a JWT
The service-mesh will need to drop requests (403) that don't have any JWT, for services that expect user identity in the requests.
This can be achieved in multiple ways:
- By using `requestPrincipals`
- By using `request.auth.claims[iss]` in the `when` condition of an `AuthorizationPolicy` rule
The recommended way is to use `requestPrincipals: ["*"]`, as the [Istio docs suggest](https://istio.io/latest/docs/tasks/security/authorization/authz-jwt/), to accept only requests that have a valid JWT.
If an admin would like to further limit access to Kubeflow services based on specific issuers, they can do so by updating the `AuthorizationPolicies`
to instead use `request.auth.claims[iss]`.
From the above, the `AuthorizationPolicy` for the jupyter-web-app should look like:
```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
labels:
app: jupyter-web-app
kustomize.component: jupyter-web-app
name: jupyter-web-app
namespace: kubeflow
spec:
action: ALLOW
rules:
- from:
- source:
principals:
- cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
requestPrincipals: # new! Require JWT
- '*'
selector:
matchLabels:
app: jupyter-web-app
```
Similarly, KFP API Server `AuthorizationPolicy`, for allowing requests from all namespaces, should be:
```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
labels:
app.kubernetes.io/component: ml-pipeline
app.kubernetes.io/name: kubeflow-pipelines
application-crd-id: kubeflow-pipelines
name: ml-pipeline
namespace: kubeflow
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/ml-pipeline
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
- cluster.local/ns/kubeflow/sa/ml-pipeline-persistenceagent
- cluster.local/ns/kubeflow/sa/ml-pipeline-scheduledworkflow
- cluster.local/ns/kubeflow/sa/ml-pipeline-viewer-crd-service-account
- cluster.local/ns/kubeflow/sa/kubeflow-pipelines-cache
- from:
- source:
requestPrincipals: # new! Allow request from any source, as long as it has JWT
- '*'
selector:
matchLabels:
app: ml-pipeline
```
charts/kubeflow/proposals/README.md
+16
View File
@@ -0,0 +1,16 @@
## Proposals
If you wish to add a larger feature or make a major refactor to Manifests we encourage folks to write up a proposal document.
Though our process is not formal, the convention is to create a PR against Manifests with your proposal as markdown in the
proposals folder. Proposal reviews and feedback will happen on the PR with the proposal.
```
manifests/proposals <- folder
YYYYMMDD-title.md <- file
```
In your proposal it is good to consider and include some of the following:
* Goals
* Non-Goals
* Does this break any existing functionality?
* Does this fix/solve any outstanding issues?