Add kubeflow/v1.10.0

This commit is contained in:
wbsong111
2025-06-24 12:03:10 +09:00
parent 0132496142
commit 6e8dd89e48
1531 changed files with 120984 additions and 262120 deletions
+5
View File
@@ -0,0 +1,5 @@
reviewers:
- annajung
- juliusvonkohout
approvers:
- juliusvonkohout
+4
View File
@@ -0,0 +1,4 @@
The contrib directory contains third-party applications maintained externally.
All contrib components must meet the [requirements](https://github.com/kubeflow/manifests/blob/master/proposals/20220926-contrib-component-guidelines.md#component-requirements)
to be considered active and failure to meet them will result in deprecation and removal from the Kubeflow manifest repository.
+11
View File
@@ -0,0 +1,11 @@
KUBERAY_RELEASE_VERSION ?= 1.3.2
KUBERAY_HELM_CHART_REPO ?= https://ray-project.github.io/kuberay-helm/
.PHONY: kuberay-operator/base
kuberay-operator/base:
mkdir -p kuberay-operator/base
cd kuberay-operator/base && helm template --include-crds kuberay-operator kuberay-operator --version ${KUBERAY_RELEASE_VERSION} --repo ${KUBERAY_HELM_CHART_REPO} > resources.yaml
.PHONY: test
test:
./test.sh ${KF_PROFILE}
+5
View File
@@ -0,0 +1,5 @@
approvers:
- juliusvonkohout
reviewers:
- juliusvonkohout
- kimwnasptd
+150
View File
@@ -0,0 +1,150 @@
> Credit: This manifest refers a lot to the engineering blog ["Building a Machine Learning Platform with Kubeflow and Ray on Google Kubernetes Engine"](https://cloud.google.com/blog/products/ai-machine-learning/build-a-ml-platform-with-kubeflow-and-ray-on-gke) from Google Cloud.
# Ray
[Ray](https://github.com/ray-project/ray) is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for simplifying ML compute.
<figure>
<img
src="assets/map-of-ray.png"
alt="Ray">
<figcaption>Stack of Ray libraries - unified toolkit for ML workloads. (ref: https://docs.ray.io/en/latest/ray-overview/index.html)</figcaption>
</figure>
# KubeRay
[KubeRay](https://github.com/ray-project/kuberay) is an open-source Kubernetes operator for Ray. It provides several CRDs to simplify managing Ray clusters on Kubernetes. We will integrate Kubeflow and KubeRay in this document.
# Requirements
* Dependencies
* `kustomize`: v5.4.3+ (Kubeflow manifest is sensitive to `kustomize` version.)
* `Kubernetes`: v1.32+
* Computing resources:
* 16GB RAM
* 8 CPUs
# Example
<figure>
<img
src="assets/architecture.svg"
alt="ray/kubeflow integration">
<figcaption>Note: (1) Kubeflow Central Dashboard will be renamed to workbench in the future. (2) Kubeflow Pipeline (KFP) is an important component of Kubeflow, but it is not included in this example.</figcaption>
</figure>
## Step 1: Install Kubeflow
* This example installs Kubeflow with the master branch
* Install all Kubeflow official components and all common services using [one command](https://github.com/kubeflow/manifests/tree/master#install-with-a-single-command).
* If you do not want to install all components, you can comment out **KNative**, **Katib**, **Tensorboards Controller**, **Tensorboard Web App**, **Training Operator**, and **KServe** from [example/kustomization.yaml](https://github.com/kubeflow/manifests/blob/master/example/kustomization.yaml).
## Step 2: Install KubeRay operator
We never ever break Kubernetes standards and do not use the "default" namespace, but a proper one, in our case "kubeflow" for the ray operator.
```sh
# Install a KubeRay operator and custom resource definitions.
kustomize build kuberay-operator/overlays/kubeflow | kubectl apply --server-side -f -
# Check KubeRay operator
kubectl get pod -l app.kubernetes.io/component=kuberay-operator -n kubeflow
# NAME READY STATUS RESTARTS AGE
# kuberay-operator-5b8cd69758-rkpvh 1/1 Running 0 6m23s
```
> If you are creating a new namespace other than the kubeflow-user-example-com please follow below step otherwise skip the step.
## Step 3: Create a namespace
```sh
# Create a namespace: example-"development"
kubectl create ns development
# Enable istio-injection for the namespace
kubectl label namespace development istio-injection=enabled
# After creating the namespace, You have to do below mentioned changes in raycluster_example.yaml file(Required changes are also mentioned as comments in yaml file itself)
# 01. Replace the namesapce of AuthorizationPolicy principal
principals:
- "cluster.local/ns/development/sa/default-editor"
# 02. Replace the namespace of node-ip-address of headGroupSpec and workerGroupSpec
node-ip-address: $(hostname -I | tr -d ' ' | sed 's/\./-/g').raycluster-istio-headless-svc.development.svc.cluster.local
```
## Step 4: Install RayCluster
```sh
# Create a RayCluster CR, and the KubeRay operator will reconcile a Ray cluster
# with 1 head Pod and 1 worker Pod.
# $MY_KUBEFLOW_USER_NAMESPACE is the namespace that has been created in the above step.
export MY_KUBEFLOW_USER_NAMESPACE=development
kubectl apply -f raycluster_example.yaml -n $MY_KUBEFLOW_USER_NAMESPACE
# Check RayCluster
kubectl get pod -l ray.io/cluster=kubeflow-raycluster -n $MY_KUBEFLOW_USER_NAMESPACE
# NAME READY STATUS RESTARTS AGE
# kubeflow-raycluster-head-p6dpk 1/1 Running 0 70s
# kubeflow-raycluster-worker-small-group-l7j6c 1/1 Running 0 70s
#Check Raycluster headless service
kubectl get svc -n $MY_KUBEFLOW_USER_NAMESPACE
```
* `raycluster_example.yaml` uses `rayproject/ray:2.23.0-py311-cpu` as its OCI image. Ray is very sensitive to the Python versions and Ray versions between the server (RayCluster) and client (JupyterLab) sides. This image uses:
* Python 3.11
* Ray 2.23.0
## Step 5: Forward the port of Istio's Ingress-Gateway
* Follow the [instructions](https://github.com/kubeflow/manifests/tree/master#port-forward) to forward the port of Istio's Ingress-Gateway and log in to Kubeflow Central Dashboard.
## Step 6: Create a JupyterLab via Kubeflow Central Dashboard
* Click "Notebooks" icon in the left panel.
* Click "New Notebook"
* Select `kubeflownotebookswg/jupyter-scipy:v1.9.2` as OCI image (or any other with the same python version)
* Click "Launch"
* Click "CONNECT" to connect into the JupyterLab instance.
## Step 7: Use Ray client in the JupyterLab to connect to the RayCluster
* As I mentioned in Step 3, Ray is very sensitive to the Python versions and Ray versions between the server (RayCluster) and client (JupyterLab) sides.
```sh
# Check Python version. The version's MAJOR and MINOR should match with RayCluster (i.e. Python 3.11.9)
python --version
# Python 3.11.9
pip install -U ray[default]==2.23.0
```
* Connect to RayCluster via Ray client.
```python
# Open a new .ipynb page.
import ray
# For other namespaces use ray://${RAYCLUSTER_HEAD_SVC}.${NAMESPACE}.svc.cluster.local:${RAY_CLIENT_PORT}
# But we use of course our per namespace ray cluster to have multi-tenancy and
# We never ever use "default" as namespace since this would violate Kubernetes standards
ray.init(address="ray://kubeflow-raycluster-head-svc:10001")
print(ray.cluster_resources())
# {'node:10.244.0.41': 1.0, 'memory': 3000000000.0, 'node:10.244.0.40': 1.0, 'object_store_memory': 805386239.0, 'CPU': 2.0}
# Try Ray task
@ray.remote
def f(x):
return x * x
futures = [f.remote(i) for i in range(4)]
print(ray.get(futures)) # [0, 1, 4, 9]
# Try Ray actor
@ray.remote
class Counter(object):
def __init__(self):
self.n = 0
def increment(self):
self.n += 1
def read(self):
return self.n
counters = [Counter.remote() for i in range(4)]
[c.increment.remote() for c in counters]
futures = [c.read.remote() for c in counters]
print(ray.get(futures)) # [1, 1, 1, 1]
```
# Upgrading
See [UPGRADE.md](UPGRADE.md) for more details.
@@ -0,0 +1,6 @@
# Upgrading
```sh
# Step 1: Update KUBERAY_RELEASE_VERSION in Makefile
# Step 2: Create new KubeRay operator manifest
make kuberay-operator/base
```
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 217 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

@@ -0,0 +1,52 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubeflow-kuberay-admin
labels:
app: kuberay-operator
app.kubernetes.io/name: kuberay-operator
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"
rules: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubeflow-kuberay-edit
labels:
app: kuberay-operator
app.kubernetes.io/name: kuberay-operator
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"
rules:
- apiGroups:
- ray.io
resources:
- "*"
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubeflow-kuberay-view
labels:
app: kuberay-operator
app.kubernetes.io/name: kuberay-operator
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-view: "true"
rules:
- apiGroups:
- ray.io
resources:
- "*"
verbs:
- get
- list
- watch
---
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubeflow
resources:
- resources.yaml
- aggregated-roles.yaml
# The securitycontext for PSS restricted is now directly provided in the upstream manifests
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,13 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kuberay-operator
namespace: kubeflow
spec:
template:
spec:
containers:
- name: kuberay-operator
env:
- name: ENABLE_INIT_CONTAINER_INJECTION
value: "false" # TODO maybe we can drop that with istio native sidecars
@@ -0,0 +1,6 @@
namespace: kubeflow
resources:
- ../../base
patches:
- path: disable-injection.yaml
@@ -0,0 +1,3 @@
namespace: kubeflow
resources:
- ../../base
@@ -0,0 +1,167 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-ray-workers-head
spec:
action: ALLOW
rules:
- from:
- source:
principals:
# kubeflow-user-example-com should be replaced with the namespace where the Ray cluster is being deployed
# TODO automatically use the current namespace
- "cluster.local/ns/kubeflow-user-example-com/sa/default-editor"
- to:
- operation:
ports:
- "6379"
- "6380"
- "6381"
- "6382"
- "6383"
- "52365"
- "8080"
- "10012"
---
apiVersion: v1
kind: Service
metadata:
labels:
ray.io/headless-worker-svc: raycluster-istio
name: raycluster-istio-headless-svc
spec:
clusterIP: None
selector:
ray.io/cluster: kubeflow-raycluster
publishNotReadyAddresses: true
ports:
- name: node-manager-port
port: 6380
appProtocol: grpc
- name: object-manager-port
port: 6381
appProtocol: grpc
- name: runtime-env-agent-port
port: 6382
appProtocol: grpc
- name: dashboard-agent-grpc-port
port: 6383
appProtocol: grpc
- name: dashboard-agent-listen-port
port: 52365
appProtocol: http
- name: metrics-export-port
port: 8080
appProtocol: http
- name: max-worker-port
port: 10012
appProtocol: grpc
---
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: kubeflow-raycluster
spec:
rayVersion: '2.44.1'
enableInTreeAutoscaling: true
autoscalerOptions:
upscalingMode: Default
idleTimeoutSeconds: 60
headGroupSpec:
rayStartParams:
num-cpus: '1'
node-manager-port: '6380'
object-manager-port: '6381'
runtime-env-agent-port: '6382'
dashboard-agent-grpc-port: '6383'
dashboard-agent-listen-port: '52365'
metrics-export-port: '8080'
max-worker-port: '10012'
# kubeflow-user-example-com should be replaced with the namespace where the Ray cluster is being deployed
# TODO automatically use the current namespace
node-ip-address: $(hostname -I | tr -d ' ' | sed 's/\./-/g').raycluster-istio-headless-svc.kubeflow-user-example-com.svc.cluster.local
template:
metadata:
labels:
sidecar.istio.io/inject: "true"
spec:
serviceAccountName: default-editor
containers:
- name: ray-head # TODO why call it headless with a head...
image: rayproject/ray:2.44.1-py311-cpu
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","ray stop"]
volumeMounts:
- mountPath: /tmp/ray
name: ray-logs
resources:
limits:
cpu: "1"
memory: "2G"
requests:
cpu: "100m"
memory: "2G"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumes:
- name: ray-logs
emptyDir: {}
workerGroupSpecs:
- replicas: 1
minReplicas: 1
maxReplicas: 1
groupName: small-group
rayStartParams:
num-cpus: '1'
node-manager-port: '6380'
object-manager-port: '6381'
runtime-env-agent-port: '6382'
dashboard-agent-grpc-port: '6383'
dashboard-agent-listen-port: '52365'
metrics-export-port: '8080'
max-worker-port: '10012'
# kubeflow-user-example-com should be replaced with the namespace where the Ray cluster is being deployed
# TODO automatically use the current namespace
node-ip-address: $(hostname -I | tr -d ' ' | sed 's/\./-/g').raycluster-istio-headless-svc.kubeflow-user-example-com.svc.cluster.local
template:
metadata:
labels:
sidecar.istio.io/inject: "true"
spec:
serviceAccountName: default-editor
containers:
- name: ray-worker
image: rayproject/ray:2.44.1-py311-cpu
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","ray stop"]
# use volumeMounts.Optional.
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
volumeMounts:
- mountPath: /tmp/ray
name: ray-logs
resources:
limits:
cpu: "1"
memory: "1G"
requests:
cpu: "300m"
memory: "1G"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumes:
- name: ray-logs
emptyDir: {}
+90
View File
@@ -0,0 +1,90 @@
#!/bin/bash
set -euxo
NAMESPACE=$1
TIMEOUT=120 # timeout in seconds
SLEEP_INTERVAL=30 # interval between checks in seconds
RAY_VERSION=2.44.1
start_time=$(date +%s)
for ((i=0; i<TIMEOUT; i+=2)); do
if [[ $(kubectl get namespace $NAMESPACE --no-headers 2>/dev/null | wc -l) -eq 1 ]]; then
echo "Namespace $NAMESPACE created."
break
fi
current_time=$(date +%s)
elapsed_time=$((current_time - start_time))
if [ "$elapsed_time" -ge "$TIMEOUT" ]; then
echo "Timeout exceeded. Namespace $NAMESPACE not created."
exit 1
fi
echo "Waiting for namespace $NAMESPACE to be created..."
sleep 2
done
echo "Namespace $NAMESPACE has been created!"
kubectl label namespace $NAMESPACE istio-injection=enabled
kubectl get namespaces --selector=istio-injection=enabled
# Install KubeRay operator
kustomize build kuberay-operator/overlays/kubeflow | kubectl -n kubeflow apply --server-side -f -
# Wait for the operator to be ready.
kubectl -n kubeflow wait --for=condition=available --timeout=600s deploy/kuberay-operator
kubectl -n kubeflow get pod -l app.kubernetes.io/component=kuberay-operator
# Install RayCluster components
kubectl -n $NAMESPACE apply -f raycluster_example.yaml
# Wait for the RayCluster to be ready.
sleep 5
kubectl -n $NAMESPACE wait --for=condition=Ready pod -l ray.io/cluster=kubeflow-raycluster --timeout=900s
kubectl -n $NAMESPACE logs -l ray.io/cluster=kubeflow-raycluster,ray.io/node-type=head
# Forward the port of Dashboard
sleep 5
kubectl -n $NAMESPACE port-forward --address 0.0.0.0 svc/kubeflow-raycluster-head-svc 8265:8265 &
PID=$!
echo "Forward the port 8265 of Ray head in the background process: $PID"
# Send a curl command to test basic Ray functionality.
sleep 5
output=$(curl -H "Content-Type: application/json" localhost:8265/api/version)
echo "output: ${output}"
# output format: {"version": ..., "ray_version": RAY_VERSION, "ray_commit": ...}
if echo "${output}" | grep -q $RAY_VERSION; then
echo "Test succeeded!"
else
echo "Test failed!"
exit 1
fi
# Delete RayCluster
kubectl -n $NAMESPACE delete -f raycluster_example.yaml
# Wait for all Ray Pods to be deleted.
start_time=$(date +%s)
for ((i=0; i<TIMEOUT; i+=SLEEP_INTERVAL)); do
pods=$(kubectl -n $NAMESPACE get pods -o json | jq '.items | length')
if [ "$pods" -eq 0 ]; then
kill $PID
break
fi
current_time=$(date +%s)
elapsed_time=$((current_time - start_time))
if [ "$elapsed_time" -ge "$TIMEOUT" ]; then
echo "Timeout exceeded. Exiting loop."
exit 1
fi
sleep $SLEEP_INTERVAL
done
# Delete KubeRay operator
kustomize build kuberay-operator/base | kubectl -n kubeflow delete -f -
@@ -0,0 +1,6 @@
approvers:
# - pschoen-itsc
- juliusvonkohout
reviewers:
# - pschoen-itsc
- juliusvonkohout
@@ -0,0 +1,51 @@
# SeaweedFS
- [Official documentation](https://github.com/seaweedfs/seaweedfs/wiki)
- [Official repository](https://github.com/seaweedfs/seaweedfs)
SeaweedFS is a simple and highly scalable distributed file system. It has an S3 interface which makes it usable as an object store for kubeflow.
## Prerequisites
- Kubernetes (any recent Version should work)
- You should have `kubectl` available and configured to talk to the desired cluster.
- `kustomize`
- If you installed kubeflow with minio, use the `istio` dir instead of `base` for the kustomize commands.
## Compile manifests
```bash
kubectl kustomize ./base/
```
## Install SeaweedFS
**WARNING**
This replaces the service `minio-service` and will redirect the traffic to seaweedfs.
```bash
# Optional, but recommended to backup existing minio-service
kubectl get -n kubeflow svc minio-service -o=jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' > svc-minio-service-backup.json
kubectl kustomize ./base/ | kubectl apply -f -
```
## Verify deployment
Run
```bash
./test.sh
```
With the ready check on the container it already verifies that the S3 starts correctly.
You can then use it with the endpoint at http://localhost:8333.
To create access keys open a shell on the pod and use `weed shell` to configure your instance.
Create a user with the command `s3.configure -user <username> -access_key <access-key> -secret-key <secret-key> -actions Read:<my-bucket>/<my-prefix>,Write::<my-bucket>/<my-prefix> -apply`
Documentation for this can also be found [here](https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API).
## Uninstall SeaweedFS
```bash
kubectl kustomize ./base/ | kubectl delete -f -
# Restore minio-service from backup
kubectl apply -f svc-minio-service-backup.json
```
@@ -0,0 +1,3 @@
# Upgrade SeaweedFS
Change the image tag in the Deployment to the desired version. You can find the available images [here](https://hub.docker.com/r/chrislusf/seaweedfs).
@@ -0,0 +1,29 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubeflow
resources:
- seaweedfs/
- ../../../apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user
configMapGenerator:
- name: kubeflow-pipelines-profile-controller-code
behavior: replace
files:
- pipeline-profile-controller/sync.py
patches:
- path: minio-service-patch.yaml
- path: pipeline-profile-controller/deployment.yaml
- patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: ml-pipeline-ui
spec:
template:
spec:
containers:
- name: ml-pipeline-ui
env:
- name: ARTIFACTS_SERVICE_PROXY_ENABLED
value: 'false'
$patch: merge
@@ -0,0 +1,13 @@
apiVersion: v1
kind: Service
metadata:
name: minio-service
namespace: kubeflow
spec:
ports:
- name: http
port: 9000
protocol: TCP
targetPort: 8333
selector:
app: seaweedfs
@@ -0,0 +1,42 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubeflow-pipelines-profile-controller
spec:
template:
spec:
containers:
- name: profile-controller
securityContext:
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
# We just need an image with the python botocore library installed
image: docker.io/alpine/k8s:1.32.3
command: ["python", "/hooks/sync.py"]
env:
- name: KFP_VERSION
valueFrom:
configMapKeyRef:
name: pipeline-install-config
key: appVersion
- name: AWS_ENDPOINT_URL
value: http://seaweedfs:8111
- name: AWS_REGION
value: us-east-1
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: accesskey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: mlpipeline-minio-artifact
key: secretkey
@@ -0,0 +1,243 @@
# Copyright 2020-2021 The Kubeflow Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import os
import base64
# From awscli installed in alpine/k8s image
import botocore.session
S3_BUCKET_NAME = 'mlpipeline'
session = botocore.session.get_session()
# To interact with seaweedfs user management. Region does not matter.
iam = session.create_client('iam', region_name='foobar')
def main():
settings = get_settings_from_env()
server = server_factory(**settings)
server.serve_forever()
def get_settings_from_env(controller_port=None,
visualization_server_image=None, frontend_image=None,
visualization_server_tag=None, frontend_tag=None, disable_istio_sidecar=None):
"""
Returns a dict of settings from environment variables relevant to the controller
Environment settings can be overridden by passing them here as arguments.
Settings are pulled from the all-caps version of the setting name. The
following defaults are used if those environment variables are not set
to enable backwards compatibility with previous versions of this script:
visualization_server_image: ghcr.io/kubeflow/kfp-visualization-server
visualization_server_tag: value of KFP_VERSION environment variable
frontend_image: ghcr.io/kubeflow/kfp-frontend
frontend_tag: value of KFP_VERSION environment variable
disable_istio_sidecar: Required (no default)
minio_access_key: Required (no default)
minio_secret_key: Required (no default)
"""
settings = dict()
settings["controller_port"] = \
controller_port or \
os.environ.get("CONTROLLER_PORT", "8080")
settings["visualization_server_image"] = \
visualization_server_image or \
os.environ.get("VISUALIZATION_SERVER_IMAGE", "ghcr.io/kubeflow/kfp-visualization-server")
settings["frontend_image"] = \
frontend_image or \
os.environ.get("FRONTEND_IMAGE", "ghcr.io/kubeflow/kfp-frontend")
# Look for specific tags for each image first, falling back to
# previously used KFP_VERSION environment variable for backwards
# compatibility
settings["visualization_server_tag"] = \
visualization_server_tag or \
os.environ.get("VISUALIZATION_SERVER_TAG") or \
os.environ["KFP_VERSION"]
settings["frontend_tag"] = \
frontend_tag or \
os.environ.get("FRONTEND_TAG") or \
os.environ["KFP_VERSION"]
settings["disable_istio_sidecar"] = \
disable_istio_sidecar if disable_istio_sidecar is not None \
else os.environ.get("DISABLE_ISTIO_SIDECAR") == "true"
return settings
def server_factory(visualization_server_image,
visualization_server_tag, frontend_image, frontend_tag,
disable_istio_sidecar, url="", controller_port=8080):
"""
Returns an HTTPServer populated with Handler with customized settings
"""
class Controller(BaseHTTPRequestHandler):
def sync(self, parent, attachments):
# parent is a namespace
namespace = parent.get("metadata", {}).get("name")
pipeline_enabled = parent.get("metadata", {}).get(
"labels", {}).get("pipelines.kubeflow.org/enabled")
if pipeline_enabled != "true":
return {"status": {}, "attachments": []}
# Compute status based on observed state.
desired_status = {
"kubeflow-pipelines-ready":
len(attachments["Secret.v1"]) == 1 and
len(attachments["ConfigMap.v1"]) == 3 and
len(attachments["Deployment.apps/v1"]) == 2 and
len(attachments["Service.v1"]) == 2 and
len(attachments["DestinationRule.networking.istio.io/v1alpha3"]) == 1 and
len(attachments["AuthorizationPolicy.security.istio.io/v1beta1"]) == 1 and
"True" or "False"
}
# Generate the desired attachment object(s).
desired_resources = [
{
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"name": "kfp-launcher",
"namespace": namespace,
},
"data": {
"defaultPipelineRoot": f"minio://{S3_BUCKET_NAME}/private-artifacts/{namespace}/v2/artifacts",
},
},
{
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"name": "metadata-grpc-configmap",
"namespace": namespace,
},
"data": {
"METADATA_GRPC_SERVICE_HOST":
"metadata-grpc-service.kubeflow",
"METADATA_GRPC_SERVICE_PORT": "8080",
},
},
{
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"name": "artifact-repositories",
"namespace": namespace,
"annotations": {
"workflows.argoproj.io/default-artifact-repository": "default-namespaced"
}
},
"data": {
"default-namespaced": json.dumps({
"archiveLogs": True,
"s3": {
"endpoint": "minio-service.kubeflow:9000",
"bucket": S3_BUCKET_NAME,
"keyFormat": f"private-artifacts/{namespace}/{{{{workflow.name}}}}/{{{{workflow.creationTimestamp.Y}}}}/{{{{workflow.creationTimestamp.m}}}}/{{{{workflow.creationTimestamp.d}}}}/{{{{pod.name}}}}",
"insecure": True,
"accessKeySecret": {
"name": "mlpipeline-minio-artifact",
"key": "accesskey",
},
"secretKeySecret": {
"name": "mlpipeline-minio-artifact",
"key": "secretkey",
}
}
})
}
},
]
print('Received request:\n', json.dumps(parent, sort_keys=True))
print('Desired resources except secrets:\n', json.dumps(desired_resources, sort_keys=True))
# Moved after the print argument because this is sensitive data.
# Check if secret is already there when the controller made the request. If yes, then
# use it. Else create a new credentials on seaweedfs for the namespace.
if s3_secret := attachments["Secret.v1"].get(f"{namespace}/mlpipeline-minio-artifact"):
desired_resources.append(s3_secret)
print('Using existing secret')
else:
print('Creating new access key.')
s3_access_key = iam.create_access_key(UserName=namespace)
# Use the AWS IAM API of seaweedfs to manage access policies to bucket.
# This policy ensures that a user can only access artifacts from his own profile.
iam.put_user_policy(
UserName=namespace,
PolicyName=f"KubeflowProject{namespace}",
PolicyDocument=json.dumps(
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:Put*",
"s3:Get*",
"s3:List*"
],
"Resource": [
f"arn:aws:s3:::{S3_BUCKET_NAME}/artifacts/*",
f"arn:aws:s3:::{S3_BUCKET_NAME}/private-artifacts/{namespace}/*",
f"arn:aws:s3:::{S3_BUCKET_NAME}/private/{namespace}/*",
f"arn:aws:s3:::{S3_BUCKET_NAME}/shared/*",
]
}]
})
)
desired_resources.insert(
0,
{
"apiVersion": "v1",
"kind": "Secret",
"metadata": {
"name": "mlpipeline-minio-artifact",
"namespace": namespace,
},
"data": {
"accesskey": base64.b64encode(s3_access_key["AccessKey"]["AccessKeyId"].encode('utf-8')).decode("utf-8"),
"secretkey": base64.b64encode(s3_access_key["AccessKey"]["SecretAccessKey"].encode('utf-8')).decode("utf-8"),
},
})
return {"status": desired_status, "attachments": desired_resources}
def do_POST(self):
# Serve the sync() function as a JSON webhook.
observed = json.loads(
self.rfile.read(int(self.headers.get("content-length"))))
desired = self.sync(observed["object"], observed["attachments"])
self.send_response(200)
self.send_header("Content-type", "application/json")
self.end_headers()
self.wfile.write(bytes(json.dumps(desired), 'utf-8'))
return HTTPServer((url, int(controller_port)), Controller)
if __name__ == "__main__":
main()
@@ -0,0 +1,11 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubeflow
resources:
- seaweedfs-deployment.yaml
- seaweedfs-pvc.yaml
- seaweedfs-networkpolicy.yaml
- seaweedfs-create-admin-user-job.yaml
- seaweedfs-service.yaml
- seaweedfs-service-account.yaml
@@ -0,0 +1,67 @@
kind: Job
apiVersion: batch/v1
metadata:
name: init-seaweedfs
spec:
template:
metadata:
name: init-seaweedfs
spec:
restartPolicy: OnFailure
containers:
- name: init-seaweedfs
image: 'chrislusf/seaweedfs:3.85'
env:
- name: WEED_CLUSTER_DEFAULT
value: "sw"
- name: WEED_CLUSTER_SW_MASTER
value: "seaweedfs.kubeflow:9333"
envFrom:
- secretRef:
name: mlpipeline-minio-artifact
command:
- "/bin/sh"
- "-ec"
- |
wait_for_service() {
local url=$1
local max_attempts=60 # 5 minutes total (5s * 60)
local attempt=1
echo "Waiting for service at $url..."
while [ $attempt -le $max_attempts ]; do
if wget -q --spider "$url" >/dev/null 2>&1; then
echo "Service at $url is up!"
return 0
fi
echo "Attempt $attempt: Service not ready yet, retrying in 5s..."
sleep 5
attempt=$((attempt + 1))
done
echo "Service at $url failed to become ready within 5 minutes"
exit 1
}
wait_for_service "http://minio-service.kubeflow:9000/status"
exec /bin/echo "s3.bucket.create --name mlpipeline" | /usr/bin/weed shell
exec /bin/echo \
"s3.configure -user kubeflow-admin \
-access_key $accesskey \
-secret_key $secretkey \
-actions Admin \
-apply" |\
/usr/bin/weed shell
securityContext: # Using restricted profile
allowPrivilegeEscalation: false
privileged: false
runAsNonRoot: true
# image defaults to root user
runAsUser: 1001
runAsGroup: 1001
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
serviceAccountName: seaweedfs
@@ -0,0 +1,72 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: seaweedfs
namespace: kubeflow
labels:
app: seaweedfs
spec:
selector:
matchLabels:
app: seaweedfs
strategy:
type: Recreate
# Single container setup not scalable
replicas: 1
template:
metadata:
labels:
app: seaweedfs
application-crd-id: kubeflow-pipelines
spec:
containers:
- name: seaweedfs
image: 'chrislusf/seaweedfs:3.85'
args:
- 'server'
- '-dir=/data'
- '-s3'
- '-iam'
ports:
- containerPort: 8333
- containerPort: 8111
- containerPort: 9333
- containerPort: 19333
- containerPort: 8888
readinessProbe:
httpGet:
path: /status
port: 8333
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 15
successThreshold: 1
failureThreshold: 100
timeoutSeconds: 10
securityContext: # Using restricted profile
allowPrivilegeEscalation: false
privileged: false
runAsNonRoot: true
# image defaults to root user
runAsUser: 1001
runAsGroup: 1001
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
volumeMounts:
- mountPath: /data
name: data
resources:
# Benchmark this, just taken from minio
requests:
cpu: 20m
memory: 100Mi
volumes:
- name: data
persistentVolumeClaim:
claimName: seaweedfs-pvc
serviceAccountName: seaweedfs
@@ -0,0 +1,43 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-same-namespace
namespace: kubeflow
spec:
podSelector: {}
ingress:
- from:
- podSelector: {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: seaweedfs
spec:
ingress:
- from:
- namespaceSelector:
matchExpressions:
- key: app.kubernetes.io/part-of
operator: In
values:
- kubeflow-profile
ports:
- port: 8333
- from:
- namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- istio-system
podSelector:
matchExpressions:
- key: app
operator: In
values:
- seaweedfs
policyTypes:
- Ingress
@@ -0,0 +1,11 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: seaweedfs-pvc
namespace: kubeflow
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
@@ -0,0 +1,4 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: seaweedfs
@@ -0,0 +1,33 @@
# Separate service for new ports of seaweedfs. If we add them to the existing minio-service there will be a problem
# with the mlpipeline api server because it relies on MINIO_SERVICE_SERVICE_PORT pointing to the S3 port.
# But with multiple ports on a service that is not really reliable. So we use the existing minio-service for
# backwards-compatibility, but everything new, seaweedfs related is here.
apiVersion: v1
kind: Service
metadata:
name: seaweedfs
namespace: kubeflow
spec:
ports:
- name: http-iam
port: 8111
protocol: TCP
targetPort: 8111
- name: http-master
port: 9333
protocol: TCP
targetPort: 9333
- name: grpc-master
port: 19333
protocol: TCP
targetPort: 19333
- name: grpc-filer
port: 18888
protocol: TCP
targetPort: 18888
- name: http-filer
port: 8888
protocol: TCP
targetPort: 8888
selector:
app: seaweedfs
@@ -0,0 +1,32 @@
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: seaweedfs-service
spec:
action: ALLOW
selector:
matchLabels:
app: seaweedfs
rules:
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/ml-pipeline
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
# Allow traffic to s3 endpoint from User Pipeline Pods, which don't have a sidecar.
# Also needed for traffic from seaweedfs init pod. Seaweedfs gives the client an ip to connect to. This can not be
# handled well by istio (AuthPolicy). Instead, access to the sensitive ports will be limited by the NetworkPolicy.
- {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: DestinationRule
metadata:
name: ml-pipeline-seaweedfs
spec:
host: seaweedfs.kubeflow.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubeflow
resources:
- ../base/
- istio-authorization-policy.yaml
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
set -xe
kubectl create ns kubeflow || echo "namespace kubeflow already exists"
kubectl get -n kubeflow svc minio-service -o=jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' > svc-minio-service-backup.json
kustomize build istio/ | kubectl apply --server-side -f -
kubectl -n kubeflow wait --for=condition=available --timeout=600s deploy/seaweedfs
kubectl -n kubeflow exec deployments/seaweedfs -c seaweedfs -- sh -c "echo \"s3.configure -user minio -access_key minio -secret_key minio123 -actions Read,Write,List -apply\" | /usr/bin/weed shell"
kubectl -n kubeflow port-forward svc/minio-service 8333:9000
echo "S3 endpoint available on localhost:8333" &
function trap_handler {
kubectl -n kubeflow logs -l app=seaweedfs --tail=100
kustomize build istio/ | kubectl delete -f -
kubectl apply -f svc-minio-service-backup.json
}
trap trap_handler EXIT
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
- name: namespace-labels-data
namespace: kubeflow
behavior: merge
files:
- namespace-labels.yaml
@@ -0,0 +1,23 @@
# This file is a duplicate of apps/profiles/upstream/base/namespace-labels.yaml
# as using only the required label to merge it with the existing config map of profiles
# deployment to enable PSS for profile namespaces, leads to creation of a new config map
# with just the PSS label and replaces the pre-exisiting labels in the deployed config map.
# Below is a list of labels to be set by default.
#
# To add a namespace label, use `key: 'value'`, for example:
# istio.io/rev: 'asm-191-1'
#
# To remove a namespace label, use `key: ''`. For example:
# istio-injection: ''
#
# Profile controller will not replace a namespace label if its key already
# exists. If you want to override the value of a previously applied label, you
# need to:
# 1. Remove the label by using `key: ''` and deploy.
# 2. Add the label by using `key: 'value'` and deploy.
#
katib.kubeflow.org/metrics-collector-injection: "enabled"
serving.kubeflow.org/inferenceservice: "enabled"
pipelines.kubeflow.org/enabled: "true"
app.kubernetes.io/part-of: "kubeflow-profile"
pod-security.kubernetes.io/enforce: "baseline"
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
- name: namespace-labels-data
namespace: kubeflow
behavior: merge
files:
- namespace-labels.yaml
@@ -0,0 +1,23 @@
# This file is a duplicate of apps/profiles/upstream/base/namespace-labels.yaml
# as using only the required label to merge it with the existing config map of profiles
# deployment to enable PSS for profile namespaces, leads to creation of a new config map
# with just the PSS label and replaces the pre-exisiting labels in the deployed config map.
# Below is a list of labels to be set by default.
#
# To add a namespace label, use `key: 'value'`, for example:
# istio.io/rev: 'asm-191-1'
#
# To remove a namespace label, use `key: ''`. For example:
# istio-injection: ''
#
# Profile controller will not replace a namespace label if its key already
# exists. If you want to override the value of a previously applied label, you
# need to:
# 1. Remove the label by using `key: ''` and deploy.
# 2. Add the label by using `key: 'value'` and deploy.
#
katib.kubeflow.org/metrics-collector-injection: "enabled"
serving.kubeflow.org/inferenceservice: "enabled"
pipelines.kubeflow.org/enabled: "true"
app.kubernetes.io/part-of: "kubeflow-profile"
pod-security.kubernetes.io/enforce: "restricted"
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/kubeflow-labels.yaml
- path: patches/istio-system-labels.yaml
- path: patches/cert-manager-labels.yaml
- path: patches/auth-labels.yaml
- path: patches/oauth2-proxy-labels.yaml
- path: patches/knative-serving-labels.yaml
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: auth
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: knative-serving
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: kubeflow
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
labels:
pod-security.kubernetes.io/enforce: baseline
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- path: patches/kubeflow-labels.yaml
- path: patches/istio-system-labels.yaml
- path: patches/cert-manager-labels.yaml
- path: patches/auth-labels.yaml
- path: patches/oauth2-proxy-labels.yaml
- path: patches/knative-serving-labels.yaml
@@ -0,0 +1,7 @@
apiVersion: v1
kind: Namespace
metadata:
name: auth
labels:
pod-security.kubernetes.io/enforce: restricted
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
labels:
pod-security.kubernetes.io/enforce: restricted
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
labels:
pod-security.kubernetes.io/enforce: restricted
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: knative-serving
labels:
pod-security.kubernetes.io/enforce: restricted
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: kubeflow
labels:
pod-security.kubernetes.io/enforce: restricted
@@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
labels:
pod-security.kubernetes.io/enforce: restricted