update kubeflow dip-catalog
This commit is contained in:
ChanghoWoo
@@ -0,0 +1,12 @@
|
||||
KUBERAY_RELEASE_VERSION ?= 1.1.1
|
||||
KUBERAY_HELM_CHART_REPO ?= https://ray-project.github.io/kuberay-helm/
|
||||
|
||||
.PHONY: kuberay-operator/base
|
||||
kuberay-operator/base:
|
||||
mkdir -p kuberay-operator/base
|
||||
cd kuberay-operator/base && helm template --include-crds kuberay-operator kuberay-operator --version ${KUBERAY_RELEASE_VERSION} --repo ${KUBERAY_HELM_CHART_REPO} > resources.yaml
|
||||
|
||||
.PHONY: test
|
||||
test:
|
||||
./test.sh
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
approvers:
|
||||
- juliusvonkohout
|
||||
reviewers:
|
||||
- juliusvonkohout
|
||||
- kimwnasptd
|
||||
@@ -0,0 +1,133 @@
|
||||
TODO
|
||||
- The ray dashboard, worker and head must only be available from inside your kubeflow user namespace
|
||||
- Reenable the istio sidecar for the ray head and worker in the user namespace and provide the corresponding Istio Authorizationpolicies. We can keep the istio sidecar for the deployment kuberay-operator in the namespace kubeflow, since it does NOT use a webhook, but something else to reconcile rayclusters. This means we also do not need a networkpolicy for the ray operator.
|
||||
|
||||
|
||||
> Credit: This manifest refers a lot to the engineering blog ["Building a Machine Learning Platform with Kubeflow and Ray on Google Kubernetes Engine"](https://cloud.google.com/blog/products/ai-machine-learning/build-a-ml-platform-with-kubeflow-and-ray-on-gke) from Google Cloud.
|
||||
|
||||
# Ray
|
||||
[Ray](https://github.com/ray-project/ray) is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for simplifying ML compute.
|
||||
|
||||
<figure>
|
||||
<img
|
||||
src="assets/map-of-ray.png"
|
||||
alt="Ray">
|
||||
<figcaption>Stack of Ray libraries - unified toolkit for ML workloads. (ref: https://docs.ray.io/en/latest/ray-overview/index.html)</figcaption>
|
||||
</figure>
|
||||
|
||||
# KubeRay
|
||||
[KubeRay](https://github.com/ray-project/kuberay) is an open-source Kubernetes operator for Ray. It provides several CRDs to simplify managing Ray clusters on Kubernetes. We will integrate Kubeflow and KubeRay in this document.
|
||||
|
||||
# Requirements
|
||||
* Dependencies
|
||||
* `kustomize`: v5.2.1+ (Kubeflow manifest is sensitive to `kustomize` version.)
|
||||
* `Kubernetes`: v1.29+
|
||||
|
||||
* Computing resources:
|
||||
* 16GB RAM
|
||||
* 8 CPUs
|
||||
|
||||
# Example
|
||||
<figure>
|
||||
<img
|
||||
src="assets/architecture.svg"
|
||||
alt="ray/kubeflow integration">
|
||||
<figcaption>Note: (1) Kubeflow Central Dashboard will be renamed to workbench in the future. (2) Kubeflow Pipeline (KFP) is an important component of Kubeflow, but it is not included in this example.</figcaption>
|
||||
</figure>
|
||||
|
||||
## Step 1: Install Kubeflow v1.7-branch
|
||||
* This example installs Kubeflow with the [v1.9-branch](https://github.com/kubeflow/manifests/tree/v1.9-branch).
|
||||
|
||||
* Install all Kubeflow official components and all common services using [one command](https://github.com/kubeflow/manifests/tree/v1.7-branch#install-with-a-single-command).
|
||||
* If you do not want to install all components, you can comment out **KNative**, **Katib**, **Tensorboards Controller**, **Tensorboard Web App**, **Training Operator**, and **KServe** from [example/kustomization.yaml](https://github.com/kubeflow/manifests/blob/v1.7-branch/example/kustomization.yaml).
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
We never ever break Kubernetes standards and do not use the "default" namespace, but a proper one, in our case "kubeflow" for the ray operator.
|
||||
|
||||
```sh
|
||||
# Install a KubeRay operator and custom resource definitions.
|
||||
kustomize build kuberay-operator/overlays/kubeflow | kubectl apply --server-side -f -
|
||||
|
||||
# Check KubeRay operator
|
||||
kubectl get pod -l app.kubernetes.io/component=kuberay-operator -n kubeflow
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kuberay-operator-5b8cd69758-rkpvh 1/1 Running 0 6m23s
|
||||
```
|
||||
|
||||
## Step 3: Install RayCluster
|
||||
```sh
|
||||
# Create a RayCluster CR, and the KubeRay operator will reconcile a Ray cluster
|
||||
# with 1 head Pod and 1 worker Pod.
|
||||
# $MY_KUBEFLOW_USER_NAMESPACE is a proper Kubeflow user namespace with istio sidecar injection and never ever the wrong "default"
|
||||
export MY_KUBEFLOW_USER_NAMESPACE=development
|
||||
kubectl apply -f raycluster_example.yaml -n $MY_KUBEFLOW_USER_NAMESPACE
|
||||
|
||||
# Check RayCluster
|
||||
kubectl get pod -l ray.io/cluster=kubeflow-raycluster -n $MY_KUBEFLOW_USER_NAMESPACE
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kubeflow-raycluster-head-p6dpk 1/1 Running 0 70s
|
||||
# kubeflow-raycluster-worker-small-group-l7j6c 1/1 Running 0 70s
|
||||
```
|
||||
* `raycluster_example.yaml` uses `rayproject/ray:2.23.0-py311-cpu` as its OCI image. Ray is very sensitive to the Python versions and Ray versions between the server (RayCluster) and client (JupyterLab) sides. This image uses:
|
||||
* Python 3.11
|
||||
* Ray 2.23.0
|
||||
|
||||
## Step 4: Forward the port of Istio's Ingress-Gateway
|
||||
* Follow the [instructions](https://github.com/kubeflow/manifests/tree/v1.7-branch#port-forward) to forward the port of Istio's Ingress-Gateway and log in to Kubeflow Central Dashboard.
|
||||
|
||||
## Step 5: Create a JupyterLab via Kubeflow Central Dashboard
|
||||
* Click "Notebooks" icon in the left panel.
|
||||
* Click "New Notebook"
|
||||
* Select `kubeflownotebookswg/jupyter-scipy:v1.9.0` as OCI image (or any other with the same python version)
|
||||
* Click "Launch"
|
||||
* Click "CONNECT" to connect into the JupyterLab instance.
|
||||
|
||||
## Step 6: Use Ray client in the JupyterLab to connect to the RayCluster
|
||||
* As I mentioned in Step 3, Ray is very sensitive to the Python versions and Ray versions between the server (RayCluster) and client (JupyterLab) sides.
|
||||
```sh
|
||||
# Check Python version. The version's MAJOR and MINOR should match with RayCluster (i.e. Python 3.11.9)
|
||||
python --version
|
||||
# Python 3.11.9
|
||||
pip install -U ray[default]==2.23.0
|
||||
```
|
||||
* Connect to RayCluster via Ray client.
|
||||
```python
|
||||
# Open a new .ipynb page.
|
||||
|
||||
import ray
|
||||
# For other namespaces use ray://${RAYCLUSTER_HEAD_SVC}.${NAMESPACE}.svc.cluster.local:${RAY_CLIENT_PORT}
|
||||
# But we use of course our per namespace ray cluster to have multi-tenancy and
|
||||
# We never ever use "default" as namespace since this would violate Kubernetes standards
|
||||
ray.init(address="ray://kubeflow-raycluster-head-svc:10001")
|
||||
print(ray.cluster_resources())
|
||||
# {'node:10.244.0.41': 1.0, 'memory': 3000000000.0, 'node:10.244.0.40': 1.0, 'object_store_memory': 805386239.0, 'CPU': 2.0}
|
||||
|
||||
# Try Ray task
|
||||
@ray.remote
|
||||
def f(x):
|
||||
return x * x
|
||||
|
||||
futures = [f.remote(i) for i in range(4)]
|
||||
print(ray.get(futures)) # [0, 1, 4, 9]
|
||||
|
||||
# Try Ray actor
|
||||
@ray.remote
|
||||
class Counter(object):
|
||||
def __init__(self):
|
||||
self.n = 0
|
||||
|
||||
def increment(self):
|
||||
self.n += 1
|
||||
|
||||
def read(self):
|
||||
return self.n
|
||||
|
||||
counters = [Counter.remote() for i in range(4)]
|
||||
[c.increment.remote() for c in counters]
|
||||
futures = [c.read.remote() for c in counters]
|
||||
print(ray.get(futures)) # [1, 1, 1, 1]
|
||||
```
|
||||
|
||||
# Upgrading
|
||||
See [UPGRADE.md](UPGRADE.md) for more details.
|
||||
@@ -0,0 +1,6 @@
|
||||
# Upgrading
|
||||
```sh
|
||||
# Step 1: Update KUBERAY_RELEASE_VERSION in Makefile
|
||||
# Step 2: Create new KubeRay operator manifest
|
||||
make kuberay-operator/base
|
||||
```
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 217 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
@@ -0,0 +1,52 @@
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: kubeflow-kuberay-admin
|
||||
labels:
|
||||
app: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay-operator
|
||||
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"
|
||||
rules: []
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: kubeflow-kuberay-edit
|
||||
labels:
|
||||
app: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay-operator
|
||||
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
|
||||
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ray.io
|
||||
resources:
|
||||
- "*"
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- create
|
||||
- update
|
||||
- patch
|
||||
- delete
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: kubeflow-kuberay-view
|
||||
labels:
|
||||
app: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay-operator
|
||||
rbac.authorization.kubeflow.org/aggregate-to-kubeflow-view: "true"
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ray.io
|
||||
resources:
|
||||
- "*"
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
---
|
||||
@@ -0,0 +1,20 @@
|
||||
patches:
|
||||
# Add securityContext to KubeRay operator Pod.
|
||||
- target:
|
||||
kind: Deployment
|
||||
labelSelector: "app.kubernetes.io/name=kuberay-operator"
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/containers/0/securityContext
|
||||
value:
|
||||
runAsUser: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
namespace: kubeflow
|
||||
resources:
|
||||
- resources.yaml
|
||||
- aggregated-roles.yaml
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
namespace: kubeflow
|
||||
resources:
|
||||
- ../../base
|
||||
@@ -0,0 +1,4 @@
|
||||
namespace: kubeflow
|
||||
resources:
|
||||
- ../../base
|
||||
- namespace.yaml
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: kubeflow
|
||||
@@ -0,0 +1,128 @@
|
||||
apiVersion: ray.io/v1alpha1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: kubeflow-raycluster
|
||||
spec:
|
||||
rayVersion: '2.23.0'
|
||||
# Ray head pod configuration
|
||||
headGroupSpec:
|
||||
# Kubernetes Service Type.
|
||||
serviceType: ClusterIP
|
||||
# The following params are used to complete the ray start: ray start --head --block --dashboard-host: '0.0.0.0' ...
|
||||
rayStartParams:
|
||||
dashboard-host: '0.0.0.0'
|
||||
block: 'true'
|
||||
# pod template
|
||||
template:
|
||||
metadata:
|
||||
# Custom labels. NOTE: To avoid conflicts with KubeRay operator, do not define custom labels start with `raycluster`.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
# The ray head must not have an Istio sidecar
|
||||
# TODO add an authorizationpolicy in the future for the ray head
|
||||
labels:
|
||||
sidecar.istio.io/inject: "false"
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.23.0-py311-cpu
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: gcs
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
volumeMounts:
|
||||
- mountPath: /tmp/ray
|
||||
name: ray-logs
|
||||
# The resource requests and limits in this config are too small for production!
|
||||
# It is better to use a few large Ray pod than many small ones.
|
||||
# For production, it is ideal to size each Ray pod to take up the
|
||||
# entire Kubernetes node on which it is scheduled.
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2G"
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "2G"
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumes:
|
||||
- name: ray-logs
|
||||
emptyDir: {}
|
||||
workerGroupSpecs:
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
# logical group name, for this called small-group, also can be functional
|
||||
groupName: small-group
|
||||
rayStartParams:
|
||||
block: 'true'
|
||||
#pod template
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
# Disable the sidecars for the ray wokers
|
||||
# TODO add an authorizationpolicy in the future for the ray worker
|
||||
sidecar.istio.io/inject: "false"
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.23.0-py311-cpu
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
# use volumeMounts.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumeMounts:
|
||||
- mountPath: /tmp/ray
|
||||
name: ray-logs
|
||||
# The resource requests and limits in this config are too small for production!
|
||||
# It is better to use a few large Ray pod than many small ones.
|
||||
# For production, it is ideal to size each Ray pod to take up the
|
||||
# entire Kubernetes node on which it is scheduled.
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
requests:
|
||||
cpu: "300m"
|
||||
memory: "1G"
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
# the env var $RAY_IP is set by the operator if missing, with the value of the head service name
|
||||
- name: init
|
||||
image: busybox:1.36
|
||||
# Change the cluster postfix if you don't have a default setting
|
||||
command: ['sh', '-c', "until nslookup $RAY_IP.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for K8s Service $RAY_IP; sleep 2; done"]
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
# use volumes
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumes:
|
||||
- name: ray-logs
|
||||
emptyDir: {}
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -euxo
|
||||
|
||||
NAMESPACE=kubeflow
|
||||
TIMEOUT=120 # timeout in seconds
|
||||
SLEEP_INTERVAL=30 # interval between checks in seconds
|
||||
RAY_VERSION=2.23.0
|
||||
|
||||
function trap_handler {
|
||||
kill $PID
|
||||
# Delete RayCluster
|
||||
kubectl -n $NAMESPACE delete -f raycluster_example.yaml
|
||||
|
||||
# Wait for all Ray Pods to be deleted.
|
||||
start_time=$(date +%s)
|
||||
while true; do
|
||||
pods=$(kubectl -n $NAMESPACE get pods -o json | jq '.items | length')
|
||||
if [ "$pods" -eq 1 ]; then
|
||||
break
|
||||
fi
|
||||
current_time=$(date +%s)
|
||||
elapsed_time=$((current_time - start_time))
|
||||
if [ "$elapsed_time" -ge "$TIMEOUT" ]; then
|
||||
echo "Timeout exceeded. Exiting loop."
|
||||
exit 1
|
||||
fi
|
||||
sleep $SLEEP_INTERVAL
|
||||
done
|
||||
|
||||
# Delete KubeRay operator
|
||||
kustomize build kuberay-operator/base | kubectl -n $NAMESPACE delete -f -
|
||||
}
|
||||
|
||||
trap trap_handler EXIT
|
||||
|
||||
# Install KubeRay operator
|
||||
kustomize build kuberay-operator/overlays/standalone | kubectl -n $NAMESPACE apply --server-side -f -
|
||||
|
||||
# Wait for the operator to be ready.
|
||||
kubectl -n $NAMESPACE wait --for=condition=available --timeout=600s deploy/kuberay-operator
|
||||
kubectl -n $NAMESPACE get pod -l app.kubernetes.io/component=kuberay-operator
|
||||
|
||||
# Create a RayCluster custom resource.
|
||||
kubectl -n $NAMESPACE apply -f raycluster_example.yaml
|
||||
|
||||
# Wait for the RayCluster to be ready.
|
||||
sleep 5
|
||||
kubectl -n $NAMESPACE wait --for=condition=ready pod -l ray.io/cluster=kubeflow-raycluster --timeout=900s
|
||||
kubectl -n $NAMESPACE logs -l ray.io/cluster=kubeflow-raycluster,ray.io/node-type=head
|
||||
|
||||
# Forward the port of Dashboard
|
||||
sleep 5
|
||||
kubectl -n $NAMESPACE port-forward --address 0.0.0.0 svc/kubeflow-raycluster-head-svc 8265:8265 &
|
||||
PID=$!
|
||||
echo "Forward the port 8265 of Ray head in the background process: $PID"
|
||||
|
||||
# Send a curl command to test basic Ray functionality.
|
||||
sleep 5
|
||||
output=$(curl -H "Content-Type: application/json" localhost:8265/api/version)
|
||||
echo "output: ${output}"
|
||||
|
||||
# output format: {"version": ..., "ray_version": RAY_VERSION, "ray_commit": ...}
|
||||
if echo "${output}" | grep -q $RAY_VERSION; then
|
||||
echo "Test succeeded!"
|
||||
else
|
||||
echo "Test failed!"
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user