update kubeflow dip-catalog

2025-01-13 02:31:27 +00:00
parent 1dc1181a03
commit 5451f16d72
1959 changed files with 602337 additions and 0 deletions
@@ -0,0 +1,77 @@
+# Sample installation
+
+1. Prepare a cluster and setup kubectl context
+Do whatever you want to customize your cluster. You can use existing cluster
+or create a new one.
+- **ML Usage** GPU normally is required for deep learning task.
+You may consider create **zero-sized GPU node-pool with autoscaling**.
+Please reference [GPU Tutorial](/samples/tutorials/gpu/).
+- **Security** You may consider use **Workload Identity** in GCP cluster.
+
+Here for simplicity, we create a small cluster with **--scopes=cloud-platform**
+which grants all the GCP permissions to the cluster.
+
+```
+gcloud container clusters create mycluster \
+  --zone us-central1-a \
+  --machine-type n1-standard-2 \
+  --scopes cloud-platform \
+  --enable-autoscaling \
+  --min-nodes 1 \
+  --max-nodes 5 \
+  --num-nodes 3
+```
+
+2. Prepare CloudSQL
+
+Create CloudSQL instance. [Console](https://console.cloud.google.com/sql/instances).
+
+Here is a sample for demo.
+
+```
+gcloud beta sql instances create mycloudsqlname \
+  --database-version=MYSQL_5_7 \
+  --tier=db-n1-standard-1 \
+  --region=us-central1 \
+  --root-password=password123
+```
+
+You may use **Private IP** to well protect your CloudSQL.
+If you use **Private IP**, please go to [VPC network peering](https://console.cloud.google.com/networking/peering/list)
+to double check whether the "cloudsql-mysql-googleais-com" is created and the "Exchange custom routes" is enabled. You
+are expected to see "Peer VPC network is connected".
+
+3. Prepare GCS Bucket
+
+Create Cloud Storage bucket. [Console](https://console.cloud.google.com/storage).
+
+```
+gsutil mb -p myProjectId gs://myBucketName/
+```
+
+4. Customize your values
+- Edit **params.env**, **params-db-secret.env** and **cluster-scoped-resources/params.env**
+- Edit kustomization.yaml to set your namespace, e.x. "kubeflow"
+
+5. (Optional.) If the cluster is on Workload Identity, please run **[gcp-workload-identity-setup.sh](../gcp-workload-identity-setup.sh)**
+  The script prints usage documentation when calling without argument. Note, you should
+  call it with `USE_GCP_MANAGED_STORAGE=true` env var.
+
+  - make sure the Google Service Account (GSA) can access the CloudSQL instance and GCS bucket
+  - if your workload calls other GCP APIs, make sure the GSA can access them
+
+6. Install
+
+```
+kubectl apply -k sample/cluster-scoped-resources/
+
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+
+kubectl apply -k sample/
+# If upper one action got failed, e.x. you used wrong value, try delete, fix and apply again
+# kubectl delete -k sample/
+
+kubectl wait applications/mypipeline -n kubeflow --for condition=Ready --timeout=1800s
+```
+
+Now you can find the installation in [Console](http://console.cloud.google.com/ai-platform/pipelines)
@@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+# !!! If you want to customize the namespace,
+# please also update sample/kustomization.yaml's namespace field to the same value
+namespace: kubeflow
+
+resources:
+  # Or github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.0.0
+  - ../../cluster-scoped-resources
@@ -0,0 +1,38 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  # Or github.com/kubeflow/pipelines/manifests/kustomize/env/gcp?ref=1.0.0
+  - ../env/gcp
+  # Kubeflow Pipelines servers are capable of collecting Prometheus metrics.
+  # If you want to monitor your Kubeflow Pipelines servers with those metrics, you'll need a Prometheus server in your Kubeflow Pipelines cluster.
+  # If you don't already have a Prometheus server up, you can uncomment the following configuration files for Prometheus.
+  # If you have your own Prometheus server up already or you don't want a Prometheus server for monitoring, you can comment the following line out.
+  # - ../third_party/prometheus
+  # - ../third_party/grafana
+
+# Identifier for application manager to apply ownerReference.
+# The ownerReference ensures the resources get garbage collected
+# when application is deleted.
+commonLabels:
+  application-crd-id: kubeflow-pipelines
+
+# Used by Kustomize
+configMapGenerator:
+  - name: pipeline-install-config
+    env: params.env
+    behavior: merge
+
+secretGenerator:
+  - name: mysql-secret
+    env: params-db-secret.env
+    behavior: merge
+
+# !!! If you want to customize the namespace,
+# please also update sample/cluster-scoped-resources/kustomization.yaml's namespace field to the same value
+namespace: kubeflow
+#### Customization ###
+# 1. Change values in params.env file
+# 2. Change values in params-db-secret.env file for CloudSQL username and password
+# 3. kubectl apply -k ./
+####
@@ -0,0 +1,2 @@
+username=root
+password=
@@ -0,0 +1,4 @@
+appName=mypipeline
+bucketName=mybucketname
+gcsProjectId=myprojectid
+gcsCloudSqlInstanceName=myprojectid:myregion:myinstance