diff --git a/charts/kafka-cluster/CUSTOM-README.md b/charts/kafka-cluster/CUSTOM-README.md new file mode 100644 index 0000000..76aa18f --- /dev/null +++ b/charts/kafka-cluster/CUSTOM-README.md @@ -0,0 +1,255 @@ +# Apache Kafka 배포 (Strimzi Operator 기반) + +## 1. 배포 방법 + +### 1) 배포시 주의 사항 + +- Apache Kafka 4.0.0은 KRaft 모드로만 동작하며 ZooKeeper가 필요하지 않습니다. +- Controller와 Broker 노드를 분리하여 배포합니다. +- OAuth 2.0 인증이 활성화되어 있으므로 Keycloak 설정이 필요합니다. +- Kafka Connect가 기본적으로 활성화되어 있습니다. +- 프로덕션 환경에서는 리소스 설정과 영속성 스토리지 설정을 반드시 확인해야 합니다. + +### 2) 배포 방법 + +```sh +git clone https://github.com/paasup/service-catalog.git +cd charts/kafka +helm upgrade kafka ./ -f custom-values.yaml --install -n kafka --create-namespace +``` + +## 2. custom-values.yaml 예시 + +다음은 배포 시 사용할 수 있는 custom-values.yaml 파일의 예시입니다: + +```yaml +kafka: + version: "4.0.0" + metadataVersion: "4.0" + timezone: "Asia/Seoul" + +controller: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: {} + +broker: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: {} + +config: + offsetsTopicReplicationFactor: 3 + transactionStateLogReplicationFactor: 3 + transactionStateLogMinIsr: 2 + defaultReplicationFactor: 3 + minInsyncReplicas: 2 + autoCreateTopicsEnable: false + numPartitions: 3 + deleteTopicEnable: true + +oauth: + enabled: true + keycloakUrl: "https://keycloak.example.com" + keycloakRealm: "your-realm" + keycloakClientId: "kafka-client" + +kafkaConnect: + enabled: true + image: paasup/kafka-connect:0.2 + replicas: 1 + +entityOperator: + topicOperator: {} + userOperator: {} + +connectTopics: + partitions: 1 + replicas: 3 + config: + cleanupPolicy: compact + retentionMs: 604800000 + segmentBytes: 1073741824 + +logging: + kafka: + rootLoggerLevel: INFO + connect: + rootLoggerLevel: INFO +``` + +## 3. custom-values.yaml 설정 설명 + +### 1) Kafka 기본 설정 + +| Name | 설명 | 기본값 | +| ----------------------- | --------------------- | ------------ | +| `kafka.version` | Kafka 버전 | `4.0.0` | +| `kafka.metadataVersion` | Kafka 메타데이터 버전 | `4.0` | +| `kafka.timezone` | Kafka 클러스터 타임존 | `Asia/Seoul` | + +### 2) Controller 노드 설정 + +| Name | 설명 | 기본값 | +| -------------------------- | ----------------------------------- | ---------- | +| `controller.replicas` | Controller 노드 수 | `3` | +| `controller.storage.size` | Controller 영속성 볼륨 크기 | `50Gi` | +| `controller.storage.class` | Controller 영속성 볼륨 StorageClass | `longhorn` | +| `controller.resources` | Controller 리소스 제한 설정 | `{}` | + +### 3) Broker 노드 설정 + +| Name | 설명 | 기본값 | +| ---------------------- | ------------------------------- | ---------- | +| `broker.replicas` | Broker 노드 수 | `3` | +| `broker.storage.size` | Broker 영속성 볼륨 크기 | `50Gi` | +| `broker.storage.class` | Broker 영속성 볼륨 StorageClass | `longhorn` | +| `broker.resources` | Broker 리소스 제한 설정 | `{}` | + +### 4) Kafka 클러스터 설정 + +| Name | 설명 | 기본값 | +| --------------------------------------------- | ------------------------------------------ | ------- | +| `config.offsetsTopicReplicationFactor` | \_\_consumer_offsets 토픽의 복제 팩터 | `3` | +| `config.transactionStateLogReplicationFactor` | 트랜잭션 상태 로그의 복제 팩터 | `3` | +| `config.transactionStateLogMinIsr` | 트랜잭션 상태 로그의 최소 동기화 복제본 수 | `2` | +| `config.defaultReplicationFactor` | 새 토픽의 기본 복제 팩터 | `3` | +| `config.minInsyncReplicas` | 쓰기 작업에 필요한 최소 동기화 복제본 수 | `2` | +| `config.autoCreateTopicsEnable` | 토픽 자동 생성 활성화 여부 | `false` | +| `config.numPartitions` | 새 토픽의 기본 파티션 수 | `3` | +| `config.deleteTopicEnable` | 토픽 삭제 활성화 여부 | `true` | + +### 5) OAuth 2.0 인증 설정 + +| Name | 설명 | 기본값 | +| ------------------------ | -------------------------- | ------------------------------ | +| `oauth.enabled` | OAuth 2.0 인증 활성화 여부 | `true` | +| `oauth.keycloakUrl` | Keycloak 서버 URL | `https://keycloak.example.com` | +| `oauth.keycloakRealm` | Keycloak Realm 이름 | `your-realm` | +| `oauth.keycloakClientId` | Keycloak Client ID | `kafka-client` | + +### 6) Kafka Connect 설정 + +| Name | 설명 | 기본값 | +| ----------------------- | ------------------------- | -------------------------- | +| `kafkaConnect.enabled` | Kafka Connect 활성화 여부 | `true` | +| `kafkaConnect.image` | Kafka Connect 이미지 | `paasup/kafka-connect:0.2` | +| `kafkaConnect.replicas` | Kafka Connect 워커 수 | `1` | + +### 7) Entity Operator 설정 + +| Name | 설명 | 기본값 | +| ------------------------------ | ------------------- | ------ | +| `entityOperator.topicOperator` | Topic Operator 설정 | `{}` | +| `entityOperator.userOperator` | User Operator 설정 | `{}` | + +### 8) Kafka Connect 내부 토픽 설정 + +| Name | 설명 | 기본값 | +| ------------------------------------ | -------------------------------- | ------------ | +| `connectTopics.partitions` | Connect 내부 토픽의 파티션 수 | `1` | +| `connectTopics.replicas` | Connect 내부 토픽의 복제 팩터 | `3` | +| `connectTopics.config.cleanupPolicy` | 로그 정리 정책 (compact/delete) | `compact` | +| `connectTopics.config.retentionMs` | 로그 보관 기간 (밀리초, 7일) | `604800000` | +| `connectTopics.config.segmentBytes` | 로그 세그먼트 크기 (바이트, 1GB) | `1073741824` | + +### 9) 로깅 설정 + +| Name | 설명 | 기본값 | +| --------------------------------- | ---------------------------- | ------ | +| `logging.kafka.rootLoggerLevel` | Kafka 루트 로거 레벨 | `INFO` | +| `logging.connect.rootLoggerLevel` | Kafka Connect 루트 로거 레벨 | `INFO` | + +## 4. 배포 아키텍처 + +### 1) Controller와 Broker 분리 배포 + +- `controller.replicas: 3` - Controller 전용 노드 3개 +- `broker.replicas: 3` - Broker 전용 노드 3개 +- Controller와 Broker 역할을 분리하여 성능 최적화 +- 대규모 프로덕션 환경에 적합 + +### 2) 고가용성 설정 + +- 복제 팩터 3으로 데이터 손실 방지 +- 최소 동기화 복제본 2로 일관성 보장 +- 각 노드에 50Gi 영속성 스토리지 할당 + +## 5. 클라이언트 연결 + +### 1) OAuth 2.0 인증을 사용한 클라이언트 설정 예시 + +```properties +bootstrap.servers=kafka-kafka-bootstrap:9092 +security.protocol=SASL_PLAINTEXT +sasl.mechanism=OAUTHBEARER +sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ + oauth.client.id="kafka-client" \ + oauth.client.secret="" \ + oauth.token.endpoint.uri="https://keycloak.example.com/realms/your-realm/protocol/openid-connect/token"; +sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler +``` + +### 2) Keycloak 설정 확인 + +Keycloak에서 다음 설정을 확인하세요: + +- Realm: `your-realm` +- Client ID: `kafka-client` +- Client Secret 생성 및 확인 +- Service Account 활성화 + +## 6. Kafka Connect 사용 + +### 1) Connector 상태 확인 + +```sh +# Kafka Connect 로그 확인 +kubectl logs -n kafka -l strimzi.io/cluster=kafka-connect + +# Connector 목록 확인 +kubectl get kafkaconnectors -n kafka +``` + +## 7. 트러블슈팅 + +### 1) 일반적인 문제 + +- **Pod 시작 실패**: Longhorn 스토리지 클래스 확인 및 리소스 부족 여부 확인 +- **OAuth 인증 실패**: Keycloak URL, Realm, Client ID 설정 확인 +- **Kafka Connect 연결 실패**: Connect 내부 토픽 생성 및 권한 확인 +- **성능 문제**: 리소스 설정 및 파티션 수 조정 + +### 2) 로그 확인 + +```sh +# Kafka Controller 로그 확인 +kubectl logs -n kafka kafka-kafka-controller-0 + +# Kafka Broker 로그 확인 +kubectl logs -n kafka kafka-kafka-broker-0 + +# Kafka Connect 로그 확인 +kubectl logs -n kafka -l strimzi.io/cluster=kafka-connect +``` + +### 3) 토픽 관리 + +```sh +# 토픽 목록 확인 +kubectl get kafkatopics -n kafka + +# 토픽 상세 정보 확인 +kubectl describe kafkatopic test-topic -n kafka +``` + +### 4) 사용자 관리 + +```sh +# 사용자 목록 확인 +kubectl get kafkausers -n kafka +``` diff --git a/charts/kafka-cluster/Chart.yaml b/charts/kafka-cluster/Chart.yaml new file mode 100644 index 0000000..e66405a --- /dev/null +++ b/charts/kafka-cluster/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: kafka-cluster +description: A Helm chart for Kafka cluster with Strimzi operator +type: application +version: 0.1.0 +appVersion: "4.0.0" diff --git a/charts/kafka-cluster/README.md b/charts/kafka-cluster/README.md new file mode 100644 index 0000000..e67e2eb --- /dev/null +++ b/charts/kafka-cluster/README.md @@ -0,0 +1,162 @@ +# Kafka Cluster Helm Chart + +Strimzi 오퍼레이터를 사용하는 Kafka 클러스터 배포를 위한 Helm 차트입니다. + +## 사전 요구사항 + +- Kubernetes 1.19+ +- Helm 3.0+ +- Strimzi Kafka Operator가 클러스터에 설치되어 있어야 합니다 + +## 설치 방법 + +### 1. Strimzi Operator 설치 (아직 설치하지 않은 경우) + +```bash +kubectl create namespace kafka +kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka +``` + +### 2. Helm 차트 설치 + +```bash +# values.yaml 파일을 수정하여 환경에 맞게 설정 +helm install kafka-cluster . -n kafka \ + --set oauth.keycloakUrl="https://your-keycloak-url" \ + --set oauth.keycloakRealm="your-realm" \ + --set oauth.keycloakClientId="your-client-id" +``` + +또는 values 파일을 사용: + +```bash +helm install kafka-cluster . -n kafka -f custom-values.yaml +``` + +## 설정 + +주요 설정 값들: + +### Kafka 클러스터 설정 + +```yaml +kafka: + name: kafka-cluster + version: "4.0.0" + metadataVersion: "4.0" + timezone: "Asia/Seoul" +``` + +### Node Pool 설정 + +```yaml +controller: + replicas: 3 + storage: + size: 50Gi + class: longhorn + +broker: + replicas: 3 + storage: + size: 50Gi + class: longhorn +``` + +### OAuth/Keycloak 설정 + +```yaml +oauth: + enabled: true + keycloakUrl: "" # 필수: Keycloak URL + keycloakRealm: "" # 필수: Keycloak Realm + keycloakClientId: "" # 필수: Client ID +``` + +### Kafka Connect 설정 + +```yaml +kafkaConnect: + enabled: true + image: paasup/kafka-connect:0.2 + replicas: 1 +``` + +## 업그레이드 + +```bash +helm upgrade kafka-cluster . -n kafka -f custom-values.yaml +``` + +## 삭제 + +```bash +helm uninstall kafka-cluster -n kafka +``` + +## 필수 Secret 생성 + +배포 전에 다음 Secret들을 생성해야 합니다: + +### 1. Kafka OAuth Secret + +```bash +kubectl create secret generic kafka-cluster-oauth-secret \ + --from-literal=client-secret=YOUR_CLIENT_SECRET \ + -n kafka +``` + +### 2. Kafka Connect OAuth Secret + +```bash +kubectl create secret generic kafka-connect-oauth-secret \ + --from-literal=client-secret=YOUR_CONNECT_CLIENT_SECRET \ + -n kafka +``` + +### 3. Keycloak TLS Certificate + +```bash +kubectl create secret generic keycloak-tls \ + --from-file=ca.crt=path/to/ca.crt \ + -n kafka +``` + +### 4. Truststore Secret + +```bash +kubectl create secret generic truststore \ + --from-file=truststore.jks=path/to/truststore.jks \ + -n kafka +``` + +## 확인 + +배포 상태 확인: + +```bash +# Kafka 클러스터 상태 확인 +kubectl get kafka -n kafka + +# Pod 상태 확인 +kubectl get pods -n kafka + +# Kafka Connect 상태 확인 +kubectl get kafkaconnect -n kafka +``` + +## 문제 해결 + +로그 확인: + +```bash +# Kafka broker 로그 +kubectl logs -n kafka kafka-cluster-broker-0 + +# Kafka Connect 로그 +kubectl logs -n kafka -l strimzi.io/cluster=kafka-cluster,strimzi.io/kind=KafkaConnect +``` + +## 라이선스 + +이 차트는 Apache License 2.0 하에 배포됩니다. diff --git a/charts/kafka-cluster/custom-values.yaml b/charts/kafka-cluster/custom-values.yaml new file mode 100644 index 0000000..a6cdbf2 --- /dev/null +++ b/charts/kafka-cluster/custom-values.yaml @@ -0,0 +1,57 @@ +kafka: + version: "4.0.0" + metadataVersion: "4.0" + timezone: "Asia/Seoul" + +controller: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: {} + +broker: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: {} + +config: + offsetsTopicReplicationFactor: 3 + transactionStateLogReplicationFactor: 3 + transactionStateLogMinIsr: 2 + defaultReplicationFactor: 3 + minInsyncReplicas: 2 + autoCreateTopicsEnable: false + numPartitions: 3 + deleteTopicEnable: true + +oauth: + enabled: true + keycloakUrl: "https://keycloak.example.com" + keycloakRealm: "your-realm" + keycloakClientId: "kafka-client" + +kafkaConnect: + enabled: true + image: paasup/kafka-connect:0.2 + replicas: 1 + +entityOperator: + topicOperator: {} + userOperator: {} + +connectTopics: + partitions: 1 + replicas: 3 + config: + cleanupPolicy: compact + retentionMs: 604800000 + segmentBytes: 1073741824 + +logging: + kafka: + rootLoggerLevel: INFO + connect: + rootLoggerLevel: INFO diff --git a/charts/kafka-cluster/kafka-cluster.yaml b/charts/kafka-cluster/kafka-cluster.yaml new file mode 100644 index 0000000..b662394 --- /dev/null +++ b/charts/kafka-cluster/kafka-cluster.yaml @@ -0,0 +1,260 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: controller + namespace: "{{ .Namespace }}" + labels: + strimzi.io/cluster: "{{ .Name }}" +spec: + replicas: 3 + roles: + - controller + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: 50Gi + kraftMetadata: shared + class: longhorn + resources: + requests: + cpu: 500m + memory: 1Gi + limits: + cpu: 1 + memory: 2Gi +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: broker + namespace: "{{ .Namespace }}" + labels: + strimzi.io/cluster: "{{ .Name }}" +spec: + replicas: 3 + roles: + - broker + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: 50Gi + class: longhorn + kraftMetadata: shared + resources: + requests: + cpu: 500m + memory: 1Gi + limits: + cpu: 1 + memory: 2Gi +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: Kafka +metadata: + name: "{{ .Name }}" + namespace: "{{ .Namespace }}" + annotations: + strimzi.io/node-pools: enabled + strimzi.io/kraft: enabled +spec: + kafka: + version: 4.0.0 + metadataVersion: "4.0" + template: + kafkaContainer: + env: + - name: KAFKA_OPTS + value: "-Duser.timezone=Asia/Seoul" + listeners: + - name: tls + type: cluster-ip + port: 9093 + tls: false + authentication: + type: oauth + clientId: "{{.KEYCLOAK_CLIENT_ID}}" + clientSecret: + key: client-secret + secretName: kafka-cluster-oauth-secret + checkIssuer: true + checkAccessTokenType: true + accessTokenIsJwt: true + checkAudience: false + enableOauthBearer: true + validIssuerUri: "{{.KEYCLOAK_URL}}/realms/$KEYCLOAK_REALM" + jwksEndpointUri: "{{.KEYCLOAK_URL}}/realms/$KEYCLOAK_REALM/protocol/openid-connect/certs" + userNameClaim: preferred_username + customClaimCheck: "'{{.KEYCLOAK_CLIENT_ID}}' in @.realm_access.roles" + tlsTrustedCertificates: + - secretName: keycloak-tls + certificate: ca.crt + config: + offsets.topic.replication.factor: 3 + transaction.state.log.replication.factor: 3 + transaction.state.log.min.isr: 2 + default.replication.factor: 3 + min.insync.replicas: 2 + auto.create.topics.enable: false + num.partitions: 3 + delete.topic.enable: true + authorization: + type: simple + logging: + type: inline + loggers: + kafka.root.logger.level: INFO + entityOperator: + entityOperator: + topicOperator: + resources: + requests: + memory: 128Mi + cpu: 100m + limits: + memory: 256Mi + cpu: 200m + userOperator: + resources: + requests: + memory: 128Mi + cpu: 100m + limits: + memory: 256Mi + cpu: 200m + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaConnect +metadata: + name: "{{ .Name }}" + namespace: "{{ .Namespace }}" + labels: + strimzi.io/cluster: "{{ .Name }}" + annotations: + strimzi.io/use-connector-resources: "true" +spec: + image: paasup/kafka-connect:0.2 + replicas: 1 + bootstrapServers: "kafka-cluster-kafka-tls-bootstrap.{{ .Namespace }}.svc.cluster.local:9093" + config: + group.id: "{{ .Name }}-default-connect" + offset.storage.topic: "{{ .Name }}.connect-offsets" + config.storage.topic: "{{ .Name }}.connect-configs" + status.storage.topic: "{{ .Name }}.connect-status" + key.converter: org.apache.kafka.connect.json.JsonConverter + value.converter: org.apache.kafka.connect.json.JsonConverter + plugin.path: /opt/kafka/plugins + topic.creation.enable: "true" + authentication: + type: oauth + tokenEndpointUri: "{{.KEYCLOAK_URL}}/realms/$KEYCLOAK_REALM/protocol/openid-connect/token" + clientId: "{{.KEYCLOAK_CLIENT_ID}}-kafka-connect" + clientSecret: + secretName: kafka-connect-oauth-secret + key: client-secret + tlsTrustedCertificates: + - secretName: keycloak-tls + pattern: "ca.crt" + + template: + connectContainer: + volumeMounts: + - name: truststore-volume + mountPath: /mnt/truststore/truststore.jks + subPath: truststore.jks + readOnly: true + pod: + volumes: + - name: truststore-volume + secret: + secretName: truststore + + logging: + type: inline + loggers: + rootLogger.level: INFO + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaUser +metadata: + name: "service-account-{{.KEYCLOAK_CLIENT_ID}}-kafka-connect" + namespace: "{{ .Namespace }}" + labels: + strimzi.io/cluster: "{{ .Name }}" +spec: + authorization: + type: simple + acls: + - resource: + type: topic + name: "*" + patternType: literal + operations: + - Read + - Describe + - DescribeConfigs + - Write + - resource: + type: group + name: "*" + patternType: literal + operations: + - Read + - Write + - Describe + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: kafka-cluster + name: "{{ .Name }}-connect-offsets" + namespace: "{{ .Namespace }}" +spec: + partitions: 1 + replicas: 3 + topicName: "{{ .Name }}.connect-offsets" + config: + cleanup.policy: compact + retention.ms: 604800000 + segment.bytes: 1073741824 + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: kafka-cluster + name: "{{ .Name }}-connect-configs" + namespace: "{{ .Namespace }}" +spec: + partitions: 1 + replicas: 3 + topicName: "{{ .Name }}.connect-configs" + config: + cleanup.policy: compact + retention.ms: 604800000 + segment.bytes: 1073741824 + +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: kafka-cluster + name: "{{ .Name }}-connect-status" + namespace: "{{ .Namespace }}" +spec: + partitions: 1 + replicas: 3 + topicName: "{{ .Name }}.connect-status" + config: + cleanup.policy: compact + retention.ms: 604800000 + segment.bytes: 1073741824 diff --git a/charts/kafka-cluster/templates/_helpers.tpl b/charts/kafka-cluster/templates/_helpers.tpl new file mode 100644 index 0000000..72b6d5e --- /dev/null +++ b/charts/kafka-cluster/templates/_helpers.tpl @@ -0,0 +1,55 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "kafka-cluster.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +*/}} +{{- define "kafka-cluster.fullname" -}} +{{- if .Values.nameOverride }} +{{- .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- .Release.Name | default "kafka-cluster" | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "kafka-cluster.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "kafka-cluster.labels" -}} +helm.sh/chart: {{ include "kafka-cluster.chart" . }} +{{ include "kafka-cluster.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "kafka-cluster.selectorLabels" -}} +app.kubernetes.io/name: {{ include "kafka-cluster.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the namespace to use +*/}} +{{- define "kafka-cluster.namespace" -}} +{{- if .Values.namespaceOverride }} +{{- .Values.namespaceOverride }} +{{- else }} +{{- .Release.Namespace }} +{{- end }} +{{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka-connect.yaml b/charts/kafka-cluster/templates/kafka-connect.yaml new file mode 100644 index 0000000..07fb932 --- /dev/null +++ b/charts/kafka-cluster/templates/kafka-connect.yaml @@ -0,0 +1,53 @@ +{{- if .Values.kafkaConnect.enabled }} +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaConnect +metadata: + name: {{ include "kafka-cluster.fullname" . }} + namespace: {{ include "kafka-cluster.namespace" . }} + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} + annotations: + strimzi.io/use-connector-resources: "true" +spec: + image: {{ .Values.kafkaConnect.image }} + replicas: {{ .Values.kafkaConnect.replicas }} + bootstrapServers: {{ include "kafka-cluster.fullname" . }}-kafka-tls-bootstrap.{{ include "kafka-cluster.namespace" . }}.svc.cluster.local:{{ .Values.listener.port }} + config: + group.id: {{ include "kafka-cluster.fullname" . }}-default-connect + offset.storage.topic: {{ include "kafka-cluster.fullname" . }}.connect-offsets + config.storage.topic: {{ include "kafka-cluster.fullname" . }}.connect-configs + status.storage.topic: {{ include "kafka-cluster.fullname" . }}.connect-status + key.converter: org.apache.kafka.connect.json.JsonConverter + value.converter: org.apache.kafka.connect.json.JsonConverter + plugin.path: /opt/kafka/plugins + topic.creation.enable: "true" + {{- if .Values.oauth.enabled }} + authentication: + type: oauth + tokenEndpointUri: {{ .Values.oauth.keycloakUrl }}/realms/{{ .Values.oauth.keycloakRealm }}/protocol/openid-connect/token + clientId: {{ .Values.oauth.keycloakClientId }}-kafka-connect + clientSecret: + secretName: {{ .Values.kafkaConnect.oauthSecretName }} + key: client-secret + tlsTrustedCertificates: + - secretName: {{ .Values.oauth.tlsSecretName }} + pattern: {{ .Values.oauth.tlsCertificate | quote }} + {{- end }} + template: + connectContainer: + volumeMounts: + - name: truststore-volume + mountPath: /mnt/truststore/truststore.jks + subPath: truststore.jks + readOnly: true + pod: + volumes: + - name: truststore-volume + secret: + secretName: {{ .Values.kafkaConnect.truststoreSecretName }} + logging: + type: inline + loggers: + rootLogger.level: {{ .Values.logging.connect.rootLoggerLevel }} +{{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka-nodepool-broker.yaml b/charts/kafka-cluster/templates/kafka-nodepool-broker.yaml new file mode 100644 index 0000000..c6aa690 --- /dev/null +++ b/charts/kafka-cluster/templates/kafka-nodepool-broker.yaml @@ -0,0 +1,33 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: broker + namespace: {{ include "kafka-cluster.namespace" . }} + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.broker.replicas }} + roles: + - broker + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: {{ .Values.broker.storage.size }} + class: {{ .Values.broker.storage.class }} + kraftMetadata: shared + {{- if .Values.broker.resources }} + resources: + {{- if .Values.broker.resources.requests }} + requests: + cpu: {{ .Values.broker.resources.requests.cpu }} + memory: {{ .Values.broker.resources.requests.memory }} + {{- end }} + {{- if .Values.broker.resources.limits }} + limits: + cpu: {{ .Values.broker.resources.limits.cpu }} + memory: {{ .Values.broker.resources.limits.memory }} + {{- end }} + {{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka-nodepool-controller.yaml b/charts/kafka-cluster/templates/kafka-nodepool-controller.yaml new file mode 100644 index 0000000..af09ac7 --- /dev/null +++ b/charts/kafka-cluster/templates/kafka-nodepool-controller.yaml @@ -0,0 +1,33 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaNodePool +metadata: + name: controller + namespace: {{ include "kafka-cluster.namespace" . }} + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.controller.replicas }} + roles: + - controller + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: {{ .Values.controller.storage.size }} + kraftMetadata: shared + class: {{ .Values.controller.storage.class }} + {{- if .Values.controller.resources }} + resources: + {{- if .Values.controller.resources.requests }} + requests: + cpu: {{ .Values.controller.resources.requests.cpu }} + memory: {{ .Values.controller.resources.requests.memory }} + {{- end }} + {{- if .Values.controller.resources.limits }} + limits: + cpu: {{ .Values.controller.resources.limits.cpu }} + memory: {{ .Values.controller.resources.limits.memory }} + {{- end }} + {{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka-topics.yaml b/charts/kafka-cluster/templates/kafka-topics.yaml new file mode 100644 index 0000000..ed16bdc --- /dev/null +++ b/charts/kafka-cluster/templates/kafka-topics.yaml @@ -0,0 +1,53 @@ +{{- if .Values.kafkaConnect.enabled }} +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} + name: {{ include "kafka-cluster.fullname" . }}-connect-offsets + namespace: {{ include "kafka-cluster.namespace" . }} +spec: + partitions: {{ .Values.connectTopics.partitions }} + replicas: {{ .Values.connectTopics.replicas }} + topicName: {{ include "kafka-cluster.fullname" . }}.connect-offsets + config: + cleanup.policy: {{ .Values.connectTopics.config.cleanupPolicy }} + retention.ms: {{ .Values.connectTopics.config.retentionMs }} + segment.bytes: {{ .Values.connectTopics.config.segmentBytes }} +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} + name: {{ include "kafka-cluster.fullname" . }}-connect-configs + namespace: {{ include "kafka-cluster.namespace" . }} +spec: + partitions: {{ .Values.connectTopics.partitions }} + replicas: {{ .Values.connectTopics.replicas }} + topicName: {{ include "kafka-cluster.fullname" . }}.connect-configs + config: + cleanup.policy: {{ .Values.connectTopics.config.cleanupPolicy }} + retention.ms: {{ .Values.connectTopics.config.retentionMs }} + segment.bytes: {{ .Values.connectTopics.config.segmentBytes }} +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} + name: {{ include "kafka-cluster.fullname" . }}-connect-status + namespace: {{ include "kafka-cluster.namespace" . }} +spec: + partitions: {{ .Values.connectTopics.partitions }} + replicas: {{ .Values.connectTopics.replicas }} + topicName: {{ include "kafka-cluster.fullname" . }}.connect-status + config: + cleanup.policy: {{ .Values.connectTopics.config.cleanupPolicy }} + retention.ms: {{ .Values.connectTopics.config.retentionMs }} + segment.bytes: {{ .Values.connectTopics.config.segmentBytes }} +{{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka-user.yaml b/charts/kafka-cluster/templates/kafka-user.yaml new file mode 100644 index 0000000..2790c6f --- /dev/null +++ b/charts/kafka-cluster/templates/kafka-user.yaml @@ -0,0 +1,31 @@ +{{- if .Values.kafkaConnect.enabled }} +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaUser +metadata: + name: service-account-{{ .Values.oauth.keycloakClientId }}-kafka-connect + namespace: {{ include "kafka-cluster.namespace" . }} + labels: + strimzi.io/cluster: {{ include "kafka-cluster.fullname" . }} + {{- include "kafka-cluster.labels" . | nindent 4 }} +spec: + authorization: + type: simple + acls: + - resource: + type: topic + name: "*" + patternType: literal + operations: + - Read + - Describe + - DescribeConfigs + - Write + - resource: + type: group + name: "*" + patternType: literal + operations: + - Read + - Write + - Describe +{{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/templates/kafka.yaml b/charts/kafka-cluster/templates/kafka.yaml new file mode 100644 index 0000000..ed7cfa6 --- /dev/null +++ b/charts/kafka-cluster/templates/kafka.yaml @@ -0,0 +1,80 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: Kafka +metadata: + name: {{ include "kafka-cluster.fullname" . }} + namespace: {{ include "kafka-cluster.namespace" . }} + labels: + {{- include "kafka-cluster.labels" . | nindent 4 }} + annotations: + strimzi.io/node-pools: enabled + strimzi.io/kraft: enabled +spec: + kafka: + version: {{ .Values.kafka.version }} + metadataVersion: {{ .Values.kafka.metadataVersion | quote }} + template: + kafkaContainer: + env: + - name: KAFKA_OPTS + value: "-Duser.timezone={{ .Values.kafka.timezone }}" + listeners: + - name: tls + type: cluster-ip + port: {{ .Values.listener.port }} + tls: {{ .Values.listener.tls }} + {{- if .Values.oauth.enabled }} + authentication: + type: oauth + clientId: {{ .Values.oauth.keycloakClientId | quote }} + clientSecret: + key: {{ .Values.oauth.secretKey }} + secretName: {{ .Values.oauth.secretName }} + checkIssuer: true + checkAccessTokenType: true + accessTokenIsJwt: true + checkAudience: false + enableOauthBearer: true + validIssuerUri: {{ .Values.oauth.keycloakUrl }}/realms/{{ .Values.oauth.keycloakRealm }} + jwksEndpointUri: {{ .Values.oauth.keycloakUrl }}/realms/{{ .Values.oauth.keycloakRealm }}/protocol/openid-connect/certs + userNameClaim: preferred_username + customClaimCheck: "'{{ .Values.oauth.keycloakClientId }}' in @.realm_access.roles" + tlsTrustedCertificates: + - secretName: {{ .Values.oauth.tlsSecretName }} + certificate: {{ .Values.oauth.tlsCertificate }} + {{- end }} + config: + offsets.topic.replication.factor: {{ .Values.config.offsetsTopicReplicationFactor }} + transaction.state.log.replication.factor: {{ .Values.config.transactionStateLogReplicationFactor }} + transaction.state.log.min.isr: {{ .Values.config.transactionStateLogMinIsr }} + default.replication.factor: {{ .Values.config.defaultReplicationFactor }} + min.insync.replicas: {{ .Values.config.minInsyncReplicas }} + auto.create.topics.enable: {{ .Values.config.autoCreateTopicsEnable }} + num.partitions: {{ .Values.config.numPartitions }} + delete.topic.enable: {{ .Values.config.deleteTopicEnable }} + authorization: + type: simple + logging: + type: inline + loggers: + kafka.root.logger.level: {{ .Values.logging.kafka.rootLoggerLevel }} + entityOperator: + topicOperator: + {{- if .Values.entityOperator.topicOperator.resources }} + resources: + requests: + memory: {{ .Values.entityOperator.topicOperator.resources.requests.memory }} + cpu: {{ .Values.entityOperator.topicOperator.resources.requests.cpu }} + limits: + memory: {{ .Values.entityOperator.topicOperator.resources.limits.memory }} + cpu: {{ .Values.entityOperator.topicOperator.resources.limits.cpu }} + {{- end }} + userOperator: + {{- if .Values.entityOperator.userOperator.resources }} + resources: + requests: + memory: {{ .Values.entityOperator.userOperator.resources.requests.memory }} + cpu: {{ .Values.entityOperator.userOperator.resources.requests.cpu }} + limits: + memory: {{ .Values.entityOperator.userOperator.resources.limits.memory }} + cpu: {{ .Values.entityOperator.userOperator.resources.limits.cpu }} + {{- end }} \ No newline at end of file diff --git a/charts/kafka-cluster/values.yaml b/charts/kafka-cluster/values.yaml new file mode 100644 index 0000000..f51a963 --- /dev/null +++ b/charts/kafka-cluster/values.yaml @@ -0,0 +1,106 @@ +# Default values for kafka-cluster +nameOverride: "" +namespaceOverride: "" + +# Kafka cluster configuration +kafka: + version: "4.0.0" + metadataVersion: "4.0" + timezone: "Asia/Seoul" + +# Node pool configuration +controller: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: + requests: + cpu: 500m + memory: 1Gi + limits: + cpu: 1 + memory: 2Gi + +broker: + replicas: 3 + storage: + size: 50Gi + class: longhorn + resources: + requests: + cpu: 500m + memory: 1Gi + limits: + cpu: 1 + memory: 2Gi + +# Kafka configuration +config: + offsetsTopicReplicationFactor: 3 + transactionStateLogReplicationFactor: 3 + transactionStateLogMinIsr: 2 + defaultReplicationFactor: 3 + minInsyncReplicas: 2 + autoCreateTopicsEnable: false + numPartitions: 3 + deleteTopicEnable: true + +# Listener configuration +listener: + port: 9093 + tls: false + +# OAuth/Keycloak configuration +oauth: + enabled: true + keycloakUrl: "" + keycloakRealm: "" + keycloakClientId: "" + secretName: kafka-cluster-oauth-secret + secretKey: client-secret + tlsSecretName: keycloak-tls + tlsCertificate: ca.crt + +# Kafka Connect configuration +kafkaConnect: + enabled: true + image: paasup/kafka-connect:0.2 + replicas: 1 + oauthSecretName: kafka-connect-oauth-secret + truststoreSecretName: truststore + +# Entity Operator +entityOperator: + topicOperator: + resources: + requests: + memory: 128Mi + cpu: 100m + limits: + memory: 256Mi + cpu: 200m + userOperator: + resources: + requests: + memory: 128Mi + cpu: 100m + limits: + memory: 256Mi + cpu: 200m + +# Kafka Topics for Connect +connectTopics: + partitions: 1 + replicas: 3 + config: + cleanupPolicy: compact + retentionMs: 604800000 + segmentBytes: 1073741824 + +# Logging +logging: + kafka: + rootLoggerLevel: INFO + connect: + rootLoggerLevel: INFO