Created
February 11, 2022 13:35
-
-
Save ajiteshk-G/102c1f3c87b57c2b8837251b1cc8bad2 to your computer and use it in GitHub Desktop.
Debezium GKE Setup
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: ConfigMap | |
| metadata: | |
| name: debezium | |
| labels: | |
| app: debezium | |
| namespace: debezium | |
| data: | |
| application.properties: |- | |
| debezium.sink.type=pubsub | |
| debezium.sink.pubsub.project.id=acko-poc | |
| debezium.sink.pubsub.ordering.enabled=true | |
| debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector | |
| debezium.source.offset.storage.file.filename=data/offsets.dat | |
| debezium.source.offset.flush.interval.ms=0 | |
| debezium.source.database.hostname=localhost #as we are connecting via Cloud Proxy | |
| debezium.source.database.port= <Postgres Port> #generally 5432 | |
| debezium.source.database.user= <user created> # in our case - debezium_user | |
| debezium.source.database.password=<password while creating new user> #in our case "admin" | |
| debezium.source.database.dbname=<Database> #OurCase postgres | |
| debezium.source.database.server.name=<Database Server name> #In Our Case debezium-db | |
| debezium.source.table.include.list=public.actor #Table which needs to captured | |
| debezium.source.plugin.name=pgoutput |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: Service | |
| metadata: | |
| name: debezium | |
| labels: | |
| name: debezium | |
| namespace: debezium | |
| spec: | |
| type: ClusterIP | |
| ports: | |
| - port: 8080 | |
| targetPort: 8080 | |
| protocol: TCP | |
| name: http | |
| selector: | |
| name: debezium |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: apps/v1 | |
| kind: StatefulSet | |
| metadata: | |
| name: debezium-gke | |
| labels: | |
| name: debezium | |
| namespace: debezium | |
| spec: | |
| replicas: 1 | |
| serviceName: debezium | |
| selector: | |
| matchLabels: | |
| name: debezium | |
| template: | |
| metadata: | |
| labels: | |
| name: debezium | |
| version: v1 | |
| spec: | |
| serviceAccountName: <Service Name Created while creating Cluster> | |
| nodeSelector: | |
| iam.gke.io/gke-metadata-server-enabled: "true" | |
| containers: | |
| - name: debezium | |
| image: debezium/server:1.7.0.Final | |
| volumeMounts: | |
| - name: debezium-config-volume | |
| mountPath: /debezium/conf | |
| - name: debezium-data-volume | |
| mountPath: /debezium/data | |
| - name: cloud-sql-proxy | |
| image: gcr.io/cloudsql-docker/gce-proxy:1.17 | |
| command: | |
| - "/cloud_sql_proxy" | |
| - "-instances=<ConnectionName from CloudSQL Page>=tcp:<Port>" | |
| securityContext: | |
| runAsNonRoot: true | |
| volumes: | |
| - name: debezium-config-volume | |
| configMap: | |
| name: debezium | |
| volumeClaimTemplates: | |
| - metadata: | |
| name: debezium-data-volume | |
| spec: | |
| accessModes: [ "ReadWriteOnce" ] | |
| resources: | |
| requests: | |
| storage: 10Gb |
Hey As the exception says- the Sink Type config is missing
Please use this debezium.sink.type=pubsub
Hi , i have already added this debezium.sink.type=pubsub in my config map file, but I'm still getting this error
Check if your configMap is updated and deployed. Also, the Pub Sub topic name needs to be in the proper format as mentioned
Hi that error is gone now, but I can this new error
Seems to be related to your configuration, Please validate
My PG SQL version is 11, do you think this may be causing the issue ?
And also i just want to ask one more thing in the debezium-statefulset.yaml file at line no-20, which service account do we need to pass? the one which is using by GKE VM's?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @ajiteshk-G , i was trying to replicate this process, but the debezium container is failing with the below error.
Did faced the same or if you can share some insights on how to fix this.
Also in the debezium-statefulset.yaml file at line no-20, which service account do we need to pass? the one which is using by GKE VM's?
Awaiting for reply