PGsync on kubernetes (K8)

[behnood-R]

We have an environment that needs to run everything only on K8. Is there any way to run the Pgsync directly on Kubernetes?
How about the Pgsync K8 deployment YAML file?

[toluaina]

I would welcome a PR for a K8 yaml file.
There is however some work to do related to this.
For instance, you need to store the pgsync state/checkpoint file.
Also you should not run multiple instance of pgsync.

[errietta]

Semi-related. Do you think it would be good to have an option to store the state on s3 for people to not have to set up persistent storage on k8s?

[toluaina]

Yes there have been a few discussions around this. One was Redis and the other was S3.
Having an option for any seems like the ideal. Still thinking about how to go about this in the most user frielndly way

[errietta]

If it helps anyone, I made this for my job just the other day:

resource "aws_ecr_repository" "example-pgsync" {
  name                 = "example-pgsync"
  image_tag_mutability = "MUTABLE"
  image_scanning_configuration {
    scan_on_push = true
  }
  tags = {
    Environment = var.environment
    terraform   = true
  }
}

resource "aws_ebs_volume" "pgsync_ebs_volume" {
  availability_zone = "${var.region}b"
  size              = 1
}

resource "kubernetes_persistent_volume" "pgsync_checkpoint_pv" {
  metadata {
    name = "pgsync-checkpoint-pv"
  }

  spec {
    persistent_volume_source {
      aws_elastic_block_store {
        volume_id = aws_ebs_volume.pgsync_ebs_volume.id
        fs_type   = "ext4"
      }
    }
    capacity = {
      storage = "1Gi"
    }

    storage_class_name = "ebs-sc"

    access_modes = ["ReadWriteOnce"]
  }
}

resource "kubernetes_persistent_volume_claim" "pgsync_checkpoint_claim" {
  metadata {
    name = "pgsync-checkpoint-claim"
  }
  spec {
    access_modes = ["ReadWriteOnce"]
    resources {
      requests = {
        storage = "1Gi"
      }
    }

    selector {
      match_labels = {
        app = "pgsync-example"
      }
    }
    volume_name        = kubernetes_persistent_volume.pgsync_checkpoint_pv.metadata[0].name
    storage_class_name = "ebs-sc"
  }
}

resource "kubernetes_deployment" "example-pgsync" {
  wait_for_rollout = local.wait_for_rollout
  metadata {
    name      = "pgsync-example"
    namespace = local.namespace
    labels = {
      app      = "pgsync-example"
      platform = "fargate"
    }
  }
  spec {
    replicas = 1

    strategy {
      # you can run only strictly one replica of pgsync
      type = "Recreate"
    }

    selector {
      match_labels = {
        app = "pgsync-example"
      }
    }
    template {
      metadata {
        labels = {
          app = "pgsync-example"
        }
      }
      spec {
        node_selector = {
          "az" = "${var.region}b"
        }
        volume {
          name = "pgsync-persistent-storage"
          persistent_volume_claim {
            claim_name = kubernetes_persistent_volume_claim.pgsync_checkpoint_claim.metadata.0.name
          }
        }

        container {
          volume_mount {
            name       = "pgsync-persistent-storage"
            mount_path = "/checkpoint"
          }


          image             = "your-ecr-image"
          image_pull_policy = "Always"
          name              = "pgsync-example"

          command = ["poetry", "run", "pgsync", "--config", "/app/schema.json", "--daemon"]

          resources {
            limits   = {}
            requests = {}
          }
          env {
            # Trying to avoid too many requests from elastic
            name  = "ELASTICSEARCH_CHUNK_SIZE"
            value = var.pgsync_es_chunk_size
          }
          env {
            name  = "ELASTICSEARCH_STREAMING_BULK"
            value = 1
          }

          env {
            name  = "PG_HOST"
            value = local.terraform_secrets["PGSYNC_PG_HOST"]
          }
          env {
            name  = "PG_USER"
            value = local.terraform_secrets["PGSYNC_PG_USER"]
          }
          env {
            name  = "PG_PASSWORD"
            value = local.terraform_secrets["PGSYNC_PG_PASSWORD"]
          }
          env {
            name  = "ELASTICSEARCH_HOST"
            value = local.terraform_secrets["PGSYNC_ES_HOST"]
          }
          env {
            name  = "ELASTICSEARCH_PORT"
            value = 443
          }
          env {
            name  = "ELASTICSEARCH_SCHEME"
            value = "https"
          }
          env {
            name  = "REDIS_HOST"
            value = local.terraform_secrets["PGSYNC_REDIS_HOST"]
          }
          env {
            name  = "REDIS_DB"
            value = local.terraform_secrets["PGSYNC_REDIS_DB"]
          }
          # needs to be max the number of CPUs available for the ES cluster
          # (to avoid 429 too many requests)
          env {
            name  = "ELASTICSEARCH_THREAD_COUNT"
            value = var.pgsync_thread_count
          }

          env {
            # put pgsync checkpoint (which tells it where it left off) in the pv
            name  = "CHECKPOINT_PATH"
            value = "/checkpoint/"
          }

          env {
            name  = "ELASTICSEARCH_MAX_RETRIES"
            value = 3
          }
        }
      }
    }
  }
}

terraform config that sets up pgsync instance, including with persistent storage.
It may not work for someone else the way it is because of our setup but it would be a good starting point.

[errietta]

I then just push a docker image to ECR which includes a build of pgsync together with my schema.