Kafka Connect for z/OS support #1323

stan-is-hate · 2023-02-02T23:08:31Z

Description

This PR adds support for running kafka connect on z/OS. This is in BETA, as in not everything has been tested and some things don't work.
I'm not really familiar with your branching strategy and where to point this PR, would appreciate any pointers.

I've tested it using IBM zD&T and cp-env.
A sister PR for cp-env is here: https://github.com/confluentinc/cp-test-env-manager/pull/136

Implementation details:

z/OS requires a set of environment variables to be set. cp-ansible provided a variable called proxy_env. Even though it was supposed to be only used for specifying proxy details, in fact all playbooks set it as environment for all tasks automatically. I've renamed it env_vars to be more explicit about what the variable does. This is the only breaking change, all the rest of the changes are hidden behind the ansible_os_family check.
z/OS doesn't have SystemD or package managers. It also doesn't have gunzip. So what we do is download confluent tgz locally on your laptop, gunzip it there (but not untar) and then pass the tar to z/os, where we untar it.
z/OS doesn't support confluent hub, so we can only install local and remote connectors.
This is aimed primarily at testing IBM MQ connectors, and these connectors require IBM jars and .so files to be present in their classpath. So I've added a special hook when installing connectors which will recognize ibm mq connectors and copy jars and .so files onto their classpath.
Because we're running in virtualized z/OS, I've added a health_check_hostname var, to make sure guest OS hits its guest IP rather than host IP when doing health checks. I'll do more tests without it as well to make sure the issues I was having were not a fluke, but in any case this change is non breaking.

Limitations and unknowns:

SSL doesn't work. Even though I have confirmed that the playbook creates keystores with correct certificates, connect still refuses to work correctly. It's better if connect team handles this. We can document it as not supported for now.
Confluent CLI doesn't work on z/OS (that's a product limitation), so any functionality involving it won't work either, e.g. secrets protection.
I have not tested deploy_connectors.yml because it is not exposed in cp-env yet, but looking at the code it's all done via REST, so should work fine
I have tested RBAC with kerberos, and it does work. However, when rbac is enabled, connect does not use kerberos at all, so there's no guarantee kerberos would work. Because we're running in virtualized instances, there are multiple hostnames involved, one of the host OS, another of the guest OS Kerberos relies on hostnames a lot, so there might be issues with that. I'll test it some more.

Type of change

Bug fix (non-breaking change which fixes an issue)
[x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Any variable changes have been validated to be backwards compatible

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
Any dependent changes have been merged and published in downstream modules

…ader

Copy only MQ jars to ext folder to be loaded by the extension classloader

nsharma-git · 2023-03-13T06:11:28Z

playbooks/all.yml

@@ -2,7 +2,7 @@
 - name: Host Prerequisites
  hosts: zookeeper:kafka_broker:schema_registry:kafka_connect:ksql:control_center:kafka_rest:kafka_connect_replicator
  gather_facts: false
-  environment: "{{ proxy_env }}"


This is a good but breaking change. Can we make an alias for backward compatibility and deprecate proxy_env for couple of releases.

nsharma-git · 2023-03-13T06:24:30Z

roles/common/tasks/copy_files.yml

+
+- name: Copy Files (z/OS)
+  when: ansible_os_family == "OS/390"
+  ibm.ibm_zos_core.zos_copy:


If this part of ansible core? If not, we need to add this dependency in our documentation.

nsharma-git · 2023-03-13T06:27:22Z

roles/common/tasks/host_validations.yml

@@ -91,111 +96,114 @@
      "Not enough free disk space for package installation.  Minimum space required is {{ required_disk_space_mb }}MB, please free up more disk space
       to continue installation. To skip host validations, set validate_hosts to false."
  when: (item.size_available < required_disk_space_mb|float)
-  with_items: "{{ ansible_mounts }}"
+  with_items: "{{ ansible_mounts if ansible_mounts is defined else {} }}"


We have used default keys for such conditions in many places.

nsharma-git · 2023-03-13T06:27:48Z

roles/common/tasks/host_validations.yml

+- name: Check Internet Access
+  # for z/OS we don't download the archive from the internet
+  # but rather download it locally on your laptop, and then push to z/os node
+  when: ansible_os_family != "OS/390"


Will there be no check for Z/OS ?

No need to check internet access, because z/OS won't download anything from the internet.
I think it was done due to z/os python not being able to validate any ssl certificates, even when I setup correct ssl flags, but I might be wrong, it's been a while :)

nsharma-git · 2023-03-13T07:35:33Z

roles/common/tasks/host_validations.yml

-    - validate
-    - validate_memory_usage
-    - validate_memory_usage_zookeeper
+- name: Validate memory usage


Thanks for this!

nsharma-git · 2023-03-13T07:38:33Z

roles/common/tasks/main.yml

@@ -62,6 +62,11 @@
  tags:
    - validate

+- name: Symlink bash on z/OS


Can we please use ansible file or some other core module for the this.

I wanted to, originally, but built in module was failing, complaining /bin/bash didn't exist.

nsharma-git · 2023-03-13T07:40:44Z

roles/common/tasks/main.yml

+    - installation_method == "archive"
+    - ansible_os_family != "OS/390"
+
+- name: Expand remote Confluent Platform archive (z/OS)


This is a good improvement but its diverging from existing structure. I would like to take this up in separate PR and probably implement for all OS

Existing structure just wouldn't work for z/OS, because it is way too special :D

nsharma-git · 2023-03-13T15:01:19Z

roles/kafka_connect/tasks/main.yml

@@ -4,6 +4,13 @@
  when: not common_role_completed|bool
  tags: common

+- name: Check z/OS Preconditions
+  fail:
+    msg: 'z/OS must be installed from archive'


Should I read it as "On z/OS, Connect service can be installed via archive"

Yeah, I'll change it to "On z/OS Confluent Platform Services MUST be installed via archive"

stan-is-hate · 2023-04-07T23:58:06Z

Thanks @nsharma-git , I don't have much bandwidth these days, but I'll try to address your comments in the oncoming weeks. For anything where z/os diverges from the way we do things on other platforms it's done so out of necessity, e.g. z/os cannot unpack tar.gz files, it can only work with uncompressed tar or some weird compression algorithm, but not gz or bz2. That's why we download archive locally first, unzip but not untar it, and then move to the remote machine. We don't untar it locally because un-taring it on z/os updates encoding tags on the files correctly, if memory serves

cla-assistant · 2023-08-21T16:34:21Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ Veerabhadra
❌ stan-confluent
❌ pbadani

Veerabhadra seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

stan-is-hate added 7 commits January 31, 2023 17:32

zos support for connect initial

4e34992

zos support for connect initial

843193c

zos support for connect initial

e3e6c25

kerberos zos

37b17de

force symlink bash

b533188

processing for ibm mq connectors

d8d759d

cleanup

a86b67b

stan-is-hate requested a review from a team as a code owner February 2, 2023 23:08

fix env vars

2da9f1f

imcdo previously approved these changes Feb 6, 2023

View reviewed changes

stan-is-hate and others added 4 commits February 22, 2023 23:40

only copy jar files

a71b033

Copy only MQ jars to ext folder to be loaded by the extension classlo…

ce6347e

…ader

Remove unused var.

fa6f6ee

Merge pull request #1357 from confluentinc/pbadani/ibm-mq

ac2a417

Copy only MQ jars to ext folder to be loaded by the extension classloader

nsharma-git reviewed Mar 13, 2023

View reviewed changes

updated parameters for zos_encode

ae0ef8a

vr029 requested a review from a team as a code owner August 21, 2023 16:34

stan-is-hate dismissed imcdo’s stale review via ae0ef8a October 30, 2023 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka Connect for z/OS support #1323

Kafka Connect for z/OS support #1323

stan-is-hate commented Feb 2, 2023 •

edited by rramakrishnan7

Loading

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023 •

edited

Loading

nsharma-git Mar 13, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

nsharma-git Mar 13, 2023

stan-is-hate Apr 7, 2023

stan-is-hate commented Apr 7, 2023

cla-assistant bot commented Aug 21, 2023 •

edited

Loading

Kafka Connect for z/OS support #1323

Are you sure you want to change the base?

Kafka Connect for z/OS support #1323

Conversation

stan-is-hate commented Feb 2, 2023 • edited by rramakrishnan7 Loading

Description

Type of change

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stan-is-hate Apr 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stan-is-hate commented Apr 7, 2023

cla-assistant bot commented Aug 21, 2023 • edited Loading

stan-is-hate commented Feb 2, 2023 •

edited by rramakrishnan7

Loading

stan-is-hate Apr 7, 2023 •

edited

Loading

cla-assistant bot commented Aug 21, 2023 •

edited

Loading