feat(uat): add connection state check to Paho Python client

[eschastlivtsev]

Issue #, if available:
MQTT Client Control does not send shutdown agent request
Windows build: script does not fail in case of error

Description of changes:

• Add connection state check before publish, subscribe, unsubscribe, shutdown
• Add shutdown initiated check before shutdown
• Add errors handling for Windows build script - fail in case of error

Why is this change necessary:
Currently paho python client throws an exception when closeMqttConnection gRPC request is called on connection already closed from broker's side.
We need unify behavior of clients to simplify control and MQTT steps.

Currently Windows build script does not fail in case of error, it can be confusing.

How was this change tested:
Manually by run Control as application and local mosquitto broker.
When client is connected terminate broker before first subscribe.
Observe results in client and control logs:

Test results:
Control:

java -Dmqtt_client_id=Python_Paho_Client -Dmqtt_broker_addr=127.0.0.1 -Dmqtt_broker_port=1883 -Dmqtt_client_ca_file=creds_aws/rootCA.pem -Dmqtt_client_cert_file=creds_aws/thingCert.crt -Dmqtt_client_key_file=creds_aws/privKey.key -jar target/aws-greengrass-testing-mqtt-client-control.jar 47619 false true
[INFO ] 2023-08-03 17:11:55.159 [main] ExampleControl - Control: port 47619, without TLS, MQTT v5
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
[INFO ] 2023-08-03 17:11:57.690 [main] EngineControlImpl - MQTT client control gRPC server started, listening on 47619
[INFO ] 2023-08-03 17:12:03.862 [grpc-default-executor-0] GRPCDiscoveryServer - RegisterAgent: agentId paho-python-agent
[INFO ] 2023-08-03 17:12:03.888 [grpc-default-executor-0] GRPCDiscoveryServer - DiscoveryClient: agentId paho-python-agent address 127.0.0.1 port 61638
[INFO ] 2023-08-03 17:12:03.893 [grpc-default-executor-0] EngineControlImpl - Created new agent control for paho-python-agent on 127.0.0.1:61638
[INFO ] 2023-08-03 17:12:03.948 [grpc-default-executor-0] ExampleControl - Agent paho-python-agent is connected
[INFO ] 2023-08-03 17:12:03.953 [pool-2-thread-1] AgentTestScenario - Playing test scenario for agent id paho-python-agent
[INFO ] 2023-08-03 17:12:06.962 [pool-2-thread-1] AgentTestScenario - Set CONNECT user property: region, US
[INFO ] 2023-08-03 17:12:06.962 [pool-2-thread-1] AgentTestScenario - Set CONNECT user property: type, JSON
[INFO ] 2023-08-03 17:12:06.976 [pool-2-thread-1] AgentTestScenario - Set CONNECT request response information true
[INFO ] 2023-08-03 17:12:07.156 [pool-2-thread-1] AgentControlImpl - Created connection with id 1 CONNACK 'sessionPresent: false
reasonCode: 0
receiveMaximum: 20
topicAliasMaximum: 10
'
[INFO ] 2023-08-03 17:12:07.157 [pool-2-thread-1] AgentControlImpl - createMqttConnection: MQTT connectionId 1 created
[INFO ] 2023-08-03 17:12:07.161 [pool-2-thread-1] AgentTestScenario - MQTT connection with id 1 is established
[INFO ] 2023-08-03 17:12:08.972 [grpc-default-executor-0] GRPCDiscoveryServer - OnMqttDisconnect: agentId paho-python-agent connectionId 1 disconnect 'reasonCode: 7
' error ''
[INFO ] 2023-08-03 17:12:08.973 [grpc-default-executor-0] AgentTestScenario - MQTT disconnected on agentId paho-python-agent connectionId 1 disconnect 'reasonCode: 7
' error ''
[INFO ] 2023-08-03 17:12:12.163 [pool-2-thread-1] AgentTestScenario - Set SUBSCRIBE user property: region, US
[INFO ] 2023-08-03 17:12:12.163 [pool-2-thread-1] AgentTestScenario - Set SUBSCRIBE user property: type, JSON
[INFO ] 2023-08-03 17:12:12.169 [pool-2-thread-1] AgentControlImpl - SubscribeMqtt: subscribe on connection 1
[ERROR] 2023-08-03 17:12:12.180 [pool-2-thread-1] AgentTestScenario - gRPC error code INTERNAL: description: MQTT client is not in connected state
io.grpc.StatusRuntimeException: INTERNAL: MQTT client is not in connected state
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at com.aws.greengrass.testing.mqtt.client.MqttClientControlGrpc$MqttClientControlBlockingStub.subscribeMqtt(MqttClientControlGrpc.java:511) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at com.aws.greengrass.testing.mqtt.client.control.implementation.AgentControlImpl.subscribeMqtt(AgentControlImpl.java:224) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at com.aws.greengrass.testing.mqtt.client.control.implementation.ConnectionControlImpl.subscribeMqtt(ConnectionControlImpl.java:123) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at com.aws.greengrass.testing.mqtt.client.control.AgentTestScenario.testSubscribe(AgentTestScenario.java:255) ~[aws-greengrass-testing-mqtt-client-control.jar:?]
        at com.aws.greengrass.testing.mqtt.client.control.AgentTestScenario.run(AgentTestScenario.java:183) [aws-greengrass-testing-mqtt-client-control.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[INFO ] 2023-08-03 17:12:12.195 [pool-2-thread-1] AgentTestScenario - Set DISCONNECT user property: region, US
[INFO ] 2023-08-03 17:12:12.199 [pool-2-thread-1] AgentTestScenario - Set DISCONNECT user property: type, JSON
[INFO ] 2023-08-03 17:12:12.212 [pool-2-thread-1] AgentControlImpl - closeMqttConnection: MQTT connectionId 1 closed
[INFO ] 2023-08-03 17:12:12.215 [pool-2-thread-1] AgentControlImpl - sending shutdown request
[INFO ] 2023-08-03 17:12:12.222 [pool-2-thread-1] AgentControlImpl - shutdown request sent successfully
[INFO ] 2023-08-03 17:12:12.233 [grpc-default-executor-0] GRPCDiscoveryServer - UnregisterAgent: agentId paho-python-agent reason Agent shutdown by OTF request 'That's it.'
[INFO ] 2023-08-03 17:12:12.234 [grpc-default-executor-0] AgentControlImpl - shutting down channel with agent id paho-python-agent
[INFO ] 2023-08-03 17:12:12.237 [grpc-default-executor-0] AgentControlImpl - channel terminated with agent id paho-python-agent
[INFO ] 2023-08-03 17:12:12.237 [grpc-default-executor-0] ExampleControl - Agent paho-python-agent is disconnected

Paho Python client:

client-python-paho.exe paho-python-agent
[INFO ] 2023-08-03 17:12:03.674 GRPCLib - Initialize gRPC library
[INFO ] 2023-08-03 17:12:03.675 GRPCLink - Making gPRC client connection with 127.0.0.1:47619 as paho-python-agent...
[INFO ] 2023-08-03 17:12:03.878 GRPCLink - Client connection with Control is established, local address is 127.0.0.1
[INFO ] 2023-08-03 17:12:03.955 MQTTLib - Initialize Paho MQTT library
[INFO ] 2023-08-03 17:12:03.956 GRPCLink - Handle gRPC requests
[INFO ] 2023-08-03 17:12:03.957 GRPCControlServer - Server awaiting termination
[INFO ] 2023-08-03 17:12:07.035 GRPCControlServer - createMqttConnection: clientId Python_Paho_Client broker 127.0.0.1:1883
[INFO ] 2023-08-03 17:12:07.035 MqttConnection - Creating MQTT 5 client without TLS
[INFO ] 2023-08-03 17:12:07.036 MqttConnection - CONNECT TX user property 'region', 'US'
[INFO ] 2023-08-03 17:12:07.037 MqttConnection - CONNECT TX user property 'type', 'JSON'
[INFO ] 2023-08-03 17:12:07.039 MqttConnection - Copied TX request response information 'True'
[INFO ] 2023-08-03 17:12:07.039 MqttConnection - MQTT connection ID 1 connecting...
[INFO ] 2023-08-03 17:12:07.056 MqttConnection - MQTT connection ID 1 connected, client id Python_Paho_Client
[INFO ] 2023-08-03 17:12:08.964 GRPCDiscoveryClient - Sending OnMqttDisconnect request agent_id 'paho-python-agent' connection_id 1 error 'None'
[INFO ] 2023-08-03 17:12:08.981 MqttConnection - MQTT connectionId 1 disconnected error ''
[INFO ] 2023-08-03 17:12:12.174 GRPCControlServer - SubscribeMqtt connection_id 1
[WARNING] 2023-08-03 17:12:12.175 GRPCControlServer - SubscribeMqtt: exception during subscribing
[INFO ] 2023-08-03 17:12:12.204 GRPCControlServer - CloseMqttConnection connection_id 1 reason 4
[INFO ] 2023-08-03 17:12:12.205 MqttConnection - Disconnect MQTT connection with reason code 4
[WARNING] 2023-08-03 17:12:12.207 MqttConnection - DISCONNECT was not sent on the dead connection
[INFO ] 2023-08-03 17:12:12.219 GRPCControlServer - shutdownAgent: reason That's it.
[INFO ] 2023-08-03 17:12:12.222 GRPCControlServer - Server termination done
[INFO ] 2023-08-03 17:12:12.225 GRPCLink - Shutdown gPRC link
[INFO ] 2023-08-03 17:12:12.240 Main - Execution done successfully

Maven Windows build log on the machine where python is not installed - we can see that now build fails because of "pip is not installed" error

[INFO] --- jar:3.3.0:jar (default-jar) @ client-devices-auth-uat-client-mosquitto-c ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: C:\2\aws-greengrass-client-device-auth\uat\custom-components\client-mosquitto-c\target\client-devices-auth-uat-client-mosquitto-c-1.0-SNAPSHOT.jar
[INFO]
[INFO] ---< com.aws.greengrass:client-devices-auth-uat-client-python-paho >----
[INFO] Building client-devices-auth-uat-client-python-paho 1.0-SNAPSHOT   [7/9]
[INFO]   from custom-components\client-python-paho\pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- clean:3.2.0:clean (default-clean) @ client-devices-auth-uat-client-python-paho ---
[INFO]
[INFO] --- license:4.2.rc3:check (default) @ client-devices-auth-uat-client-python-paho ---
[INFO] Checking licenses...
[WARNING] Unknown file extension: C:/2/aws-greengrass-client-device-auth/uat/custom-components/client-python-paho/src/pyproject.toml
[WARNING] Unable to find a comment style definition for some files. You may want to add a custom mapping for the relevant file extensions.
[INFO]
[INFO] --- resources:3.3.0:resources (default-resources) @ client-devices-auth-uat-client-python-paho ---
[INFO] skip non existing resourceDirectory C:\2\aws-greengrass-client-device-auth\uat\custom-components\client-python-paho\src\main\resources
[INFO]
[INFO] --- compiler:3.10.1:compile (default-compile) @ client-devices-auth-uat-client-python-paho ---
[INFO] No sources to compile
[INFO]
[INFO] --- exec:3.1.0:exec (client-python-docker-build) @ client-devices-auth-uat-client-python-paho ---
'pip' is not recognized as an internal or external command,
operable program or batch file.
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:1000)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:947)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:471)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for client-devices-auth-uat 1.0-SNAPSHOT:
[INFO]
[INFO] client-devices-auth-uat ............................ SUCCESS [  0.288 s]
[INFO] client-devices-auth-custom-components .............. SUCCESS [  0.007 s]
[INFO] client-devices-auth-uat-client-ipc ................. SUCCESS [ 46.812 s]
[INFO] client-devices-auth-uat-client-java-paho ........... SUCCESS [ 59.498 s]
[INFO] client-devices-auth-uat-client-java-sdk ............ SUCCESS [ 52.892 s]
[INFO] client-devices-auth-uat-client-mosquitto-c ......... SUCCESS [  0.782 s]
[INFO] client-devices-auth-uat-client-python-paho ......... FAILURE [  0.222 s]
[INFO] aws-greengrass-testing-mqtt-client-control ......... SKIPPED
[INFO] client-devices-auth-testing-features ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:43 min
[INFO] Finished at: 2023-08-03T17:43:11+03:00
[INFO] ------------------------------------------------------------------------

Any additional information or context required to review the change:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.