Issue starting cadence server using docker-compose #4540

naresh8t7 · 2021-10-05T20:30:27Z

I am trying to run the cadence server using docker-compose and I am seeing below issues.

docker-compose up
Starting cadence_node-exporter_1 ... done
Starting cadence_prometheus_1    ... done
Starting cadence_cassandra_1     ... done
Starting cadence_cadence_1       ... done
Starting cadence_grafana_1       ... done
Starting cadence_cadence-web_1   ... done
Attaching to cadence_cassandra_1, cadence_prometheus_1, cadence_node-exporter_1, cadence_grafana_1, cadence_cadence_1, cadence_cadence-web_1
cadence_1        | + DB=cassandra
cadence_1        | + ENABLE_ES=false
cadence_1        | + ES_PORT=9200
cadence_1        | + ES_VERSION=v6
grafana_1        | GF_PATHS_CONFIG='/etc/grafana/grafana.ini' is not readable.
grafana_1        | GF_PATHS_DATA='/var/lib/grafana' is not writable.
grafana_1        | GF_PATHS_HOME='/usr/share/grafana' is not readable.
cadence_1        | + RF=1
prometheus_1     | level=info ts=2021-10-05T14:31:48.909Z caller=main.go:400 msg="No time or size retention was set so using the default time retention" duration=15d
prometheus_1     | level=info ts=2021-10-05T14:31:48.909Z caller=main.go:438 msg="Starting Prometheus" version="(version=2.30.2, branch=HEAD, revision=b30db03f35651888e34ac101a06e25d27d15b476)"
prometheus_1     | level=info ts=2021-10-05T14:31:48.909Z caller=main.go:443 build_context="(go=go1.17.1, user=root@80d1e84a93dd, date=20211001-12:26:58)"
prometheus_1     | level=info ts=2021-10-05T14:31:48.909Z caller=main.go:444 host_details="(Linux 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 0c84c73bf676 (none))"
cadence_1        | + export CASSANDRA_USER=cassandra
cadence_1        | + CASSANDRA_USER=cassandra
node-exporter_1  | level=info ts=2021-10-05T14:31:48.637Z caller=node_exporter.go:182 msg="Starting node_exporter" version="(version=1.2.2, branch=HEAD, revision=26645363b486e12be40af7ce4fc91e731a33104e)"
node-exporter_1  | level=info ts=2021-10-05T14:31:48.638Z caller=node_exporter.go:183 msg="Build context" build_context="(go=go1.16.7, user=root@b9cb4aa2eb17, date=20210806-13:44:18)"
prometheus_1     | level=info ts=2021-10-05T14:31:48.910Z caller=main.go:445 fd_limits="(soft=1048576, hard=1048576)"
prometheus_1     | level=info ts=2021-10-05T14:31:48.910Z caller=main.go:446 vm_limits="(soft=unlimited, hard=unlimited)"
grafana_1        | You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
cadence_1        | + export CASSANDRA_PASSWORD=cassandra
cadence_1        | + CASSANDRA_PASSWORD=cassandra
prometheus_1     | level=info ts=2021-10-05T14:31:48.915Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Starting Grafana" logger=server version=8.1.5 commit=c725a4259a branch=HEAD compiled=2021-09-21T08:54:08+0000
node-exporter_1  | level=info ts=2021-10-05T14:31:48.639Z caller=filesystem_common.go:110 collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+)($|/)
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=filesystem_common.go:112 collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
cadence_1        | + export KEYSPACE=cadence
cadence_1        | + KEYSPACE=cadence
cadence_1        | + export VISIBILITY_KEYSPACE=cadence_visibility
cadence_1        | + VISIBILITY_KEYSPACE=cadence_visibility
prometheus_1     | level=info ts=2021-10-05T14:31:48.915Z caller=main.go:822 msg="Starting TSDB ..."
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="App mode production" logger=settings
prometheus_1     | level=info ts=2021-10-05T14:31:48.920Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:108 msg="Enabled collectors"
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
prometheus_1     | level=info ts=2021-10-05T14:31:48.923Z caller=head.go:466 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
prometheus_1     | level=info ts=2021-10-05T14:31:48.924Z caller=head.go:500 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.2µs
prometheus_1     | level=info ts=2021-10-05T14:31:48.924Z caller=head.go:506 component=tsdb msg="Replaying WAL, this may take a while"
cadence_1        | + export CASSANDRA_PROTO_VERSION=4
cadence_1        | + CASSANDRA_PROTO_VERSION=4
cadence_1        | + export DBNAME=cadence
cadence_1        | + DBNAME=cadence
cadence_1        | + export VISIBILITY_DBNAME=cadence_visibility
cadence_1        | + VISIBILITY_DBNAME=cadence_visibility
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=arp
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=bcache
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=bonding
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=btrfs
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=conntrack
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=cpu
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=cpufreq
prometheus_1     | level=info ts=2021-10-05T14:31:48.926Z caller=head.go:577 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2
cadence_1        | + export DB_PORT=3306
cadence_1        | + DB_PORT=3306
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=diskstats
node-exporter_1  | level=info ts=2021-10-05T14:31:48.640Z caller=node_exporter.go:115 collector=edac
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=entropy
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=fibrechannel
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=filefd
prometheus_1     | level=info ts=2021-10-05T14:31:48.928Z caller=head.go:577 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=filesystem
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=hwmon
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=infiniband
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=ipvs
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=loadavg
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=mdadm
cadence_1        | + wait_for_db
cadence_1        | + '[' cassandra == mysql ']'
cadence_1        | + '[' cassandra == postgres ']'
cadence_1        | + '[' cassandra == scylla ']'
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=meminfo
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=netclass
prometheus_1     | level=info ts=2021-10-05T14:31:48.928Z caller=head.go:577 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2
prometheus_1     | level=info ts=2021-10-05T14:31:48.929Z caller=head.go:583 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=89.8µs wal_replay_duration=4.9123ms total_replay_duration=5.1209ms
cadence_1        | + wait_for_cassandra
prometheus_1     | level=info ts=2021-10-05T14:31:48.930Z caller=main.go:849 fs_type=EXT4_SUPER_MAGIC
prometheus_1     | level=info ts=2021-10-05T14:31:48.930Z caller=main.go:852 msg="TSDB started"
prometheus_1     | level=info ts=2021-10-05T14:31:48.930Z caller=main.go:979 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=netdev
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=netstat
cadence_1        | ++ echo cassandra
prometheus_1     | level=info ts=2021-10-05T14:31:48.936Z caller=main.go:1016 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=5.3829ms db_storage=3.8µs remote_storage=4.1µs web_handler=3.8µs query_engine=4.1µs scrape=1.8244ms scrape_sd=58.9µs notify=4.8µs notify_sd=5.1µs rules=4.2µs
prometheus_1     | level=info ts=2021-10-05T14:31:48.936Z caller=main.go:794 msg="Server is ready to receive web requests."
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=nfs
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=nfsd
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=nvme
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=powersupplyclass
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=pressure
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=rapl
cadence_1        | ++ awk -F , '{print $1}'
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=schedstat
node-exporter_1  | level=info ts=2021-10-05T14:31:48.641Z caller=node_exporter.go:115 collector=sockstat
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=softnet
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=stat
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=tapestats
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=textfile
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=thermal_zone
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=time
cadence_1        | + server=cassandra
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=timex
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=udp_queues
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=uname
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=vmstat
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=xfs
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:115 collector=zfs
node-exporter_1  | level=info ts=2021-10-05T14:31:48.642Z caller=node_exporter.go:199 msg="Listening on" address=:9100
node-exporter_1  | level=info ts=2021-10-05T14:31:48.643Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Starting DB migrations" logger=migrator
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=330 duration=3.6179ms
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Starting plugin search" logger=plugins
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | waiting for cassandra to start up
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Registering plugin" logger=plugins id=input
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
grafana_1        | t=2021-10-05T14:31:49+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
cassandra_1      | CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
cassandra_1      | CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
cassandra_1      | CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
cassandra_1      | CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
cassandra_1      | CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | waiting for cassandra to start up
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,018 YamlConfigurationLoader.java:92 - Configuration location: file:/etc/cassandra/cassandra.yaml
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,462 Config.java:539 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.19.0.3; broadcast_rpc_address=172.19.0.3; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@1e730495; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=true; enable_sasi_indexes=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=<REDACTED>; endpoint_snitch=SimpleSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.19.0.3; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=-2147483648; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; repair_session_max_tree_depth=18; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.19.0.3}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_on_duplicate_row_detection=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@7d3a22a9; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,463 DatabaseDescriptor.java:381 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,464 DatabaseDescriptor.java:439 - Global memtable on-heap threshold is enabled at 242MB
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,464 DatabaseDescriptor.java:443 - Global memtable off-heap threshold is enabled at 242MB
cadence-web_1    | cadence-web up and listening on port 8088
cassandra_1      | WARN  [main] 2021-10-05 14:31:51,675 DatabaseDescriptor.java:579 - Only 12.096GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,723 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,724 DatabaseDescriptor.java:775 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,899 JMXServerUtils.java:253 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,922 CassandraDaemon.java:490 - Hostname: 03773e5a156c
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,922 CassandraDaemon.java:497 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_302
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,923 CassandraDaemon.java:498 - Heap size: 971.250MiB/971.250MiB
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,923 CassandraDaemon.java:503 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4521792(4415K) committed = 4587520(4480K) max = 251658240(245760K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,924 CassandraDaemon.java:503 - Metaspace Non-heap memory: init = 0(0K) used = 19082896(18635K) committed = 19791872(19328K) max = -1(-1K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,924 CassandraDaemon.java:503 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2296168(2242K) committed = 2490368(2432K) max = 1073741824(1048576K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,924 CassandraDaemon.java:503 - Par Eden Space Heap memory: init = 208142336(203264K) used = 100235096(97885K) committed = 208142336(203264K) max = 208142336(203264K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,925 CassandraDaemon.java:503 - Par Survivor Space Heap memory: init = 25952256(25344K) used = 0(0K) committed = 25952256(25344K) max = 25952256(25344K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,925 CassandraDaemon.java:503 - CMS Old Gen Heap memory: init = 784334848(765952K) used = 0(0K) committed = 784334848(765952K) max = 784334848(765952K)
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,925 CassandraDaemon.java:505 - Classpath: /etc/cassandra:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.6.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-3.11.11.jar:/opt/cassandra/lib/apache-cassandra-thrift-3.11.11.jar:/opt/cassandra/lib/asm-5.0.4.jar:/opt/cassandra/lib/caffeine-2.2.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.1.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/cassandra/lib/disruptor-3.0.1.jar:/opt/cassandra/lib/ecj-4.4.2.jar:/opt/cassandra/lib/guava-18.0.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.5.4.jar:/opt/cassandra/lib/jackson-annotations-2.9.10.jar:/opt/cassandra/lib/jackson-core-2.9.10.jar:/opt/cassandra/lib/jackson-databind-2.9.10.8.jar:/opt/cassandra/lib/jamm-0.3.0.jar:/opt/cassandra/lib/javax.inject-1.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/cassandra/lib/jctools-core-1.2.1.jar:/opt/cassandra/lib/jflex-1.6.0.jar:/opt/cassandra/lib/jna-4.2.2.jar:/opt/cassandra/lib/joda-time-2.4.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/libthrift-0.9.2.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/cassandra/lib/logback-classic-1.1.3.jar:/opt/cassandra/lib/logback-core-1.1.3.jar:/opt/cassandra/lib/lz4-1.3.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/netty-all-4.0.44.Final.jar:/opt/cassandra/lib/ohc-core-0.4.4.jar:/opt/cassandra/lib/ohc-core-j8-0.4.4.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/slf4j-api-1.7.7.jar:/opt/cassandra/lib/snakeyaml-1.11.jar:/opt/cassandra/lib/snappy-java-1.1.1.7.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/thrift-server-0.3.7.jar:/opt/cassandra/lib/jsr223/*/*.jar::/opt/cassandra/lib/jamm-0.3.0.jar
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,925 CassandraDaemon.java:507 - JVM Arguments: [-Xloggc:/opt/cassandra/logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms995M, -Xmx995M, -Xmn248M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes]
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | waiting for cassandra to start up
cadence_1        | + sleep 1
cassandra_1      | WARN  [main] 2021-10-05 14:31:51,999 NativeLibrary.java:189 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
cassandra_1      | INFO  [main] 2021-10-05 14:31:51,999 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so
cassandra_1      | WARN  [main] 2021-10-05 14:31:52,000 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
cassandra_1      | INFO  [main] 2021-10-05 14:31:52,003 SigarLibrary.java:44 - Initializing SIGAR library
cassandra_1      | WARN  [main] 2021-10-05 14:31:52,023 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true 
cassandra_1      | WARN  [main] 2021-10-05 14:31:52,024 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 262144 is too low, recommended value: 1048575, you can change it with sysctl.
cassandra_1      | INFO  [main] 2021-10-05 14:31:52,228 QueryProcessor.java:116 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
cassandra_1      | INFO  [main] 2021-10-05 14:31:52,736 ColumnFamilyStore.java:432 - Initializing system.IndexInfo
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | waiting for cassandra to start up
cassandra_1      | INFO  [SSTableBatchOpen:1] 2021-10-05 14:31:53,300 BufferPool.java:234 - Global buffer pool is enabled, when pool is exhausted (max is 242.000MiB) it will allocate on heap
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,644 CacheService.java:100 - Initializing key cache with capacity of 48 MBs.
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,655 CacheService.java:122 - Initializing row cache with capacity of 0 MBs
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,659 CacheService.java:151 - Initializing counter cache with capacity of 24 MBs
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,660 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,738 ColumnFamilyStore.java:432 - Initializing system.batches
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,750 ColumnFamilyStore.java:432 - Initializing system.paxos
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,778 ColumnFamilyStore.java:432 - Initializing system.local
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,793 ColumnFamilyStore.java:432 - Initializing system.peers
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,800 ColumnFamilyStore.java:432 - Initializing system.peer_events
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,807 ColumnFamilyStore.java:432 - Initializing system.range_xfers
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,818 ColumnFamilyStore.java:432 - Initializing system.compaction_history
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,840 ColumnFamilyStore.java:432 - Initializing system.sstable_activity
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,861 ColumnFamilyStore.java:432 - Initializing system.size_estimates
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,875 ColumnFamilyStore.java:432 - Initializing system.available_ranges
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,882 ColumnFamilyStore.java:432 - Initializing system.transferred_ranges
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,889 ColumnFamilyStore.java:432 - Initializing system.views_builds_in_progress
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,900 ColumnFamilyStore.java:432 - Initializing system.built_views
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,907 ColumnFamilyStore.java:432 - Initializing system.hints
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,914 ColumnFamilyStore.java:432 - Initializing system.batchlog
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,923 ColumnFamilyStore.java:432 - Initializing system.prepared_statements
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,941 ColumnFamilyStore.java:432 - Initializing system.schema_keyspaces
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,949 ColumnFamilyStore.java:432 - Initializing system.schema_columnfamilies
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,957 ColumnFamilyStore.java:432 - Initializing system.schema_columns
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,964 ColumnFamilyStore.java:432 - Initializing system.schema_triggers
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,974 ColumnFamilyStore.java:432 - Initializing system.schema_usertypes
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,983 ColumnFamilyStore.java:432 - Initializing system.schema_functions
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,991 ColumnFamilyStore.java:432 - Initializing system.schema_aggregates
cassandra_1      | INFO  [main] 2021-10-05 14:31:53,997 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
prometheus_1     | level=info ts=2021-10-05T14:31:54.022Z caller=compact.go:518 component=tsdb msg="write block" mint=1633362468215 maxt=1633363200000 ulid=01FH8DWEW47CZ64DZG3XWBDX5Q duration=33.7676ms
prometheus_1     | level=info ts=2021-10-05T14:31:54.024Z caller=head.go:785 component=tsdb msg="Head GC completed" duration=739.6µs
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,188 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,252 ColumnFamilyStore.java:432 - Initializing system_schema.keyspaces
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,267 ColumnFamilyStore.java:432 - Initializing system_schema.tables
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep waiting for cassandra to start up
cadence_1        | 1
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,283 ColumnFamilyStore.java:432 - Initializing system_schema.columns
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,297 ColumnFamilyStore.java:432 - Initializing system_schema.triggers
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,310 ColumnFamilyStore.java:432 - Initializing system_schema.dropped_columns
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,322 ColumnFamilyStore.java:432 - Initializing system_schema.views
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,337 ColumnFamilyStore.java:432 - Initializing system_schema.types
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,360 ColumnFamilyStore.java:432 - Initializing system_schema.functions
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,373 ColumnFamilyStore.java:432 - Initializing system_schema.aggregates
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,387 ColumnFamilyStore.java:432 - Initializing system_schema.indexes
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,400 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,725 StorageService.java:681 - Populating token metadata from system tables
cassandra_1      | INFO  [main] 2021-10-05 14:31:54,745 StorageService.java:688 - Token metadata: Normal Tokens:
cassandra_1      | /172.19.0.3:[-9025747668169669436, -8976547030515574221, -8931530688828161457, -8725018472440959531, -8714693082944678685, -8708130345547140030, -8671520067618995500, -8671085242682906489, -8533031138214360887, -8527979837841530526, -8520762099594448941, -8512716553390702643, -8481681762843977605, -8294287226884123698, -8265616245191894253, -8258024198086110142, -8163558331631520728, -8083603489573596689, -7940909043210147910, -7927049140945705282, -7905427074042272162, -7857546695177177147, -7807247663183746638, -7788315777952321641, -7749190600706844704, -7639442077415690077, -7594896555154273901, -7588073924527269061, -7547561865247748379, -7357044428057369581, -7141037397624867350, -7061870748068840503, -7016836923649426509, -6990210165790372285, -6968573642360941379, -6949387225471466007, -6913294961547232873, -6911754417327631517, -6897068103376611366, -6642120145988415450, -6602356186368413965, -6483152447252162856, -6433683032474167143, -6319629534071795505, -6224252255599246372, -6170063711210749860, -6154904591166704536, -6121741707245821909, -6108623634024109744, -6066127284838847761, -6009905777875667645, -5992727760224021794, -5933124005159839671, -5888382376228995666, -5846801751397747150, -5789578156351730406, -5692327962282945613, -5649563959855917160, -5627199751928584507, -5555922861725237469, -5443893630664859430, -5423769288055512629, -5413705458970224778, -5238935209319905347, -5211443026642835882, -5177309836511892201, -5154222109343312018, -5090940885101319155, -5081157238585099994, -5066829257895643307, -5058293312581703478, -5051666070035008103, -4981755795965358828, -4959906218021546735, -4956838566170462713, -4906286838163421355, -4876472521921666402, -4825416515848767083, -4792457453558589465, -4697775164842700661, -4689526928508440319, -4616630903692438637, -4439972446541890546, -4435521294638839872, -4357998211199364467, -4299844818643854207, -4225824989139694640, -4200736852030156977, -4171049510936454854, -4144448459612371140, -4108464356374912252, -3987643888809691101, -3959424457093676393, -3837852489788784789, -3758453113954541395, -3726630628613200179, -3690265278086395705, -3683430810504430794, -3592253734054549185, -3525709212499961136, -3313294486218215140, -3242789513235063206, -3224069880239558467, -3016202338445350509, -2774219432178352696, -2672595018118239153, -2529520366848579606, -2508586107526922605, -2463320137143441406, -2351817051546280460, -2345351774970896800, -2231017963796240040, -2181468083967110513, -2169335546573333565, -2143266389000204104, -2081428540348882305, -1998396971850480489, -1985680694858396453, -1878634331558881560, -1869872641473398244, -1864646457514305028, -1776777366696970680, -1727227844421855391, -1721690868225446275, -1690266384943340512, -1561247653868683622, -1543070376527616917, -1503136684594131331, -1484870394115791348, -1470983523715751846, -1426004203173309411, -1417447568390704287, -1201229865081155911, -1174811694035851617, -946931641976630166, -830536483182725812, -824353867736328119, -802232180665794236, -764092988382190674, -532322402815698140, -525460887038033188, -436029641232445875, -362635851513950714, -156757182973576083, -50833407499340583, -40138858623949479, 23185460867930612, 34542634332754307, 92509576253055246, 112482334096820255, 115547657550119484, 527303364805148637, 648593993952232732, 649209047872427758, 733154852152382779, 830878784685594225, 926575780505258701, 977091535487262928, 1050541928696384493, 1118230132522840932, 1225448265434917688, 1458850811486785783, 1472073796103717170, 1489751320857709640, 1580181750081918503, 1669047976074844876, 1813226772367605548, 1817396314596226583, 1850049939742503639, 1901853460923428914, 1974453122004749960, 2026606097803832701, 2039175911400657593, 2134115197433333435, 2192954407049890301, 2199119981099175811, 2265147406794470679, 2411644769677803260, 2425756736060254585, 2541587559864086585, 2700015385632438776, 2712912879134109718, 2722294909313700302, 2726071145878806086, 2791368817440161079, 2799186721469031749, 2829929334656763910, 2847420638808041918, 2983159825354997166, 3046168441934264774, 3171595426165082156, 3176862225437470202, 3350205629869835919, 3377617331454958078, 3449789872309067605, 3489221076961511447, 3578597977771649327, 3673993421894510398, 3684603571999009483, 3891586941932539629, 3919448989567522495, 3964918550921156482, 4010859926985455805, 4043743461419303189, 4050626547944959006, 4123988295498578293, 4297693896694090035, 4395371632215340376, 4459719388478850126, 4466736214026586423, 4472315653536873756, 4483570641102858943, 4505155577191611031, 4580419064019476811, 4639749004362417260, 4725672894304261509, 4803897651207738594, 4812216988317545093, 4846858441791822467, 5058211295890425724, 5058287211513534918, 5112335886476209898, 5490779292215666774, 5543740298833034228, 5618254726325829675, 5808378579338901726, 5873503782620926004, 5966303288233031604, 5971000226683121591, 6119429603984499671, 6161259687648574239, 6193743004381794000, 6282852340544732328, 6413409785783597608, 6450249923549778178, 6556502580154761384, 6572627429797967874, 6616824592620686733, 7104841181040924612, 7278504021979574636, 7606570501637896179, 7725871281286968589, 7832598594308490528, 7882417505097548996, 8040509707313676094, 8177201087003782926, 8221556994127607962, 8255985416766895863, 8589857201083063819, 8603924386152220942, 8608675435876387649, 8684042848934742790, 8938877056425644278, 8975425137617402612, 8984671111685867268, 9118719478441076632]
cassandra_1      | 
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,027 ColumnFamilyStore.java:432 - Initializing system_distributed.parent_repair_history
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,066 ColumnFamilyStore.java:432 - Initializing system_distributed.repair_history
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,081 ColumnFamilyStore.java:432 - Initializing system_distributed.view_build_status
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,092 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,095 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,101 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions.closed_by_close_time
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,145 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions.closed_by_status
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,167 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions.closed_by_type
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,193 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions.closed_by_workflow_id
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,216 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions_v2
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,220 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions_v2.closed_by_close_time_v2
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,248 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions_v2.closed_by_status_v2
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,269 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions_v2.closed_by_type_v2
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,305 ColumnFamilyStore.java:432 - Initializing cadence_visibility.closed_executions_v2.closed_by_workflow_id_v2
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,329 ColumnFamilyStore.java:432 - Initializing cadence_visibility.open_executions
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,332 ColumnFamilyStore.java:432 - Initializing cadence_visibility.open_executions.open_by_type
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,359 ColumnFamilyStore.java:432 - Initializing cadence_visibility.open_executions.open_by_workflow_id
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,382 ColumnFamilyStore.java:432 - Initializing cadence_visibility.schema_update_history
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,411 ColumnFamilyStore.java:432 - Initializing cadence_visibility.schema_version
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,423 ViewManager.java:137 - Not submitting build tasks for views in keyspace cadence_visibility as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,426 ColumnFamilyStore.java:432 - Initializing system_auth.resource_role_permissons_index
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,433 ColumnFamilyStore.java:432 - Initializing system_auth.role_members
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | waiting for cassandra to start up
cadence_1        | + sleep 1
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,440 ColumnFamilyStore.java:432 - Initializing system_auth.role_permissions
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,446 ColumnFamilyStore.java:432 - Initializing system_auth.roles
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,456 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,460 ColumnFamilyStore.java:432 - Initializing system_traces.events
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,471 ColumnFamilyStore.java:432 - Initializing system_traces.sessions
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,481 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,483 ColumnFamilyStore.java:432 - Initializing cadence.cluster_config
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,488 ColumnFamilyStore.java:432 - Initializing cadence.domains
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,498 ColumnFamilyStore.java:432 - Initializing cadence.domains_by_name_v2
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,508 ColumnFamilyStore.java:432 - Initializing cadence.events
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,513 ColumnFamilyStore.java:432 - Initializing cadence.executions
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,529 ColumnFamilyStore.java:432 - Initializing cadence.history_node
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,535 ColumnFamilyStore.java:432 - Initializing cadence.history_tree
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,539 ColumnFamilyStore.java:432 - Initializing cadence.queue
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,545 ColumnFamilyStore.java:432 - Initializing cadence.queue_metadata
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,556 ColumnFamilyStore.java:432 - Initializing cadence.schema_update_history
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,570 ColumnFamilyStore.java:432 - Initializing cadence.schema_version
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,583 ColumnFamilyStore.java:432 - Initializing cadence.tasks
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,588 ViewManager.java:137 - Not submitting build tasks for views in keyspace cadence as storage service is not initialized
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,590 AutoSavingCache.java:174 - Completed loading (1 ms; 26 keys) KeyCache cache
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,609 CommitLog.java:147 - Replaying /opt/cassandra/data/commitlog/CommitLog-6-1633362784120.log
cassandra_1      | WARN  [main] 2021-10-05 14:31:55,626 CommitLogReplayer.java:253 - Origin of 10 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
cassandra_1      | WARN  [main] 2021-10-05 14:31:55,626 CommitLogReplayer.java:253 - Origin of 2 sstables is unknown or doesn't match the local node; commitLogIntervals for them were ignored
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,642 CommitLog.java:149 - Log replay complete, 0 replayed mutations
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,642 StorageService.java:681 - Populating token metadata from system tables
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,651 StorageService.java:688 - Token metadata: Normal Tokens:
cassandra_1      | /172.19.0.3:[-9025747668169669436, -8976547030515574221, -8931530688828161457, -8725018472440959531, -8714693082944678685, -8708130345547140030, -8671520067618995500, -8671085242682906489, -8533031138214360887, -8527979837841530526, -8520762099594448941, -8512716553390702643, -8481681762843977605, -8294287226884123698, -8265616245191894253, -8258024198086110142, -8163558331631520728, -8083603489573596689, -7940909043210147910, -7927049140945705282, -7905427074042272162, -7857546695177177147, -7807247663183746638, -7788315777952321641, -7749190600706844704, -7639442077415690077, -7594896555154273901, -7588073924527269061, -7547561865247748379, -7357044428057369581, -7141037397624867350, -7061870748068840503, -7016836923649426509, -6990210165790372285, -6968573642360941379, -6949387225471466007, -6913294961547232873, -6911754417327631517, -6897068103376611366, -6642120145988415450, -6602356186368413965, -6483152447252162856, -6433683032474167143, -6319629534071795505, -6224252255599246372, -6170063711210749860, -6154904591166704536, -6121741707245821909, -6108623634024109744, -6066127284838847761, -6009905777875667645, -5992727760224021794, -5933124005159839671, -5888382376228995666, -5846801751397747150, -5789578156351730406, -5692327962282945613, -5649563959855917160, -5627199751928584507, -5555922861725237469, -5443893630664859430, -5423769288055512629, -5413705458970224778, -5238935209319905347, -5211443026642835882, -5177309836511892201, -5154222109343312018, -5090940885101319155, -5081157238585099994, -5066829257895643307, -5058293312581703478, -5051666070035008103, -4981755795965358828, -4959906218021546735, -4956838566170462713, -4906286838163421355, -4876472521921666402, -4825416515848767083, -4792457453558589465, -4697775164842700661, -4689526928508440319, -4616630903692438637, -4439972446541890546, -4435521294638839872, -4357998211199364467, -4299844818643854207, -4225824989139694640, -4200736852030156977, -4171049510936454854, -4144448459612371140, -4108464356374912252, -3987643888809691101, -3959424457093676393, -3837852489788784789, -3758453113954541395, -3726630628613200179, -3690265278086395705, -3683430810504430794, -3592253734054549185, -3525709212499961136, -3313294486218215140, -3242789513235063206, -3224069880239558467, -3016202338445350509, -2774219432178352696, -2672595018118239153, -2529520366848579606, -2508586107526922605, -2463320137143441406, -2351817051546280460, -2345351774970896800, -2231017963796240040, -2181468083967110513, -2169335546573333565, -2143266389000204104, -2081428540348882305, -1998396971850480489, -1985680694858396453, -1878634331558881560, -1869872641473398244, -1864646457514305028, -1776777366696970680, -1727227844421855391, -1721690868225446275, -1690266384943340512, -1561247653868683622, -1543070376527616917, -1503136684594131331, -1484870394115791348, -1470983523715751846, -1426004203173309411, -1417447568390704287, -1201229865081155911, -1174811694035851617, -946931641976630166, -830536483182725812, -824353867736328119, -802232180665794236, -764092988382190674, -532322402815698140, -525460887038033188, -436029641232445875, -362635851513950714, -156757182973576083, -50833407499340583, -40138858623949479, 23185460867930612, 34542634332754307, 92509576253055246, 112482334096820255, 115547657550119484, 527303364805148637, 648593993952232732, 649209047872427758, 733154852152382779, 830878784685594225, 926575780505258701, 977091535487262928, 1050541928696384493, 1118230132522840932, 1225448265434917688, 1458850811486785783, 1472073796103717170, 1489751320857709640, 1580181750081918503, 1669047976074844876, 1813226772367605548, 1817396314596226583, 1850049939742503639, 1901853460923428914, 1974453122004749960, 2026606097803832701, 2039175911400657593, 2134115197433333435, 2192954407049890301, 2199119981099175811, 2265147406794470679, 2411644769677803260, 2425756736060254585, 2541587559864086585, 2700015385632438776, 2712912879134109718, 2722294909313700302, 2726071145878806086, 2791368817440161079, 2799186721469031749, 2829929334656763910, 2847420638808041918, 2983159825354997166, 3046168441934264774, 3171595426165082156, 3176862225437470202, 3350205629869835919, 3377617331454958078, 3449789872309067605, 3489221076961511447, 3578597977771649327, 3673993421894510398, 3684603571999009483, 3891586941932539629, 3919448989567522495, 3964918550921156482, 4010859926985455805, 4043743461419303189, 4050626547944959006, 4123988295498578293, 4297693896694090035, 4395371632215340376, 4459719388478850126, 4466736214026586423, 4472315653536873756, 4483570641102858943, 4505155577191611031, 4580419064019476811, 4639749004362417260, 4725672894304261509, 4803897651207738594, 4812216988317545093, 4846858441791822467, 5058211295890425724, 5058287211513534918, 5112335886476209898, 5490779292215666774, 5543740298833034228, 5618254726325829675, 5808378579338901726, 5873503782620926004, 5966303288233031604, 5971000226683121591, 6119429603984499671, 6161259687648574239, 6193743004381794000, 6282852340544732328, 6413409785783597608, 6450249923549778178, 6556502580154761384, 6572627429797967874, 6616824592620686733, 7104841181040924612, 7278504021979574636, 7606570501637896179, 7725871281286968589, 7832598594308490528, 7882417505097548996, 8040509707313676094, 8177201087003782926, 8221556994127607962, 8255985416766895863, 8589857201083063819, 8603924386152220942, 8608675435876387649, 8684042848934742790, 8938877056425644278, 8975425137617402612, 8984671111685867268, 9118719478441076632]
cassandra_1      | 
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,965 QueryProcessor.java:163 - Preloaded 116 prepared statements
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,966 StorageService.java:699 - Cassandra version: 3.11.11
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,966 StorageService.java:700 - Thrift API version: 20.1.0
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,966 StorageService.java:701 - CQL supported versions: 3.4.4 (default: 3.4.4)
cassandra_1      | INFO  [main] 2021-10-05 14:31:55,966 StorageService.java:703 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,012 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 48 MB and a resize interval of 60 minutes
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,033 MessagingService.java:750 - Starting Messaging Service on /172.19.0.3:7000 (eth0)
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,057 StorageService.java:631 - Unable to gossip with any peers but continuing anyway since node is in its own seed list
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,077 StorageService.java:785 - Loading persisted ring state
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,078 StorageService.java:916 - Starting up server gossip
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,135 TokenMetadata.java:500 - Updating topology for /172.19.0.3
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,136 TokenMetadata.java:500 - Updating topology for /172.19.0.3
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,172 StorageService.java:1104 - Using saved tokens [-1174811694035851617, -1201229865081155911, -1417447568390704287, -1426004203173309411, -1470983523715751846, -1484870394115791348, -1503136684594131331, -1543070376527616917, -1561247653868683622, -156757182973576083, -1690266384943340512, -1721690868225446275, -1727227844421855391, -1776777366696970680, -1864646457514305028, -1869872641473398244, -1878634331558881560, -1985680694858396453, -1998396971850480489, -2081428540348882305, -2143266389000204104, -2169335546573333565, -2181468083967110513, -2231017963796240040, -2345351774970896800, -2351817051546280460, -2463320137143441406, -2508586107526922605, -2529520366848579606, -2672595018118239153, -2774219432178352696, -3016202338445350509, -3224069880239558467, -3242789513235063206, -3313294486218215140, -3525709212499961136, -3592253734054549185, -362635851513950714, -3683430810504430794, -3690265278086395705, -3726630628613200179, -3758453113954541395, -3837852489788784789, -3959424457093676393, -3987643888809691101, -40138858623949479, -4108464356374912252, -4144448459612371140, -4171049510936454854, -4200736852030156977, -4225824989139694640, -4299844818643854207, -4357998211199364467, -436029641232445875, -4435521294638839872, -4439972446541890546, -4616630903692438637, -4689526928508440319, -4697775164842700661, -4792457453558589465, -4825416515848767083, -4876472521921666402, -4906286838163421355, -4956838566170462713, -4959906218021546735, -4981755795965358828, -5051666070035008103, -5058293312581703478, -5066829257895643307, -5081157238585099994, -50833407499340583, -5090940885101319155, -5154222109343312018, -5177309836511892201, -5211443026642835882, -5238935209319905347, -525460887038033188, -532322402815698140, -5413705458970224778, -5423769288055512629, -5443893630664859430, -5555922861725237469, -5627199751928584507, -5649563959855917160, -5692327962282945613, -5789578156351730406, -5846801751397747150, -5888382376228995666, -5933124005159839671, -5992727760224021794, -6009905777875667645, -6066127284838847761, -6108623634024109744, -6121741707245821909, -6154904591166704536, -6170063711210749860, -6224252255599246372, -6319629534071795505, -6433683032474167143, -6483152447252162856, -6602356186368413965, -6642120145988415450, -6897068103376611366, -6911754417327631517, -6913294961547232873, -6949387225471466007, -6968573642360941379, -6990210165790372285, -7016836923649426509, -7061870748068840503, -7141037397624867350, -7357044428057369581, -7547561865247748379, -7588073924527269061, -7594896555154273901, -7639442077415690077, -764092988382190674, -7749190600706844704, -7788315777952321641, -7807247663183746638, -7857546695177177147, -7905427074042272162, -7927049140945705282, -7940909043210147910, -802232180665794236, -8083603489573596689, -8163558331631520728, -824353867736328119, -8258024198086110142, -8265616245191894253, -8294287226884123698, -830536483182725812, -8481681762843977605, -8512716553390702643, -8520762099594448941, -8527979837841530526, -8533031138214360887, -8671085242682906489, -8671520067618995500, -8708130345547140030, -8714693082944678685, -8725018472440959531, -8931530688828161457, -8976547030515574221, -9025747668169669436, -946931641976630166, 1050541928696384493, 1118230132522840932, 112482334096820255, 115547657550119484, 1225448265434917688, 1458850811486785783, 1472073796103717170, 1489751320857709640, 1580181750081918503, 1669047976074844876, 1813226772367605548, 1817396314596226583, 1850049939742503639, 1901853460923428914, 1974453122004749960, 2026606097803832701, 2039175911400657593, 2134115197433333435, 2192954407049890301, 2199119981099175811, 2265147406794470679, 23185460867930612, 2411644769677803260, 2425756736060254585, 2541587559864086585, 2700015385632438776, 2712912879134109718, 2722294909313700302, 2726071145878806086, 2791368817440161079, 2799186721469031749, 2829929334656763910, 2847420638808041918, 2983159825354997166, 3046168441934264774, 3171595426165082156, 3176862225437470202, 3350205629869835919, 3377617331454958078, 3449789872309067605, 34542634332754307, 3489221076961511447, 3578597977771649327, 3673993421894510398, 3684603571999009483, 3891586941932539629, 3919448989567522495, 3964918550921156482, 4010859926985455805, 4043743461419303189, 4050626547944959006, 4123988295498578293, 4297693896694090035, 4395371632215340376, 4459719388478850126, 4466736214026586423, 4472315653536873756, 4483570641102858943, 4505155577191611031, 4580419064019476811, 4639749004362417260, 4725672894304261509, 4803897651207738594, 4812216988317545093, 4846858441791822467, 5058211295890425724, 5058287211513534918, 5112335886476209898, 527303364805148637, 5490779292215666774, 5543740298833034228, 5618254726325829675, 5808378579338901726, 5873503782620926004, 5966303288233031604, 5971000226683121591, 6119429603984499671, 6161259687648574239, 6193743004381794000, 6282852340544732328, 6413409785783597608, 6450249923549778178, 648593993952232732, 649209047872427758, 6556502580154761384, 6572627429797967874, 6616824592620686733, 7104841181040924612, 7278504021979574636, 733154852152382779, 7606570501637896179, 7725871281286968589, 7832598594308490528, 7882417505097548996, 8040509707313676094, 8177201087003782926, 8221556994127607962, 8255985416766895863, 830878784685594225, 8589857201083063819, 8603924386152220942, 8608675435876387649, 8684042848934742790, 8938877056425644278, 8975425137617402612, 8984671111685867268, 9118719478441076632, 92509576253055246, 926575780505258701, 977091535487262928]
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,184 StorageService.java:1564 - JOINING: Finish joining ring
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,207 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence_visibility', ColumnFamily='schema_version')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,208 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence_visibility', ColumnFamily='closed_executions_v2')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,209 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence_visibility', ColumnFamily='closed_executions')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,209 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence_visibility', ColumnFamily='schema_update_history')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,209 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence_visibility', ColumnFamily='open_executions')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,210 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='schema_version')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,210 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='queue_metadata')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,211 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='queue')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,211 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='history_tree')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,211 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='executions')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,212 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='schema_update_history')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,212 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='history_node')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,212 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='events')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,213 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='cluster_config')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,213 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='domains_by_name_v2')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,213 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='domains')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,214 SecondaryIndexManager.java:512 - Executing pre-join tasks for: CFS(Keyspace='cadence', ColumnFamily='tasks')
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,249 StorageService.java:2480 - Node /172.19.0.3 state jump to NORMAL
cassandra_1      | INFO  [main] 2021-10-05 14:31:56,257 Gossiper.java:1832 - Waiting for gossip to settle...
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | waiting for cassandra to start up
cadence_1        | + sleep 1
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | waiting for cassandra to start up
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1waiting for cassandra to start up
cadence_1        | 
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | waiting for cassandra to start up
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | waiting for cassandra to start up
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | waiting for cassandra to start up
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cadence_1        | waiting for cassandra to start up
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | + sleep 1
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,262 Gossiper.java:1863 - No gossip backlog; proceeding
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | Connection error: ('Unable to connect to any servers', {'172.19.0.3:9042': error(111, "Tried connecting to [('172.19.0.3', 9042)]. Last error: Connection refused")})
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,487 NativeTransportService.java:68 - Netty using native Epoll event loop
cadence_1        | + echo 'waiting for cassandra to start up'
cadence_1        | waiting for cassandra to start up
cadence_1        | + sleep 1
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,566 Server.java:158 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,566 Server.java:159 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,621 CassandraDaemon.java:564 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
cassandra_1      | INFO  [main] 2021-10-05 14:32:04,621 CassandraDaemon.java:650 - Startup complete
cadence_1        | + cqlsh -u cassandra -p cassandra --cqlversion=3.4.4 --protocol-version=4 cassandra
cadence_1        | + echo 'cassandra started'
cadence_1        | cassandra started
cadence_1        | + '[' false == true ']'
cadence_1        | + '[' '' '!=' true ']'
cadence_1        | + setup_schema
cadence_1        | + '[' cassandra == mysql ']'
cadence_1        | + '[' cassandra == postgres ']'
cadence_1        | + echo 'setup cassandra schema'
cadence_1        | + setup_cassandra_schema
cadence_1        | setup cassandra schema
cadence_1        | + SCHEMA_DIR=/etc/cadence/schema/cassandra/cadence/versioned
cadence_1        | + cadence-cassandra-tool --ep cassandra create -k cadence --rf 1
cadence_1        | + cadence-cassandra-tool --ep cassandra -k cadence setup-schema -v 0.0
cadence_1        | 2021/10/05 14:32:06 Starting schema setup, config=&{SchemaFilePath: InitialVersion:0.0 Overwrite:false DisableVersioning:false}
cadence_1        | 2021/10/05 14:32:06 Setting up version tables
cadence_1        | 2021/10/05 14:32:06 Cannot add already existing table "schema_version" to keyspace "cadence"
cadence_1        | + cadence-cassandra-tool --ep cassandra -k cadence update-schema -d /etc/cadence/schema/cassandra/cadence/versioned
cadence_1        | 2021/10/05 14:32:06 UpdateSchemeTask started, config=&{DBName: TargetVersion: SchemaDir:/etc/cadence/schema/cassandra/cadence/versioned IsDryRun:false}
cadence_1        | 2021/10/05 14:32:06 found zero updates from current version 0.32
cadence_1        | 2021/10/05 14:32:06 UpdateSchemeTask done
cadence_1        | + VISIBILITY_SCHEMA_DIR=/etc/cadence/schema/cassandra/visibility/versioned
cadence_1        | + cadence-cassandra-tool --ep cassandra create -k cadence_visibility --rf 1
cadence_1        | + cadence-cassandra-tool --ep cassandra -k cadence_visibility setup-schema -v 0.0
cadence_1        | 2021/10/05 14:32:06 Starting schema setup, config=&{SchemaFilePath: InitialVersion:0.0 Overwrite:false DisableVersioning:false}
cadence_1        | 2021/10/05 14:32:06 Setting up version tables
cadence_1        | 2021/10/05 14:32:06 Cannot add already existing table "schema_version" to keyspace "cadence_visibility"
cadence_1        | + cadence-cassandra-tool --ep cassandra -k cadence_visibility update-schema -d /etc/cadence/schema/cassandra/visibility/versioned
cadence_1        | 2021/10/05 14:32:06 UpdateSchemeTask started, config=&{DBName: TargetVersion: SchemaDir:/etc/cadence/schema/cassandra/visibility/versioned IsDryRun:false}
cadence_1        | 2021/10/05 14:32:06 found zero updates from current version 0.6
cadence_1        | 2021/10/05 14:32:06 UpdateSchemeTask done
cadence_1        | + '[' false == true ']'
cadence_1        | + exec /start-cadence.sh
cadence_1        | + CONFIG_TEMPLATE_PATH=/etc/cadence/config/config_template.yaml
cadence_1        | + dockerize -template /etc/cadence/config/config_template.yaml:/etc/cadence/config/docker.yaml
cadence_1        | + exec cadence-server --root /etc/cadence --env docker start --services=history,matching,frontend,worker
cadence_1        | 2021/10/05 14:32:06 Loading config; env=docker,zone=,configDir=/etc/cadence/config
cadence_1        | 2021/10/05 14:32:06 Loading configFiles=[/etc/cadence/config/docker.yaml]
cadence_1        | 2021/10/05 14:32:06 Trying to initialize File Based Dynamic Config Client
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.839Z","msg":"Updated dynamic config","service":"cadence-history","logging-call-at":"file_based_client.go:267"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.841Z","msg":"Listening for TChannel requests","service":"cadence-history","address":"172.19.0.6:7934","logging-call-at":"factory.go:67"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.841Z","msg":"Listening for GRPC requests","service":"cadence-history","address":"172.19.0.6:7834","logging-call-at":"factory.go:83"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.843Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.advancedVisibilityWritingMode,clusterName:primary","value":"off","default-value":"off","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:06 failed to create file blobstore client, will continue startup without it: output directory not given for file blobstore
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.843Z","msg":"Starting service history","service":"cadence-history","logging-call-at":"server.go:235"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.844Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.enableDebugMode,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.844Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.historyMgrNumConns,clusterName:primary","value":"50","default-value":"50","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.844Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.throttledLogRPS,clusterName:primary","value":"4","default-value":"4","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.845Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.enableGRPCOutbound,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.845Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.grpcMaxSizeInByte,clusterName:primary","value":"4194304","default-value":"4194304","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.845Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.845Z","msg":"Creating GRPC dispatcher outbound","service":"cadence-history","address":"cadence:7833","logging-call-at":"dispatcher_provider.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.847Z","msg":"Add new peers by DNS lookup","service":"cadence-history","address":"cadence","addresses":"[172.19.0.6:7833]","logging-call-at":"dns_updater.go:80"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.847Z","msg":"First loading dynamic config","service":"cadence-history","key":"admin.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.847Z","msg":"First loading dynamic config","service":"cadence-history","key":"frontend.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.847Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.persistenceGlobalMaxQPS,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.847Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.persistenceMaxQPS,clusterName:primary","value":"9000","default-value":"9000","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.885Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.persistenceErrorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"warn","ts":"2021-10-05T14:32:06.921Z","msg":"missing visibility and EnableReadFromClosedExecutionV2 config","service":"cadence-history","value":"&{0x105f980 0x105f980 0x105f980 <nil> 0x1060fb0 0x1060d20 <nil> 0x105fb10 0x105fb10 <nil> <nil> <nil> <nil>}","logging-call-at":"factory.go:332"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:06.968Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.enableVisibilitySampling,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.118Z","msg":"First loading dynamic config","service":"cadence-history","key":"matching.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.118Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheGlobalInitialSize,clusterName:primary","value":"4096","default-value":"4096","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.118Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheGlobalMaxSize,clusterName:primary","value":"131072","default-value":"131072","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.118Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheTTL,clusterName:primary","value":"1h0m0s","default-value":"1h0m0s","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.118Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheMaxSizeInBytes,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 Trying to initialize File Based Dynamic Config Client
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.119Z","msg":"Updated dynamic config","service":"cadence-matching","logging-call-at":"file_based_client.go:267"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.119Z","msg":"Listening for TChannel requests","service":"cadence-matching","address":"172.19.0.6:7935","logging-call-at":"factory.go:67"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.119Z","msg":"Listening for GRPC requests","service":"cadence-matching","address":"172.19.0.6:7835","logging-call-at":"factory.go:83"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.120Z","msg":"elastic search config","service":"cadence-history","es-config":"<nil>","logging-call-at":"service.go:88"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.120Z","msg":"history starting","service":"cadence-history","logging-call-at":"service.go:89"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.121Z","msg":"First loading dynamic config","service":"cadence-matching","key":"system.advancedVisibilityWritingMode,clusterName:primary","value":"off","default-value":"off","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 failed to create file blobstore client, will continue startup without it: output directory not given for file blobstore
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.121Z","msg":"Starting service matching","service":"cadence-matching","logging-call-at":"server.go:235"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.121Z","msg":"First loading dynamic config","service":"cadence-matching","key":"system.enableDebugMode,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.121Z","msg":"First loading dynamic config","service":"cadence-matching","key":"matching.throttledLogRPS,clusterName:primary","value":"20","default-value":"20","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.121Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.rps,clusterName:primary","value":"3000","default-value":"3000","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"First loading dynamic config","service":"cadence-matching","key":"system.enableGRPCOutbound,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"First loading dynamic config","service":"cadence-matching","key":"system.grpcMaxSizeInByte,clusterName:primary","value":"4194304","default-value":"4194304","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"First loading dynamic config","service":"cadence-matching","key":"history.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"Creating GRPC dispatcher outbound","service":"cadence-matching","address":"cadence:7833","logging-call-at":"dispatcher_provider.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"First loading dynamic config","service":"cadence-matching","key":"admin.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.122Z","msg":"First loading dynamic config","service":"cadence-matching","key":"frontend.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.123Z","msg":"First loading dynamic config","service":"cadence-matching","key":"matching.persistenceGlobalMaxQPS,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.123Z","msg":"First loading dynamic config","service":"cadence-matching","key":"matching.persistenceMaxQPS,clusterName:primary","value":"3000","default-value":"3000","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.123Z","msg":"Add new peers by DNS lookup","service":"cadence-matching","address":"cadence","addresses":"[172.19.0.6:7833]","logging-call-at":"dns_updater.go:80"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.127Z","msg":"RuntimeMetricsReporter started","service":"cadence-history","logging-call-at":"runtime.go:169"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.127Z","msg":"PProf not started due to port not set","service":"cadence-history","logging-call-at":"pprof.go:64"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.162Z","msg":"First loading dynamic config","service":"cadence-matching","key":"system.persistenceErrorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.342Z","msg":"First loading dynamic config","service":"cadence-matching","key":"matching.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 Trying to initialize File Based Dynamic Config Client
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.342Z","msg":"Updated dynamic config","service":"cadence-frontend","logging-call-at":"file_based_client.go:267"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.342Z","msg":"matching starting","service":"cadence-matching","logging-call-at":"service.go:85"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.343Z","msg":"Listening for TChannel requests","service":"cadence-frontend","address":"172.19.0.6:7933","logging-call-at":"factory.go:67"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.343Z","msg":"Listening for GRPC requests","service":"cadence-frontend","address":"172.19.0.6:7833","logging-call-at":"factory.go:83"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.344Z","msg":"First loading dynamic config","service":"cadence-matching","key":"matching.rps,clusterName:primary","value":"1200","default-value":"1200","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.345Z","msg":"RuntimeMetricsReporter started","service":"cadence-matching","logging-call-at":"runtime.go:169"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.345Z","msg":"PProf not started due to port not set","service":"cadence-matching","logging-call-at":"pprof.go:64"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.346Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.advancedVisibilityWritingMode,clusterName:primary","value":"off","default-value":"off","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 failed to create file blobstore client, will continue startup without it: output directory not given for file blobstore
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.347Z","msg":"Starting service frontend","service":"cadence-frontend","logging-call-at":"server.go:235"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.347Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.historyMgrNumConns,clusterName:primary","value":"10","default-value":"10","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.347Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.throttledLogRPS,clusterName:primary","value":"20","default-value":"20","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.348Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.enableGRPCOutbound,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.348Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.grpcMaxSizeInByte,clusterName:primary","value":"4194304","default-value":"4194304","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.348Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"history.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.348Z","msg":"Creating GRPC dispatcher outbound","service":"cadence-frontend","address":"cadence:7833","logging-call-at":"dispatcher_provider.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.348Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"admin.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.349Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.349Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.persistenceGlobalMaxQPS,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.349Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.persistenceMaxQPS,clusterName:primary","value":"2000","default-value":"2000","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.349Z","msg":"Add new peers by DNS lookup","service":"cadence-frontend","address":"cadence","addresses":"[172.19.0.6:7833]","logging-call-at":"dns_updater.go:80"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.386Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.persistenceErrorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.419Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.enableReadFromClosedExecutionV2,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.469Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.enableVisibilitySampling,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.612Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"matching.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 Trying to initialize File Based Dynamic Config Client
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.612Z","msg":"frontend starting","service":"cadence-frontend","logging-call-at":"service.go:250"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.612Z","msg":"Updated dynamic config","service":"cadence-worker","logging-call-at":"file_based_client.go:267"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.613Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.enableGlobalDomain,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.613Z","msg":"Listening for TChannel requests","service":"cadence-worker","address":"172.19.0.6:7939","logging-call-at":"factory.go:67"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.613Z","msg":"Listening for GRPC requests","service":"cadence-worker","address":"172.19.0.6:0","logging-call-at":"factory.go:83"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.613Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.rps,clusterName:primary","value":"1200","default-value":"1200","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.613Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.minRetentionDays,clusterName:primary","value":"0","default-value":"1","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.614Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.advancedVisibilityWritingMode,clusterName:primary","value":"off","default-value":"off","logging-call-at":"config.go:93"}
cadence_1        | 2021/10/05 14:32:07 failed to create file blobstore client, will continue startup without it: output directory not given for file blobstore
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"Starting service worker","service":"cadence-worker","logging-call-at":"server.go:235"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.advancedVisibilityWritingMode,clusterName:primary","value":"off","default-value":"off","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.throttledLogRPS,clusterName:primary","value":"20","default-value":"20","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.enableGRPCOutbound,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.grpcMaxSizeInByte,clusterName:primary","value":"4194304","default-value":"4194304","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"First loading dynamic config","service":"cadence-worker","key":"history.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.615Z","msg":"Creating GRPC dispatcher outbound","service":"cadence-worker","address":"cadence:7833","logging-call-at":"dispatcher_provider.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.616Z","msg":"First loading dynamic config","service":"cadence-worker","key":"admin.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.617Z","msg":"Add new peers by DNS lookup","service":"cadence-worker","address":"cadence","addresses":"[172.19.0.6:7833]","logging-call-at":"dns_updater.go:80"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.617Z","msg":"First loading dynamic config","service":"cadence-worker","key":"frontend.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.617Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.persistenceGlobalMaxQPS,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.617Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.persistenceMaxQPS,clusterName:primary","value":"500","default-value":"500","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.618Z","msg":"RuntimeMetricsReporter started","service":"cadence-frontend","logging-call-at":"runtime.go:169"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.618Z","msg":"PProf not started due to port not set","service":"cadence-frontend","logging-call-at":"pprof.go:64"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.655Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.persistenceErrorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.824Z","msg":"First loading dynamic config","service":"cadence-worker","key":"matching.errorInjectionRate,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.827Z","msg":"worker starting","service":"cadence-worker","component":"worker","logging-call-at":"service.go:189"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.827Z","msg":"RuntimeMetricsReporter started","service":"cadence-worker","logging-call-at":"runtime.go:169"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.827Z","msg":"PProf not started due to port not set","service":"cadence-worker","logging-call-at":"pprof.go:64"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.832Z","msg":"Current reachable members","service":"cadence-worker","component":"service-resolver","addresses":"[172.19.0.6:7939]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.854Z","msg":"First loading dynamic config","service":"cadence-worker","key":"system.enableGlobalDomain,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.854Z","msg":"service started","service":"cadence-worker","logging-call-at":"resourceImpl.go:382"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.executionsScannerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.concreteExecutionFixerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.currentExecutionsScannerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.currentExecutionFixerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.timersScannerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.timersFixerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.historyScannerEnabled,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.replicationTaskMaxRetryDuration,clusterName:primary","value":"10m0s","default-value":"10m0s","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.858Z","msg":"First loading dynamic config","service":"cadence-worker","key":"worker.enableBatcher,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.862Z","caller":"internal/internal_worker.go:223","msg":"No logger configured for cadence worker. Created default one."}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.983Z","msg":"Current reachable members","service":"cadence-frontend","component":"service-resolver","addresses":"[172.19.0.6:7939]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.984Z","msg":"Current reachable members","service":"cadence-frontend","component":"service-resolver","addresses":"[172.19.0.6:7933]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.990Z","msg":"service started","service":"cadence-frontend","logging-call-at":"resourceImpl.go:382"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.991Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"system.enableGracefulFailover,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.991Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.domainFailoverRefreshInterval,clusterName:primary","value":"10s","default-value":"10s","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.991Z","msg":"First loading dynamic config","service":"cadence-frontend","key":"frontend.domainFailoverRefreshTimerJitterCoefficient,clusterName:primary","value":"0.1","default-value":"0.1","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.992Z","msg":"Domain failover processor started.","service":"cadence-frontend","logging-call-at":"failover_watcher.go:107"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:07.992Z","msg":"frontend started","service":"cadence-frontend","logging-call-at":"service.go:291"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.002Z","msg":"Current reachable members","service":"cadence-history","component":"service-resolver","addresses":"[172.19.0.6:7939]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.003Z","msg":"Current reachable members","service":"cadence-history","component":"service-resolver","addresses":"[172.19.0.6:7933]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.003Z","msg":"Current reachable members","service":"cadence-history","component":"service-resolver","addresses":"[172.19.0.6:7934]","logging-call-at":"rpServiceResolver.go:246"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.012Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.enableGlobalDomain,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.012Z","msg":"service started","service":"cadence-history","logging-call-at":"resourceImpl.go:382"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.012Z","msg":"history resource started","service":"cadence-history","lifecycle":"Started","logging-call-at":"resource.go:59"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"Replication task fetchers started.","service":"cadence-history","logging-call-at":"task_fetcher.go:134"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerType,clusterName:primary","value":"2","default-value":"2","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerQueueSize,clusterName:primary","value":"10000","default-value":"10000","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerDispatcherCount,clusterName:primary","value":"1","default-value":"1","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerShardWorkerCount,clusterName:primary","value":"0","default-value":"0","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerRoundRobinWeight,clusterName:primary","value":"map[1:500 17:5 9:20]","default-value":"map[1:500 17:5 9:20]","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.013Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.taskSchedulerWorkerCount,clusterName:primary","value":"200","default-value":"200","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.014Z","msg":"Parallel task processor started.","service":"cadence-history","logging-call-at":"parallelTaskProcessor.go:99"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.014Z","msg":"Weighted round robin task scheduler started.","service":"cadence-history","logging-call-at":"weightedRoundRobinTaskScheduler.go:126"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.014Z","msg":"Queue task processor started.","service":"cadence-history","logging-call-at":"processor.go:120"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.014Z","msg":"First loading dynamic config","service":"cadence-history","key":"system.enableGracefulFailover,clusterName:primary","value":"true","default-value":"true","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"Coordinator state changed","service":"cadence-history","component":"failover-coordinator","lifecycle":"Started","logging-call-at":"coordinator.go:152"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.acquireShardConcurrency,clusterName:primary","value":"1","default-value":"1","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.NotifyFailoverMarkerInterval,clusterName:primary","value":"5s","default-value":"5s","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.NotifyFailoverMarkerTimerJitterCoefficient,clusterName:primary","value":"0.15","default-value":"0.15","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"Shard item state changed","service":"cadence-history","shard-id":0,"address":"172.19.0.6:7934","lifecycle":"Started","component":"shard-item","logging-call-at":"controller.go:313"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.015Z","msg":"Shard engine state changed","service":"cadence-history","shard-id":0,"address":"172.19.0.6:7934","lifecycle":"Starting","component":"shard-engine","logging-call-at":"controller.go:452"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.076Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheInitialSize,clusterName:primary","value":"128","default-value":"128","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.077Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheMaxSize,clusterName:primary","value":"512","default-value":"512","logging-call-at":"config.go:93"}
cadence_1        | {"level":"info","ts":"2021-10-05T14:32:08.077Z","msg":"First loading dynamic config","service":"cadence-history","key":"history.eventsCacheGlobalEnable,clusterName:primary","value":"false","default-value":"false","logging-call-at":"config.go:93"}
cadence_1        | panic: runtime error: invalid memory address or nil pointer dereference [recovered]
cadence_1        | 	panic: runtime error: invalid memory address or nil pointer dereference
cadence_1        | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e0a796]
cadence_1        | 
cadence_1        | goroutine 1429 [running]:
cadence_1        | go.uber.org/yarpc/internal/observability.(*Middleware).handlePanicForCall(0xc000777e00, 0xc000136a10, 0x2eb0f10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/observability/middleware.go:315 +0x175
cadence_1        | panic(0x2913aa0, 0x4c485f0)
cadence_1        | 	/usr/local/go/src/runtime/panic.go:679 +0x1b2
cadence_1        | github.com/uber/cadence/common/rpc.(*authOutboundMiddleware).Call(0x0, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x34d2460, 0xc000c1d640, 0x0, 0x0, 0x0)
cadence_1        | 	/cadence/common/rpc/middleware.go:37 +0x26
cadence_1        | go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/api/middleware/outbound.go:113
cadence_1        | go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call(0xc000c1d750, 0x0, 0x0, 0x34d39e0, 0xc0006c30e0, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x4c780e0, 0x34c1360, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/outboundmiddleware/chain.go:103 +0x8d
cadence_1        | go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc000777e00, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x34d3ae0, 0xc001c8c2a0, 0x0, 0x0, 0x0)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/observability/middleware.go:213 +0x174
cadence_1        | go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call(0xc000c1d750, 0x0, 0x0, 0x34d39e0, 0xc0006c30e0, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x7f7278a952c0, 0xc0006c30e0, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/outboundmiddleware/chain.go:107 +0x194
cadence_1        | go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x4c9b018, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x34d3ae0, 0xc001c8c270, 0x0, 0x203000, 0x203000)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/firstoutboundmiddleware/middleware.go:51 +0xac
cadence_1        | go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call(0xc000c1d750, 0x1, 0x1, 0x34d39e0, 0xc0006c30e0, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x30, 0x110, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/outboundmiddleware/chain.go:107 +0x194
cadence_1        | go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/outboundmiddleware/chain.go:67
cadence_1        | go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/api/middleware/outbound.go:113
cadence_1        | go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call(0x34d39e0, 0xc00070a0c0, 0x7f7278a952c0, 0xc00070a0c0, 0x34c1360, 0xc001046d20, 0xc0007758c0, 0x3, 0x4, 0xc000cf0ce0)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/request/validator_outbound.go:46 +0xc1
cadence_1        | go.uber.org/yarpc/encoding/thrift.thriftClient.Call(0x34c2760, 0xc0000eb810, 0x34c4220, 0x4c9b018, 0x2dac886, 0xf, 0x0, 0x34c1360, 0xc001046d20, 0x34b1620, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/encoding/thrift/outbound.go:145 +0x204
cadence_1        | go.uber.org/cadence/.gen/go/cadence/workflowserviceclient.client.DescribeDomain(0x34918a0, 0xc000f2ac80, 0x34c1360, 0xc001046d20, 0xc0014900e0, 0xc001046d80, 0x4, 0x6, 0x5b5bf275, 0xc0015e7500, ...)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/.gen/go/cadence/workflowserviceclient/client.go:345 +0xda
cadence_1        | go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).DescribeDomain(0xc000b46c60, 0x34c1360, 0xc001046d20, 0xc0014900e0, 0xc001046d80, 0x4, 0x6, 0xc001046d20, 0xc0014900a0, 0xc001046d80)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/common/metrics/service_wrapper.go:149 +0xb8
cadence_1        | go.uber.org/cadence/internal.verifyDomainExist.func1(0x0, 0x0)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/internal_worker.go:251 +0x18c
cadence_1        | go.uber.org/cadence/internal/common/backoff.Retry(0x34c1320, 0xc000100450, 0xc001c95688, 0x345ce80, 0xc00033f2f0, 0x2eb0de0, 0xb, 0xb)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/common/backoff/retry.go:98 +0xfb
cadence_1        | go.uber.org/cadence/internal.verifyDomainExist(0x35047e0, 0xc000b46c60, 0x2dacacf, 0xf, 0xc001133800, 0x0, 0x0, 0x0)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/internal_worker.go:272 +0x18b
cadence_1        | go.uber.org/cadence/internal.(*workflowWorker).Start(0xc001488d00, 0xc0007704d0, 0xb)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/internal_worker.go:367 +0x7a
cadence_1        | go.uber.org/cadence/internal.(*aggregatedWorker).Start(0xc001492380, 0xc0005f3d60, 0x2dacacf)
cadence_1        | 	/go/pkg/mod/go.uber.org/[email protected]/internal/internal_worker.go:830 +0x585
cadence_1        | github.com/uber/cadence/service/worker/batcher.(*Batcher).Start(0xc00013a930, 0xc00013a930, 0xc0008d8fc0)
cadence_1        | 	/cadence/service/worker/batcher/batcher.go:99 +0x1bf
cadence_1        | github.com/uber/cadence/service/worker.(*Service).startBatcher(0xc0015d26f0)
cadence_1        | 	/cadence/service/worker/service.go:262 +0x1b9
cadence_1        | github.com/uber/cadence/service/worker.(*Service).Start(0xc0015d26f0)
cadence_1        | 	/cadence/service/worker/service.go:208 +0x416
cadence_1        | github.com/uber/cadence/cmd/server/cadence.execute(0x7f72787c3008, 0xc0015d26f0, 0xc0007549c0)
cadence_1        | 	/cadence/cmd/server/cadence/server.go:260 +0x31
cadence_1        | created by github.com/uber/cadence/cmd/server/cadence.(*server).startService
cadence_1        | 	/cadence/cmd/server/cadence/server.go:253 +0x1452
cassandra_1      | INFO  [Native-Transport-Requests-3] 2021-10-05 14:32:08,134 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED
cassandra_1      | INFO  [HANDSHAKE-/172.19.0.3] 2021-10-05 14:32:08,161 OutboundTcpConnection.java:561 - Handshaking version with /172.19.0.3
cadence_cadence_1 exited with code 2

The text was updated successfully, but these errors were encountered:

longquanzheng · 2021-10-05T21:53:11Z

Hello @naresh8t7

There was a bug we introduced in a PR yesterday and reverted after a few hours.

Please run docker pull ubercadence/server:master-auto-setup to update the image then run the docker-compose up command again.

We added a test to protect the docker image so hopefully this won't happen again.
#4535

Sorry for the inconvenience!

naresh8t7 · 2021-10-06T02:57:01Z

Thanks @longquanzheng
I am able to start the server now.
But when I run the below command seeing few errors in common package's thrift_util.

go get go.uber.org/cadence
# go.uber.org/cadence/internal/common
../../../go/pkg/mod/go.uber.org/[email protected]/internal/common/thrift_util.go:31:38: not enough arguments in call to thrift.NewTSerializer().Write
        have (thrift.TStruct)
        want (context.Context, thrift.TStruct)
../../../go/pkg/mod/go.uber.org/[email protected]/internal/common/thrift_util.go:53:27: not enough arguments in call to t.Protocol.Flush
        have ()
        want (context.Context)
../../../go/pkg/mod/go.uber.org/[email protected]/internal/common/thrift_util.go:57:28: not enough arguments in call to t.Transport.Flush
        have ()
        want (context.Context)

longquanzheng · 2021-10-06T04:00:01Z

@naresh8t7 yeah that’s a know issue , please see uber-go/cadence-client#1129 (comment) lmk if that works or not

longquanzheng · 2021-10-06T04:05:37Z

Make sure you use the right version of thrift

Thrift 0.9.3 (use homebrew or distribution)

naresh8t7 · 2021-10-11T03:28:37Z

Thanks @longquanzheng . It worked.
When running samples, I created sample domain by running
docker run --network=host --rm ubercadence/cli:master --do samples-domain domain register -rd 1
and running the helloworld sample, getting below error in the worker. Any inputs will be highly appreciated. Thanks
2021-10-10T23:20:12.717-0400 WARN internal/internal_worker.go:261 unable to verify if domain exist {"Domain": "samples-domain", "TaskList": "helloWorldGroup", "WorkerID": "56269@XYZ@helloWorldGroup", "WorkerType": "LocallyDispatchedActivityWorker", "domain": "samples-domain", "error": "code:unknown message:received unknown error calling service: "cadence-frontend", procedure: "WorkflowService::DescribeDomain", err: unexpected EOF"}

longquanzheng · 2021-10-11T04:10:22Z

What is the output of the domain register command?

I wonder if the server by docker compose is actually up

naresh8t7 · 2021-10-12T02:35:50Z

Yes, the docker compose is up and server is running fine. Domain register command output is as below.
docker run --network=host --rm ubercadence/cli:master --do samples-domain domain describe
Name: samples-domain
UUID: 8e0445b2-f9e1-482f-9bf8-0cbe94147cc7
Description:
OwnerEmail:
DomainData: map[]
Status: REGISTERED
RetentionInDays: 1
EmitMetrics: true
IsGlobal(XDC)Domain: true
ActiveClusterName: primary
Clusters: primary
HistoryArchivalStatus: DISABLED
VisibilityArchivalStatus: DISABLED
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+

longquanzheng · 2021-10-13T17:13:16Z

@naresh8t7 Sorry for the late response.
Have you figured it out? I am guessing there is something wrong with the compiling sample repo, or networking issue with the docker(because your CLI is also docker).
Something to try to eliminate the possibility of docker

If you are on a mac, can you try brew install cadence-workflow to get the cadence CLI, and then describe the domain again?
If not, can you try follow the instruction to build the CLI off the repo: https://github.com/uber/cadence#use-cadence-cli
If local CLI(not docker) can access directly, then there is no reason that sample doens't work. They are exactly the same way communicating with server. If that's the case, make sure you clean off the sample and re-build it.
If the local CLI cannot work either, then it's likely docker network is broken. Try do docker system prune --all to clean it up and restart the server. See more https://docs.docker.com/config/pruning/ .

LMK how it goes.

longquanzheng · 2021-10-18T14:52:39Z

Thanks @longquanzheng . It worked.
When running samples, I created sample domain by running
docker run --network=host --rm ubercadence/cli:master --do samples-domain domain register -rd 1
and running the helloworld sample, getting below error in the worker. Any inputs will be highly appreciated. Thanks
2021-10-10T23:20:12.717-0400 WARN internal/internal_worker.go:261 unable to verify if domain exist {"Domain": "samples-domain", "TaskList": "helloWorldGroup", "WorkerID": "56269@XYZ@helloWorldGroup", "WorkerType": "LocallyDispatchedActivityWorker", "domain": "samples-domain", "error": "code:unknown message:received unknown error calling service: "cadence-frontend", procedure: "WorkflowService::DescribeDomain", err: unexpected EOF"}

Hi, can you check the latest change in samples? We have moved to gRPC so hopefully you won’t have any issue with thrift anymore

naresh8t7 changed the title ~~Trying to set up a cadence server using docker-compose~~ Issue starting cadence server using docker-compose Oct 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue starting cadence server using docker-compose #4540

Issue starting cadence server using docker-compose #4540

naresh8t7 commented Oct 5, 2021 •

edited by longquanzheng

Loading

longquanzheng commented Oct 5, 2021 •

edited

Loading

naresh8t7 commented Oct 6, 2021 •

edited by longquanzheng

Loading

longquanzheng commented Oct 6, 2021

longquanzheng commented Oct 6, 2021

naresh8t7 commented Oct 11, 2021

longquanzheng commented Oct 11, 2021

naresh8t7 commented Oct 12, 2021

longquanzheng commented Oct 13, 2021

longquanzheng commented Oct 18, 2021

Issue starting cadence server using docker-compose #4540

Issue starting cadence server using docker-compose #4540

Comments

naresh8t7 commented Oct 5, 2021 • edited by longquanzheng Loading

longquanzheng commented Oct 5, 2021 • edited Loading

naresh8t7 commented Oct 6, 2021 • edited by longquanzheng Loading

longquanzheng commented Oct 6, 2021

longquanzheng commented Oct 6, 2021

naresh8t7 commented Oct 11, 2021

longquanzheng commented Oct 11, 2021

naresh8t7 commented Oct 12, 2021

longquanzheng commented Oct 13, 2021

longquanzheng commented Oct 18, 2021

naresh8t7 commented Oct 5, 2021 •

edited by longquanzheng

Loading

longquanzheng commented Oct 5, 2021 •

edited

Loading

naresh8t7 commented Oct 6, 2021 •

edited by longquanzheng

Loading