indexing failing

[stela-leon]

Hi people,

I tried to upgrade a cassandra cluster to elassandra(6.8.4.13), all went well until i tried to create an index in ES via rest call, curl just hangs while i can see the log entries below. The problem might be the X2 fields, different for each node, but I am not sure how to fix it, any help is more than welcome.

Questions:
How did this happen?
How do I fix it?
How do I prevent it? (if possible)

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. migrated an existing cassandra cluster to elassandra
2. started the new elassandra nodes with es disabled,
3. drained the old nodes' data then shut scaled them down to have only elassandra updated nodes,
4. repair the nodes, enable ES and recreate the nodes

Please provide the following information:

• elassandra logs (logs/system.logs or /var/lib/cassandra/system.log)
  DEBUG [elasticsearch[][masterService#updateTask][T#1]] org.elassandra.discovery.CassandraDiscovery.publishAsCoordinator(CassandraDiscovery.java:1016) Coordinator update source=create-index [test_index], cause [api] metadata=587328c5-3be1-48a9-907c-9b0d9fdfc399/0 WARN [elasticsearch[][masterService#updateTask][T#1]] org.elasticsearch.cluster.service.ClusterService.commitMetaData(ClusterService.java:1302) PAXOS Failed to update metadata source=create-index [test_index], cause [api] prevMetadata=587328c5-3be1-48a9-907c-9b0d9fdfc399/0 nextMetaData=587328c5-3be1-48a9-907c-9b0d9fdfc399/1 WARN [elasticsearch[][masterService#updateTask][T#1]] org.elassandra.discovery.CassandraDiscovery.publishAsCoordinator(CassandraDiscovery.java:1114) PAXOS concurrent update, source=create-index [test_index], cause [api] metadata=587328c5-3be1-48a9-907c-9b0d9fdfc399/1, resubmit task on next metadata change

• elasticsearch cluster state (curl http://localhost:9200/_cluster/state)

{"cluster_name":"cassandra-staging","cluster_uuid":"587328c5-3be1-48a9-907c-9b0d9fdfc399","version":1983,"state_uuid":"iJhsKS6MSVa2hs4vYUV2pA","master_node":"587328c5-3be1-48a9-907c-9b0d9fdfc399","blocks":{},"nodes":{"587328c5-3be1-48a9-907c-9b0d9fdfc399":{"name":"192.168.16.9","status":"ALIVE","ephemeral_id":"587328c5-3be1-48a9-907c-9b0d9fdfc399","transport_address":"192.168.16.9:9300","attributes":{"rack":"r1","dc":"dc1-staging"}},"1c97c04f-117c-4d8b-8cb8-5fba1720dcaa":{"name":"192.168.52.31","status":"ALIVE","ephemeral_id":"1c97c04f-117c-4d8b-8cb8-5fba1720dcaa","transport_address":"192.168.52.31:9300","attributes":{"dc":"dc1-staging","rack":"r1"}},"0cf0233f-a833-4a52-aadd-cefb40b584dd":{"name":"192.168.68.49","status":"ALIVE","ephemeral_id":"0cf0233f-a833-4a52-aadd-cefb40b584dd","transport_address":"192.168.68.49:9300","attributes":{"dc":"dc1-staging","rack":"r1"}}},"metadata":{"version":0,"cluster_uuid":"587328c5-3be1-48a9-907c-9b0d9fdfc399","templates":{},"indices":{},"index-graveyard":{"tombstones":[]}},"routing_table":{"indices":{}},"routing_nodes":{"unassigned":[],"nodes":{"1c97c04f-117c-4d8b-8cb8-5fba1720dcaa":[],"587328c5-3be1-48a9-907c-9b0d9fdfc399":[],"0cf0233f-a833-4a52-aadd-cefb40b584dd":[]}},"restore":{"snapshots":[]},"snapshot_deletions":{"snapshot_deletions":[]},"snapshots":{"snapshots":[]}}r

• cassandra schema (cqlsh>DESC KEYSPACE <your_keyspace>)
  `cqlsh> describe keyspace es_test

CREATE KEYSPACE es_test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1-staging': '2'} AND durable_writes = true;

CREATE TABLE es_test.test (
test_id text PRIMARY KEY
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
`

• cassandra gossip state (run: nodetool gossipinfo)
  root@elassandra-staging-0:/# nodetool gossipinfo /x.x.x.49.9 generation:1615320857 heartbeat:55238 STATUS:17:NORMAL,-4053865950737980 LOAD:55233:3.39504115E8 SCHEMA:54011:62c33652-5e84-30ce-93d3-2b07ef0e51ce DC:9:dc1-staging RACK:11:r1 RELEASE_VERSION:5:3.11.9.1 INTERNAL_IP:7:192.168.16.9 RPC_ADDRESS:4:192.168.16.9 NET_VERSION:2:11 HOST_ID:3:587328c5-3be1-48a9-907c-9b0d9fdfc399 RPC_READY:29:true X1:35:{} X2:33:587328c5-3be1-48a9-907c-9b0d9fdfc399/0 TOKENS:16:<hidden> /x.x.x.49.31 generation:1615322731 heartbeat:53290 STATUS:17:NORMAL,-1426293402904595724 LOAD:53239:2.42190901E8 SCHEMA:52064:62c33652-5e84-30ce-93d3-2b07ef0e51ce DC:9:dc1-staging RACK:11:r1 RELEASE_VERSION:5:3.11.9.1 INTERNAL_IP:7:192.168.52.31 RPC_ADDRESS:4:192.168.52.31 NET_VERSION:2:11 HOST_ID:3:1c97c04f-117c-4d8b-8cb8-5fba1720dcaa RPC_READY:29:true X1:35:{} X2:33:1c97c04f-117c-4d8b-8cb8-5fba1720dcaa/0 TOKENS:16:<hidden> /x.x.x.49 generation:1615320908 heartbeat:55185 STATUS:17:NORMAL,-1285334177861009996 LOAD:55171:1.47487769E8 SCHEMA:53958:62c33652-5e84-30ce-93d3-2b07ef0e51ce DC:9:dc1-staging RACK:11:r1 RELEASE_VERSION:5:3.11.9.1 INTERNAL_IP:7:192.168.68.49 RPC_ADDRESS:4:192.168.68.49 NET_VERSION:2:11 HOST_ID:3:0cf0233f-a833-4a52-aadd-cefb40b584dd RPC_READY:29:true X1:35:{} X2:33:0cf0233f-a833-4a52-aadd-cefb40b584dd/0 TOKENS:16:<hidden>
• index creation on top of an already existing cassandra table:
  curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/test_index' -d '{ "settings": { "keyspace":"es_test" }, "mappings": { "test" : { "discover" : ".*" } } }'