Sharding issue - sharded entity is duplicated

[giovanni-ferrari]

I created a cluster with three seed nodes, and 3 working nodes.
Working nodes (port 9998) create a sharded region, and I would expect, with the current configuration, to have one sharded entity in the whole cluster.

The application aims to build a sharded entity connecting to an mqtt broker, and to keep that connection resilient across the cluster in case of nodes failures.

The sharded entity is created by the region on [ConnectionConfiguration] message.

In the following logs [akka.tcp://CustomApplication@host1:9998] si running the sharded entity [akka://CustomApplication/system/sharding/EdgeActorRegion/0/mosquitto%3A1883]

[akka.tcp://CustomApplication@host2:9998] and [akka.tcp://CustomApplication@host2:8110] were temporarly offline, so that the cluster was composed only by seed nodes

[akka.tcp://CustomApplication@host1:8110]
[akka.tcp://CustomApplication@host3:8110]

and worker nodes

[akka.tcp://CustomApplication@host1:9998]
[akka.tcp://CustomApplication@host3:9998]

When [akka.tcp://CustomApplication@host2:9998] and [akka.tcp://CustomApplication@host2:8110] are back online the sharded entity is still running on [akka.tcp://CustomApplication@host1:9998], but it is also instantiated in [akka.tcp://CustomApplication@host2:8110].

From that moment on the two instances of the sharded entity start connecting/disconnecting from Mqtt broker, because they are sharing the same clientId.

host1.txt
host2.txt
host3.txt

Any ideas why this could happen and how to resolve?

[Aaronontheweb]

Related issue, which I have been investigating aggressively over the past few weeks: #6973 - mentioned it on today's Akka.NET Community Standup. This is definitely an Akka.NET bug. Let me look over your logs and data.

[Aaronontheweb]

Also, what sharding configuration are you using @giovanni-ferrari ?

[Aaronontheweb]

When [akka.tcp://CustomApplication@host2:9998] and [akka.tcp://CustomApplication@host2:8110] are back online the sharded entity is still running on [akka.tcp://CustomApplication@host1:9998], but it is also instantiated in [akka.tcp://CustomApplication@host2:8110].

It looks from the logs like the second actor is actually on 9998, the worker node that starts the ShardRegion, which would make sense

[giovanni-ferrari]

Also, what sharding configuration are you using @giovanni-ferrari ?

Currently I have this configuration

builder
            .WithRemoting(_akkaConfiguration.Host, port)
            .WithClustering(new ClusterOptions()
            {
                Roles = [role],
                SeedNodes = _akkaConfiguration.Seeds
            });

 builder.WithShardRegion<BridgeShardKey>("BridgeActorRegion",
     (system, registry) => s => DependencyResolver.For(system).Props<BridgeActor>(s),
     new BridgeShardMessageExtractor(10),
     new ShardOptions()
     {
         Role = _akkaConfiguration.Role
     });

 builder.WithActors((system, registry)
     => system
         .ActorOf(DependencyResolver.For(system)
         .Props<BridgeManagerActor>(registry.Get<BridgeShardKey>()), nameof(BridgeManagerActor)));

[Aaronontheweb]

And what database are you using for Akka.Persistence with these settings? Any? Because by-default with this HOCON it will use the in-memory database, which is 100% why you'd receive a duplicate in this case (that in-memory database is only meant to be used for testing - it's not replicated between nodes.)

To resolve this issue, set the ShardOptions.StateStorageMode = DData or add a real Akka.Persistence configuration to your application.

[giovanni-ferrari]

Thanks Aaron,
I'm going to setup HOCON with StateStorageMode = DData and let you know if that fixes the issue, but i'm 100% in the configuration you described.