Remove a single learner or member

[sebadob]

Hey,

I am trying to implement auto self-healing for my new Hiqlite project and most of it works as I need it, but there is one last thing I could not find a solution for yet.

Is there a way to remove a single learner / member?

I can of course use change_membership() in combination with retain = false to get rid of members, but in this case I might have for instance:

• 3 voting members
• 2 replica nodes as learners only

... and I want to remove just either a member or a learner. The problem is, that I can get rid of a member with retain = false setting, but this would also remove learners that I might want to keep. Is there a way to remove a single node from this list while all others stay unaffected? I did not find something like remove_member() or remove_learner().

The reason I want to do this is, that I want to be able to make nodes auto-recover from failed or corrupted volumes.
My initial idea was that I could just set loosen-follower-log-revert but that does not solve it. I guess removing a completely failed node from the Raft itself and adding it again as learner / member would do the trick, but this would remove other learners as well.
Or can I maybe work around this with a combination of loosen-follower-log-revert + manual raft.get_snapshot() or raft.trigger() somehow? In the end, the problem I need to solve is when a node was already a cluster member but lost its volume and therefore forgot its membership config, but the node should be able to do this without user interaction.

[drmingdrmer]

Oh it should be a bug that retain=false should only affect the nodes that was a voter. Existing learner should always be kept.

[sebadob]

Ohh okay, then I misunderstoof the doc about it. Maybe you could add a 3rd option explaining the retain, one where you have members + learners + retain = false.

So this means, retain = false will fully remove members (which it does correctly for me) but it will not affect learners?
I did not get to the point in my integration tests where I checked this with learners only, I am just thinking about a solution for now, but I will test this.

But, how can I remove a learner then without promoting it as a member first and then using retain = false to get rid of it?
Because I would like to have probably at least one read-only replica for analys and such tasks, but that one should not affect any voting or quorum.

[drmingdrmer]

What's the statement you calling Raft::change_member() and what's your cluster setup before calling change_member()? I can not yet reproduce this issue on my laptop.

So this means, retain = false will fully remove members (which it does correctly for me) but it will not affect learners?
I did not get to the point in my integration tests where I checked this with learners only, I am just thinking about a solution for now, but I will test this.

retain=false is meant to remove just the affected voter and leave the existing learner unaffected.

But, how can I remove a learner then without promoting it as a member first and then using retain = false to get rid of it?
Because I would like to have probably at least one read-only replica for analys and such tasks, but that one should not affect any voting or quorum.

Raft::change_membership(ChangeMembers::RemoveNodes(...)) can be used to remove a learner.

[sebadob]

What's the statement you calling Raft::change_member() and what's your cluster setup before calling change_member()? I can not yet reproduce this issue on my laptop.

I have not gotten to the point where I ran into this issue. That was basically a question from reading the tooltip for change_membership(), how I understood it. It's probably fine and I just misunderstood the docs, sorry.

Raft::change_membership(ChangeMembers::RemoveNodes(...)) can be used to remove a learner.

That + the information that retain leaves learners unaffected solves my issue.

If I do run into an issue with retain + keeping learner, I will let you know, but I don't think so. Was just a misunderstanding.

Thanks a lot for your help!

[drmingdrmer]

Alright! glad to hear that!

[drmingdrmer]

I updated the doc for change_membership() in

• Doc: fix description of retain in change_membership() #1194

pub async fn change_membership(
    &self,
    members: impl Into<ChangeMembers<C>>,
    retain: bool
) -> Result<ClientWriteResponse<C>, RaftError<C, ClientWriteError<C>>>

Propose a cluster configuration change.

A node in the proposed config has to be a learner, otherwise it fails with LearnerNotFound error.

Internally:

• It proposes a joint config.
• When the joint config is committed, it proposes a uniform config.

If retain is true, then all the members which not exists in the new membership, will be turned into learners, otherwise will be removed. If retain is false, the removed voter will be removed from the cluster. Existing learners will not be affected.

Example of retain usage: If the original membership is {"voter":{1,2,3}, "learners":{1,2,3,4,5}}, and call change_membership with voters={2,3,4}, then:

• If retain is true, the committed new membership is {"voters":{2,3,4}, "learners":{1,2,3,4,5}}.
• Otherwise if retain is false, then the new membership is {"voters":{2,3,4}, "learners":{2,3,4,5}}, in which the removed voters 1 are removed from the cluster. 5 is not affected.

If it loses leadership or crashed before committing the second uniform config log, the cluster is left in the joint config.

[sebadob]

That's really helpful. I think the updated version makes it much more clear how it's working.

Thank you!