-
Notifications
You must be signed in to change notification settings - Fork 277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix race conditions in MessageFilter #539
Open
rhaschke
wants to merge
9
commits into
ros:noetic-devel
Choose a base branch
from
rhaschke:fix-race-condition-noetic
base: noetic-devel
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Commits on Jul 2, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 07b5a27 - Browse repository at this point
Copy the full SHA 07b5a27View commit details -
Add stress test for MessageFilter
... revealing that propagating messages via a ROS callback queue is inherently unsafe. If the filter is going to be destroyed, a CBQueueCallback::call() might still be in action, locking the underlying Signal1's mutex_. This will cause as assertion failure during destruction of the mutex: boost::mutex::~mutex(): Assertion `!posix::pthread_mutex_destroy(&m)' failed.
Configuration menu - View commit details
-
Copy full SHA for 58a597d - Browse repository at this point
Copy the full SHA 58a597dView commit details -
When MessageFilter::signalMessage() is called asynchronouyls via a ROS callback queue, this call might still be in progress while another (holding Signal1::mutex_), while another thread might attempt to remove the MessageFilter. This results in a failed assertion as the MessageFilter destructor attempts to destroy a locked mutex. Here, by introducing another mutex in MessageFilter, we ensure that CBQueueCallback::call() is finished before destroying the MessageFilter.
Configuration menu - View commit details
-
Copy full SHA for 529ce02 - Browse repository at this point
Copy the full SHA 529ce02View commit details -
Configuration menu - View commit details
-
Copy full SHA for 91d7601 - Browse repository at this point
Copy the full SHA 91d7601View commit details -
Revert ros#144: Solve a bug that causes a deadlock in MessageFilter
PR ros#144 attempted to resolve a deadlock by postponing calls to callbacks. However, this has the drawback of using references to TransformableRequests - outside the protection by transformable_requests_mutex_. Thus, these requests might get deleted by another thread, resulting in segfaults: terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>' what(): boost: mutex lock failed in pthread_mutex_lock: Invalid argument This reverts commit bfb8038.
Configuration menu - View commit details
-
Copy full SHA for 23e6295 - Browse repository at this point
Copy the full SHA 23e6295View commit details -
Configuration menu - View commit details
-
Copy full SHA for 436caee - Browse repository at this point
Copy the full SHA 436caeeView commit details -
Alternative implementation to resolve deadlock ros#91/ros#144
As pointed out in ros#144 the deadlock arises due to an inversion of locking order: - MessageFilter::add(), when removing the oldest message, locks: - MessageFilter::messages_mutex_ - BufferCore::transformable_requests_mutex_ (via cancelTransformableRequest()) - BufferCore::testTransformableRequests() locks: - transformable_requests_mutex_ - MessageFilter::message_mutex_ (via MessageFilter::transformable() callback) PR ros#144 attempted to resolve the issue by postponing callback calls. However, this has the drawback of using references to TransformableRequests - outside the protection by transformable_requests_mutex_. These requests might got deleted by another thread meanwhile! Here the deadlock is resolved in MessageFilter::add, cancelling the requests outside the messages_mutex_ protection.
Configuration menu - View commit details
-
Copy full SHA for 9fdf547 - Browse repository at this point
Copy the full SHA 9fdf547View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4a05272 - Browse repository at this point
Copy the full SHA 4a05272View commit details -
Reduce probability of race condition with CBQueueCallback
While MessageFilter::clear() removes pending messages from the callback queue, there might be still callbacks active when attempting to remove the MessageFilter. Depending on what goes first, either the already destroyed mutex is tried to lock or a locked mutex is tried to destroy, both resulting in an assertion failure. This race condition cannot completely avoided. Using a shared mutex for all callers and moving the unique lock to the end of the destructor hopefully reduces the probability of such a race condition.
Configuration menu - View commit details
-
Copy full SHA for e469abf - Browse repository at this point
Copy the full SHA e469abfView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.