Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: update necessary if shadow cleared #196

Merged
merged 7 commits into from
Oct 11, 2023
Merged

fix: update necessary if shadow cleared #196

merged 7 commits into from
Oct 11, 2023

Conversation

jcosentino11
Copy link
Member

@jcosentino11 jcosentino11 commented Oct 9, 2023

Issue #, if available:

Description of changes:
Ensure the cloud update document {"state": null} isn't ignored.

Currently, if a null cloud shadow update accepted document is received, {"state": null}, shadow manager will discard the update, because it thinks no change was made to the cloud state after merging the existing cloud document with {"state": null}:

boolean isUpdateNecessary(JsonNode baseDocument, JsonNode update) {
    JsonNode merged = baseDocument.deepCopy();
    JsonMerger.merge(merged.get(SHADOW_DOCUMENT_STATE), update.get(SHADOW_DOCUMENT_STATE));
    return !baseDocument.equals(merged);
}

To disambiguate between null and empty, we can explicitly check the update request, {"state": null}, within isUpdateNecessary.

In order to enable this comparison, we need to allow ShadowDocument#state to be null, which required some refactoring to support.

Why is this change necessary:
To allow null cloud shadow updates {"state": null} to be accepted by shadow manager, which will subsequently clear the local shadow.

How was this change tested:
A new integration test has been added, and we have UAT coverage as well, which is how the issue was initially discovered.

Any additional information or context required to review the change:

Checklist:

  • Updated the README if applicable
  • Updated or added new unit tests
  • Updated or added new integration tests
  • If your code makes a remote network call, it was tested with a proxy

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@MikeDombo
Copy link
Member

Test?

@@ -242,9 +242,7 @@ public UpdateThingShadowHandlerResponse handleRequest(UpdateThingShadowRequest r
.withVersion(updatedDocument.getVersion())
.withClientToken(clientToken)
.withTimestamp(Instant.now())
// explicitly convert to shadow document to return valid state.
// this is to prevent edge cases like returning null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to think that returning null is bad... so do we think this is a problem for our customers? Does the cloud shadow service ever return null?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my initial understanding was wrong, cloud shadow service does return null for update accepted topic. verified using the IoT mqtt test client and a real shadow

*/
@Setter
@JsonIgnore
private boolean clear;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why? We cannot handle nulls without this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can, although we use shadowDocument.state a bunch of places in the code. thinking was that this would be better than adding null checks everywhere

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what it would look like of course, but I think that marking state as @Nullable and then handling the null case would make more sense.

@@ -28,6 +29,7 @@
* Class for managing operations on the Shadow Document.
*/
@Getter
@JsonDeserialize(using = ShadowDocumentDeserializer.class)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using a custom deserializer and new clear field seemed less invasive than allowing ShadowDocument#state to be null, since we'd have to add null checks everywhere ShadowDocument is used

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The document isn't null though right, just state inside the document is null

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah just the state

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did a code search and it's less cases of this than what i thought, will refactor

@aws-greengrass aws-greengrass deleted a comment from github-actions bot Oct 10, 2023
@aws-greengrass aws-greengrass deleted a comment from github-actions bot Oct 10, 2023
@aws-greengrass aws-greengrass deleted a comment from github-actions bot Oct 10, 2023
@aws-greengrass aws-greengrass deleted a comment from github-actions bot Oct 10, 2023
@github-actions
Copy link

github-actions bot commented Oct 10, 2023

Unit Tests Coverage Report

File Coverage Lines Branches
All files 83% 89% 78%
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 21% 26% 15%
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 79% 85% 73%
com.aws.greengrass.shadowmanager.sync.model.Direction 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 99% 98% 100%
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 84% 97% 71%
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 80% 78% 83%
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 98% 95% 100%
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 81% 93% 69%
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 98% 95% 100%
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 74% 76% 71%
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 2% 2% 0%
com.aws.greengrass.shadowmanager.ShadowManager 74% 78% 71%
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 96% 97% 96%
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100%
com.aws.greengrass.shadowmanager.PubSubIntegrator 89% 93% 86%
com.aws.greengrass.shadowmanager.ShadowManager$1 43% 43% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 67% 67% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 67% 67% 0%
com.aws.greengrass.shadowmanager.util.JsonMerger 94% 98% 89%
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 91% 98% 84%
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0%
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0%
com.aws.greengrass.shadowmanager.util.Validator 100% 100% 100%
com.aws.greengrass.shadowmanager.util.JsonUtil 94% 99% 90%
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 88% 100% 75%
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 89% 100%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 99% 99% 100%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 91% 95% 88%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 63% 77% 50%
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 93% 97% 89%
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 93% 86% 100%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 55% 85% 25%
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 96% 99% 93%
com.aws.greengrass.shadowmanager.sync.SyncHandler 71% 88% 53%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 91% 91% 0%
com.aws.greengrass.shadowmanager.sync.RequestMerger 79% 77% 81%
com.aws.greengrass.shadowmanager.sync.CloudDataClient 74% 72% 77%
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 87% 93% 82%
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75%
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 100% 100% 100%
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 86% 92% 80%
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80%
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 100% 100% 0%
com.aws.greengrass.shadowmanager.model.ShadowDocument 85% 90% 80%
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 93% 98% 89%
com.aws.greengrass.shadowmanager.model.ErrorMessage 75% 100% 50%
com.aws.greengrass.shadowmanager.model.ShadowRequest 100% 100% 100%
com.aws.greengrass.shadowmanager.model.ShadowState 78% 81% 75%
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0%

Minimum allowed coverage is 65%

Generated by 🐒 cobertura-action against fc7890b

@github-actions
Copy link

github-actions bot commented Oct 10, 2023

Integration Tests Coverage Report

File Coverage Lines Branches
All files 72% 76% 68%
com.aws.greengrass.shadowmanager.sync.model.MergedFullShadowSyncRequest 83% 91% 75%
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 78% 83% 73%
com.aws.greengrass.shadowmanager.sync.model.Direction 83% 91% 75%
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 54% 58% 50%
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 71% 79% 62%
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 52% 54% 50%
com.aws.greengrass.shadowmanager.sync.model.DirectionWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 73% 71% 75%
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 52% 57% 46%
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 29% 29% 0%
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 62% 60% 64%
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 60% 60% 60%
com.aws.greengrass.shadowmanager.ShadowManager 90% 91% 88%
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 81% 87% 75%
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100%
com.aws.greengrass.shadowmanager.PubSubIntegrator 25% 22% 29%
com.aws.greengrass.shadowmanager.ShadowManager$1 14% 14% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 100% 100% 0%
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 100% 100% 0%
com.aws.greengrass.shadowmanager.util.JsonMerger 66% 73% 59%
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 76% 88% 64%
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0%
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0%
com.aws.greengrass.shadowmanager.util.Validator 67% 68% 65%
com.aws.greengrass.shadowmanager.util.JsonUtil 83% 90% 77%
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 75% 74% 75%
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 82% 82% 0%
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 55% 55% 0%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 100% 88%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 86% 72% 100%
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 75% 100% 50%
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 74% 77% 71%
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 53% 53% 0%
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 95% 95% 0%
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 61% 55% 67%
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0%
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 71% 92% 50%
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100%
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 78% 87% 70%
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 83% 90% 75%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 77% 93% 61%
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 61% 70% 52%
com.aws.greengrass.shadowmanager.sync.SyncHandler 82% 93% 71%
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.RequestMerger 60% 67% 52%
com.aws.greengrass.shadowmanager.sync.CloudDataClient 49% 45% 53%
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 71% 81% 61%
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75%
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 61% 73% 50%
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0%
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 55% 69% 40%
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80%
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 91% 91% 0%
com.aws.greengrass.shadowmanager.model.ShadowDocument 89% 91% 87%
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 83% 88% 78%
com.aws.greengrass.shadowmanager.model.ErrorMessage 97% 94% 100%
com.aws.greengrass.shadowmanager.model.ShadowRequest 88% 77% 100%
com.aws.greengrass.shadowmanager.model.ShadowState 96% 98% 94%
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0%

Minimum allowed coverage is 45%

Generated by 🐒 cobertura-action against fc7890b

@MikeDombo
Copy link
Member

@jcosentino11
Copy link
Member Author

@jcosentino11
Copy link
Member Author

Check on the UAT failure as well. https://github.com/aws-greengrass/aws-greengrass-shadow-manager/actions/runs/6475607422/job/17582906225?pr=196

Looks unrelated to this PR

Given my device is registered as a Thing....................................passed
--
And my device is running Greengrass.........................................passed
And I start an assertion server.............................................passed
Given I create a Greengrass deployment with components......................failed
And I deploy the Greengrass deployment configuration........................skipped
Then the Greengrass deployment is COMPLETED on the device after 3 minutes...skipped
Then I verify the aws.greengrass.ShadowManager component is RUNNING using the greengrass-cli.skipped
When I install the component ShadowReactiveComponentPing from local store with configuration.skipped
And I install the component ShadowComponentPing from local store with configuration.skipped
Then the local Greengrass deployment is SUCCEEDED on the device after 120 seconds.skipped
When I install the component ShadowComponentPong from local store with configuration.skipped
Then the local Greengrass deployment is SUCCEEDED on the device after 120 seconds.skipped
 
StackTrace:
java.lang.NoSuchMethodError: 'software.amazon.awssdk.services.s3.internal.endpoints.S3EndpointResolverContext$Builder software.amazon.awssdk.services.s3.internal.endpoints.S3EndpointResolverContext$Builder.endpointOverridden(boolean)'
at software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor.modifyHttpRequest(EndpointAddressInterceptor.java:47)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:90)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:157)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:83)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:151)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.s3.DefaultS3Client.headBucket(DefaultS3Client.java:5262)
at com.aws.greengrass.testing.resources.s3.S3Lifecycle.bucketExists(S3Lifecycle.java:38)
at com.aws.greengrass.testing.modules.AWSResourcesCleanupModule$CleanupInterceptor.invoke(AWSResourcesCleanupModule.java:89)
at com.aws.greengrass.testing.component.RecipeComponentPreparationService.lambda$getOrCreateBucket$1(RecipeComponentPreparationService.java:130)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at com.aws.greengrass.testing.component.RecipeComponentPreparationService.getOrCreateBucket(RecipeComponentPreparationService.java:128)
at com.aws.greengrass.testing.component.RecipeComponentPreparationService.prepare(RecipeComponentPreparationService.java:187)
at com.aws.greengrass.testing.component.CompositeComponentPreparationService.lambda$prepare$0(CompositeComponentPreparationService.java:31)
at java.base/java.util.Optional.map(Optional.java:265)
at com.aws.greengrass.testing.component.CompositeComponentPreparationService.prepare(CompositeComponentPreparationService.java:29)
at com.aws.greengrass.testing.features.DeploymentSteps.lambda$parseComponentNamesAndPrepare$1(DeploymentSteps.java:353)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at com.aws.greengrass.testing.features.DeploymentSteps.parseComponentNamesAndPrepare(DeploymentSteps.java:330)
at com.aws.greengrass.testing.features.DeploymentSteps.createDeployment(DeploymentSteps.java:123)
at ✽.I create a Greengrass deployment with components(classpath:greengrass/features/shadow-1.feature:10)

@MikeDombo
Copy link
Member

aws-greengrass/aws-greengrass-testing#226
Looks like SDK version mismatch due to this update.

@jcosentino11
Copy link
Member Author

May be related to aws-greengrass/aws-greengrass-testing#226, UAT runs after this have been failing

@MikeDombo MikeDombo merged commit 6989d05 into main Oct 11, 2023
2 checks passed
@MikeDombo MikeDombo deleted the clear-edge-case branch October 11, 2023 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants