Fix issue with setting of param_group for the DPOptimizer wrapper #660
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Jul 30, 2024
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameter of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
620356b
to
96d5c6b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Jul 30, 2024
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameter of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
96d5c6b
to
705038f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Jul 31, 2024
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameter of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
705038f
to
913bfeb
Compare
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
913bfeb
to
1b4c676
Compare
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Aug 1, 2024
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameters of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Aug 1, 2024
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameters of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
1b4c676
to
265f306
Compare
…izer wrapper (pytorch#660) Summary: Pull Request resolved: pytorch#660 Fix for github issue # [649](pytorch#649) **Background**: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. `param_groups`, `state`, `defaults` are parameters of DPOPtimizer that store all parameters related to the learning algorithm, including privacy-related parameters. **Issue**: Previously, DPOptimizer passed `param_groups`, `state`, `defaults` simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters). **Fix**: In this fix, we use the property decorator to ensure that the 3 parameters remain the same between DPOptimizer and Optimizer. Differential Revision: D60453849
|
This pull request was exported from Phabricator. Differential Revision: D60453849 |
iden-kalemaj
force-pushed
the
export-D60453849
branch
from
265f306
to
91c9b1b
Compare
|
This pull request has been merged in a059670. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Background: DPOptimizer is a wrapper for the original non-DP Optimizer selected by the user. Param_groups stores all parameters related to the learning algorithm, including privacy-related parameters.
Issue: Previously, DPOptimizer passed param_groups simply by reference. Thus another object can update param_groups for the DPOptimizer, while neglecting to update such parameters for the original Optimizer. The issue is reflected e.g., in the LR (learning rate scheduler) where the learning rate looks as if its being updated for the DPOptimizer, but it is not actually updated for the original Optimizer (the one that matters).
Fix: In this fix, we set param_groups in DPOptimizer using property, which ensures that param_groups is updated both for DPOptimizer and Optimizer.
Differential Revision: D60453849