Adding configs related to DCLM #663
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM. Can you run pre-commit checks to fix the pre-commit issue? I think it is just some formatting issue. |
dlwh
reviewed
Jul 19, 2024
src/levanter/models/llama.py
Outdated
| @@ -64,6 +65,7 @@ class LlamaConfig(HFCompatConfig): | |||
| activation_function: str = "silu" | |||
| initializer_range: float = 0.02 | |||
| layer_norm_epsilon: float = 1e-5 | |||
| z_loss_weight: float = 0.0 | |||
There was a problem hiding this comment.
i would rather this not be a property of the model's config but an option on TrainLmConfig, and define a loss function in train_lm and pass it into trainer
src/levanter/models/lm_model.py
Outdated
| loss = cross_entropy_loss( | ||
| logits, self.Vocab, target_y, reduction, reduction_axis=reduction_axis, where=example.loss_mask | ||
| ) | ||
| if hasattr(self.config, "z_loss_weight") and self.config.z_loss_weight > 0: |
There was a problem hiding this comment.
really don't like using this here. much cleaner to just pull out the loss function
dlwh
reviewed
Jul 25, 2024
src/levanter/main/train_lm.py
Outdated
| from levanter.utils.jax_utils import parameter_count | ||
|
|
||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class ModuleComputeZLoss(ComputeLossFunction[M, X]): |
There was a problem hiding this comment.
i still don't like this but I think I can't really articulate what I want. i'm gonna push a change to my fork
* refactor queued-resources * fix multislice * add auto tear down * reuse docker image * tiny fix * switch to concurrent executor for parallel subprocesses & small fix & logs
* Add llama 1b with fineweb txt * replace with 50 fineweb urls * wip * revert many of the changes, which seems to fix the crashing * revert many of the changes, which seems to fix the crashing * remove now-unused option * cleanup * cleanup * sigh * Adding changes for dclm --------- Co-authored-by: Ivan Zhou <[email protected]> Co-authored-by: Abhinav Garg <[email protected]>
Bumps [ray[default]](https://github.com/ray-project/ray) from 2.32.0 to 2.34.0. - [Release notes](https://github.com/ray-project/ray/releases) - [Commits](ray-project/ray@ray-2.32.0...ray-2.34.0) --- updated-dependencies: - dependency-name: ray[default] dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* wandb seems to be broken in latest release * oops * what?
* add mounting dir * minor fix * support abs and rel path * add docs * refactor to extra context * minor fix docs * minor fix * modify docs
1. num_tpus=1 is actually a bad idea because Ray will mask out the other tpus 2. force non-docker workloads to run in a separate process for stability
…ards again, (re)add configuration metadata to cache (#752) Co-authored-by: Ahmed Ahmed <[email protected]>
Pulls in the New Mixture Features Into Audio Space! Tested that this fixes the previous epoching errors in the whisper_tiny config.
Co-authored-by: David Hall <[email protected]>
…g batches instead of a ray actor/task (#757) About a 5x speedup. Memory usage isn't super well controlled in mixtures and that needs some work
… head node, add code to change max size of actor pool
#762) This is marginally slower, but pile now builds fine on a v4-32, which is an improvement.
This PR creates a `ParquetDataSource` class to support loading `.parquet` files. Closes #763
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DCLM 7B related configs