Feat/sort FSRSItem by length to speed up training (#32)

* sort FSRSItem by length to speed up training * correct shape of logits * cargo fmt * Create batches with random seq_len * implement BatchShuffledDataset * cargo fmt * rename batch_shuffle & add English comments * Use an SQL sort and .group_by() for separating by card id * We don't need InMemDataset * Move all tests into test modules This allows easily running all tests in one file at once, and allows tests to share code that is not used in production. Also removed the duplicate test_next_stability/difficulty tests. * Remove redundant test_ prefix/suffix * Apply cosine_annealing patch from Asuka to fix test Co-authored-by: Asuka Minato <[email protected]> * Run tests in CI, except training Co-authored-by: Asuka Minato <[email protected]> * Limit checks to pull requests; bust cache --------- Co-authored-by: Damien Elmes <[email protected]> Co-authored-by: Asuka Minato <[email protected]>
open-spaced-repetition · Aug 28, 2023 · ee7f6be · ee7f6be
1 parent ed5b7b4
commit ee7f6be
Show file tree

Hide file tree

Showing 12 changed files with 601 additions and 501 deletions.
diff --git a/.github/workflows/check.sh b/.github/workflows/check.sh
@@ -9,3 +9,9 @@ cargo fmt --check || (
 )
 
 cargo clippy -- -Dwarnings
+
+install -d tests/data/
+pushd tests/data/
+wget https://github.com/open-spaced-repetition/fsrs-optimizer-burn/files/12394182/collection.anki21.zip
+unzip *.zip
+SKIP_TRAINING=1 cargo test
diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml
@@ -1,6 +1,5 @@
 name: Check code
 on:
- push:
  pull_request:
 
 jobs:
@@ -16,9 +15,9 @@ jobs:
  ~/.cargo/registry
  ~/.cargo/git
  target
- key: ${{ runner.os }}-rust-release-v5-${{ hashFiles('Cargo.lock') }}
+ key: ${{ runner.os }}-rust-release-v6-${{ hashFiles('Cargo.lock') }}
  restore-keys: |
- ${{ runner.os }}-rust-release-v5
+ ${{ runner.os }}-rust-release-v6
  ${{ runner.os }}-rust-release
 
  - name: Run checks

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -15,3 +15,4 @@ rusqlite = { version = "0.29.0" }
 chrono = "0.4.26"
 chrono-tz = "0.8.3"
 itertools = "0.11.0"
+rand = "0.8.5"