Merge branch 'master' into zaleslaw/customdb

.github/README.md

@@ -0,0 +1,33 @@
+## GitHub Actions
+
+While publishing and testing of the library is handled by [JetBrains TeamCity](https://www.jetbrains.com/teamcity/),
+there are some CI operations that are handled by GitHub actions instead.
+
+### Publishing Docs
+
+The building of the documentation website in [docs](../docs), and the publishing of it along
+with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). Careful: This action replaces the entire contents
+of the documentation website.
+
+### Gradle Wrapper Validation
+
+We run the [Gradle Wrapper Validation action](https://github.com/gradle/wrapper-validation-action)
+using a [GitHub Action](./workflows/gradle-wrapper-validation.yml) as well.
+This action validates the checksums of all Gradle Wrapper JAR files present in the repository and
+fails if any unknown Gradle Wrapper JAR files are found.
+
+### Auto-commit generated code
+
+Anytime the source code changes on [master](https://github.com/Kotlin/dataframe/tree/master),
+this [GitHub Action](./workflows/generated-sources-master.yml) makes sure
+[`processKDocsMain`](../KDOC_PREPROCESSING.md),
+and `korro` are run. If there have been any changes in either [core/generated-sources](../core/generated-sources) or
+[docs/StardustDocs/snippets](../docs/StardustDocs/snippets), these are auto-committed to the branch, to keep
+it up to date.
+
+### Show generated code in PR
+
+To make sure no unexpected code is auto-committed to [master](https://github.com/Kotlin/dataframe/tree/master),
+this [GitHub Action](./workflows/generated-sources.yml) runs the same code-generating tasks but on a separate branch.
+If there are changes, it will leave a message in the PR, informing you about the changes that will be done to the master
+branch if this PR were merged.

.github/workflows/generated-sources.yml

@@ -1,4 +1,8 @@
-name: Show generated code in PR
+name: Preview Generated Code
+
+permissions:
+  contents: write
+  statuses: write
 
 on:
   pull_request:
@@ -10,16 +14,15 @@ on:
       - ready_for_review
 
 jobs:
-  build:
+  build_preview_generated_code:
+    name: Build Preview of Generated Code
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
+      cancel-in-progress: true
     runs-on: ubuntu-latest
     env:
       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
-      - name: Cancel Previous Runs
-        uses: styfle/[email protected]
-        with:
-          access_token: ${{ github.token }}
-
       - name: Checkout repository
         uses: actions/checkout@v4
 
@@ -39,7 +42,7 @@ jobs:
 
       - name: Check for changes in generated sources
         id: git-diff
-        run: echo "::set-output name=changed::$(if git diff --quiet './core/generated-sources' './docs/StardustDocs/snippets' './docs/StardustDocs/topics'; then echo 'false'; else echo 'true'; fi)"
+        run: echo "changed=$(if git diff --quiet './core/generated-sources' './docs/StardustDocs/snippets' './docs/StardustDocs/topics'; then echo 'false'; else echo 'true'; fi)" >> $GITHUB_OUTPUT
 
       - name: Commit and push if changes
         id: git-commit
@@ -49,44 +52,21 @@ jobs:
           git add './core/generated-sources' './docs/StardustDocs/snippets' './docs/StardustDocs/topics'
           git commit -m "Update generated sources with recent changes"
           git push origin generated-sources/docs-update-${{ github.run_number }}
-          echo "::set-output name=commit::$(git rev-parse HEAD)"
-
-      - name: Remove old comments
-        uses: actions/github-script@v7
-        if: steps.git-diff.outputs.changed == 'true'
-        with:
-          # language=js
-          script: |
-            const issue_number = context.issue.number;
-            const {owner, repo} = context.repo;
-            
-            const comments = await github.rest.issues.listComments({
-                issue_number,
-                owner,
-                repo,
-            });
-            
-            const botComments = comments.data.filter(
-                (comment) => comment.user.login === 'github-actions[bot]'
-            );
-            
-            for (const comment of botComments) {
-                await github.rest.issues.deleteComment({
-                    comment_id: comment.id,
-                    owner,
-                    repo,
-                });
-            }
+          echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
 
-      - name: Add comment to PR
+      - name: Update status to PR
         uses: actions/github-script@v7
         if: steps.git-diff.outputs.changed == 'true'
         with:
           # language=js
           script: |
-            github.rest.issues.createComment({
-              issue_number: context.issue.number,
+            await github.rest.repos.createCommitStatus({
               owner: context.repo.owner,
               repo: context.repo.repo,
-              body: "Generated sources will be updated after merging this PR.\nPlease inspect the changes in [here](https://github.com/${{ github.repository }}/commit/${{ steps.git-commit.outputs.commit }}).",
+              sha: context.payload.pull_request.head.sha,
+              state: "success",
+              target_url: "https://github.com/${{ github.repository }}/commit/${{ steps.git-commit.outputs.commit }}",
+              context: "Generated sources will change, merging this PR",
+              description: "Check 'Details' to inspect changes ->"
             });
+            

CONTRIBUTING.md

@@ -60,7 +60,7 @@ so do familiarize yourself with the following guidelines.
 4. The contributor answers the comments or fixes the proposed PR.
 5. The reviewer marks the PR with the word "LGTM."
 6. The maintainer could suggest merging the `master` branch to the PR branch a few times due to changes in the `master` branch.
-7. If the PR influences generated code/samples, a bot will inform about this in the PR comments.
+7. If the PR influences generated code/samples, a bot will inform about this in the PR checks.
 8. The maintainer runs TeamCity builds (unit tests and examples as integration tests).
 9. TeamCity writes the result (passed or not passed) to the PR checks at the bottom of the proposed PR.
 10. If it is possible, maintainers share the details of the failed build with the contributor.

KDOC_PREPROCESSING.md

@@ -58,7 +58,7 @@ This can be seen in action in the `core:processKDocsMain` and `core:changeJarTas
 To make sure the generated sources can be seen and reviewed on GitHub,
 since [PR #731](https://github.com/Kotlin/dataframe/pull/731),
 there's a [GitHub action](.github/workflows/generated-sources.yml) that runs the `core:processKDocsMain` task and
-comments the results on the PR. After a PR is merged, [another action](.github/workflows/generated-sources-master.yml)
+shows the results in the PR checks. After a PR is merged, [another action](.github/workflows/generated-sources-master.yml)
 runs on the master branch and commits the generated sources automatically.
 This way, the generated sources are always up to date with the latest changes in the code.
 This means you don't have to run and commit the generated sources yourself, though it's

core/README.md

@@ -0,0 +1,49 @@
+## :core
+
+This is the core of the library, published as the `dataframe-core` package.
+It contains the DataFrame API and its implementation, as well as plenty of JUnit tests.
+
+I/O operations are split off into other modules, like [:dataframe-excel](../dataframe-excel),
+or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations
+(see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)).
+
+At the moment, these integrations are still part of the `:core` module:
+
+- csv/tsv
+- html
+- json
+- jupyter
+
+### Korro code samples
+
+Code samples for the documentation website reside
+in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples that depend solely on `dataframe-core`),
+and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
+and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
+by [Korro](https://github.com/devcrocod/korro).
+
+### Explainer dataframes
+
+Aside from code samples, `@TransformDataFrameExpressions` annotated test functions also generate sample
+dataframe HTML files that can be used as iFrames on the documentation website.
+They are tested, generated, and copied over to [docs/StardustDocs/snippets](../docs/StardustDocs/snippets) by
+our "explainer" [plugin callback proxy](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
+which hooks into [the TestBase class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
+retrieves the intermediate DataFrame expressions thanks to our "explainer" compiler plugin
+[:plugins:expressions-converter](../plugins/expressions-converter).
+
+We can also generate "normal" DataFrame samples for the website. This can be done using the
+[OtherSamples class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
+HTML files will be stored in [docs/StardustDocs/snippets/manual](../docs/StardustDocs/snippets/manual).
+
+### KDoc Preprocesor
+
+The code you're working on needs to be edited in [src](src), but the KDocs are processed by the
+[KDoc Preprocessor](https://github.com/Jolanrensen/docProcessorGradlePlugin) when the project is published (or the task
+is run manually). The generated sources with adjusted KDocs will be overwritten
+in [generated-sources](generated-sources).
+See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information.
+
+KDocs can also be exported to HTML, for them to be reused on the website.
+Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
+[docs/StardustDocs/snippets/kdocs](../docs/StardustDocs/snippets/kdocs).