From 62550baf20b221c44e96311ec0891a1c3ce6e42b Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Tue, 1 Oct 2024 14:09:58 +0200 Subject: [PATCH 1/6] small clarification about keywordGenerator in :core --- core/build.gradle.kts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/core/build.gradle.kts b/core/build.gradle.kts index 02d9eb863..3f1963766 100644 --- a/core/build.gradle.kts +++ b/core/build.gradle.kts @@ -14,13 +14,15 @@ plugins { alias(serialization) alias(jupyter.api) alias(korro) - alias(keywordGenerator) alias(kover) alias(ktlint) alias(docProcessor) alias(simpleGit) alias(buildconfig) + // generates keywords using the :generator module + alias(keywordGenerator) + // dependence on our own plugin alias(dataframe) From 0386c4ed51bfc636599ace06479ec2d9f3a20849 Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Tue, 1 Oct 2024 14:10:47 +0200 Subject: [PATCH 2/6] Wrote an initial README file to each module/important folder of the project to explain what it does. --- .github/README.md | 32 +++++++++++++++ core/README.md | 49 +++++++++++++++++++++++ dataframe-arrow/README.md | 8 ++++ dataframe-excel/README.md | 8 ++++ dataframe-jdbc/README.md | 7 ++++ dataframe-openapi/README.md | 8 ++++ docs/README.md | 36 +++++++++++++++++ examples/README.md | 2 + generator/README.md | 14 +++++++ plugins/README.md | 28 +++++++++++++ plugins/dataframe-gradle-plugin/README.md | 11 +++++ plugins/expressions-converter/README.md | 12 ++++++ plugins/kotlin-dataframe/README.md | 10 +++++ plugins/symbol-processor/README.md | 12 ++++++ tests/README.md | 8 ++++ 15 files changed, 245 insertions(+) create mode 100644 .github/README.md create mode 100644 core/README.md create mode 100644 dataframe-arrow/README.md create mode 100644 dataframe-excel/README.md create mode 100644 dataframe-jdbc/README.md create mode 100644 dataframe-openapi/README.md create mode 100644 docs/README.md create mode 100644 generator/README.md create mode 100644 plugins/README.md create mode 100644 plugins/dataframe-gradle-plugin/README.md create mode 100644 plugins/expressions-converter/README.md create mode 100644 plugins/kotlin-dataframe/README.md create mode 100644 plugins/symbol-processor/README.md create mode 100644 tests/README.md diff --git a/.github/README.md b/.github/README.md new file mode 100644 index 000000000..a5d4dd5c9 --- /dev/null +++ b/.github/README.md @@ -0,0 +1,32 @@ +## GitHub Actions + +While publishing and testing of the library is handled by [JetBrains TeamCity](https://www.jetbrains.com/teamcity/), +there are some CI operations that are handled by GitHub actions instead. + +### Publishing Docs + +The building of the documentation website in [docs](../docs), and the publishing of it along +with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). + +### Gradle Wrapper Validation + +We run the [Gradle Wrapper Validation action](https://github.com/gradle/wrapper-validation-action) +using a [GitHub Action](./workflows/gradle-wrapper-validation.yml) as well. +This action validates the checksums of all Gradle Wrapper JAR files present in the repository and +fails if any unknown Gradle Wrapper JAR files are found. + +### Auto-commit generated code + +Anytime the source code changes on [master](https://github.com/Kotlin/dataframe/tree/master), +this [GitHub Action](./workflows/generated-sources-master.yml) makes sure +[`processKDocsMain`](../KDOC_PREPROCESSING.md), +and `korro` are run. If there have been any changes in either [core/generated-sources](../core/generated-sources) or +[docs/StardustDocs/snippets](../docs/StardustDocs/snippets), these are auto-committed to the branch, to keep +it up to date. + +### Show generated code in PR + +To make sure no unexpected code is auto-committed to [master](https://github.com/Kotlin/dataframe/tree/master), +this [GitHub Action](./workflows/generated-sources.yml) runs the same code-generating tasks but on a separate branch. +If there are changes, it will leave a message in the PR, informing you about the changes that will be done to the master +branch if this PR were merged. diff --git a/core/README.md b/core/README.md new file mode 100644 index 000000000..00ba2029b --- /dev/null +++ b/core/README.md @@ -0,0 +1,49 @@ +## :core + +This is the core of the library, published as the `dataframe-core` package. +It contains the DataFrame API and its implementation, as well as plenty of JUnit tests. + +I/O operations are split off into other modules, like [:dataframe-excel](../dataframe-excel), +or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations +(see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)). + +Operations likely to be split off in the future include: + +- csv/tsv +- html +- json +- jupyter + +### Korro code samples + +Code samples for the documentation website reside +in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics) +by [Korro](https://github.com/devcrocod/korro). + +### Explainer dataframes + +Aside from code samples, `@TransformDataFrameExpressions` annotated test functions also generate sample +dataframe HTML files that can be used as iFrames on the documentation website. +They are tested, generated, and copied over to [docs/StardustDocs/snippets](../docs/StardustDocs/snippets) by +our "explainer" [plugin callback proxy](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer), +which hooks into [the TestBase class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and +retrieves the intermediate DataFrame expressions thanks to our "explainer" compiler plugin +[:plugins:expressions-converter](../plugins/expressions-converter). + +We can also generate "normal" DataFrame samples for the website. This can be done using the +[OtherSamples class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated +HTML files will be stored in [docs/StardustDocs/snippets/manual](../docs/StardustDocs/snippets/manual). + +### KDoc Preprocesor + +The code you're working on needs to be edited in [src](src), but the KDocs are processed by the +[KDoc Preprocessor](https://github.com/Jolanrensen/docProcessorGradlePlugin) when the project is published (or the task +is run manually). The generated sources with adjusted KDocs will be overwritten +in [generated-sources](generated-sources). +See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information. + +KDocs can also be exported to HTML, for them to be reused on the website. +Elements annotated with `@ExportAsHtml` will have their generated content be copied over to +[docs/StardustDocs/snippets/kdocs](../docs/StardustDocs/snippets/kdocs). diff --git a/dataframe-arrow/README.md b/dataframe-arrow/README.md new file mode 100644 index 000000000..96ffe5960 --- /dev/null +++ b/dataframe-arrow/README.md @@ -0,0 +1,8 @@ +## :dataframe-arrow + +This module, published as `dataframe-arrow`, contains all logic and tests for DataFrame to be able to work with +Apache Arrow. + +See [Read Apache Arrow formats](https://kotlin.github.io/dataframe/read.html#read-apache-arrow-formats) and +[Writing to Apache Arrow formats](https://kotlin.github.io/dataframe/write.html#writing-to-apache-arrow-formats) +for more information about how to use it. diff --git a/dataframe-excel/README.md b/dataframe-excel/README.md new file mode 100644 index 000000000..4dc59671c --- /dev/null +++ b/dataframe-excel/README.md @@ -0,0 +1,8 @@ +## :dataframe-excel + +This module, published as `dataframe-excel`, contains all logic and tests for DataFrame to be able to work with +Excel files. + +See [Read from Excel](https://kotlin.github.io/dataframe/read.html#read-from-excel) and +[Write to Excel spreadsheet](https://kotlin.github.io/dataframe/write.html#write-to-excel-spreadsheet) +for more information about how to use it. diff --git a/dataframe-jdbc/README.md b/dataframe-jdbc/README.md new file mode 100644 index 000000000..e0e9092a5 --- /dev/null +++ b/dataframe-jdbc/README.md @@ -0,0 +1,7 @@ +## :dataframe-jdbc + +This module, published as `dataframe-jdbc`, contains all logic and tests for DataFrame to be able to work with +JDBC data sources. + +See [Read from SQL databases](https://kotlin.github.io/dataframe/readsqldatabases.html) for more information +about how to use it. diff --git a/dataframe-openapi/README.md b/dataframe-openapi/README.md new file mode 100644 index 000000000..8fda2c24d --- /dev/null +++ b/dataframe-openapi/README.md @@ -0,0 +1,8 @@ +## :dataframe-openapi + +This module, published as `dataframe-openapi` contains all logic and tests for DataFrame to be able to import +OpenAPI specifications as auto-generated data schemas. + +See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and +[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html) +for more information about how to use it. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..81a9b13d7 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,36 @@ +## The Documentation Website + +This folder holds the source code of our documentation website: +[kotlin.github.io/dataframe](https://kotlin.github.io/dataframe). + +It's built using [WriterSide](https://www.jetbrains.com/writerside/) and published +by a [Github Action](../.github/workflows/main.yml). +The file structure mirrors the default WriterSide structure with some small exceptions. + +Images all README files can be stored in [docs/imgs](./imgs). + +### Explainer dataframes +`@TransformDataFrameExpressions` annotated test functions generate sample +dataframe HTML files that can be used as iFrames on the documentation website. +They are tested, generated, and copied over to [docs/StardustDocs/snippets](./StardustDocs/snippets) by +our "explainer" [plugin callback proxy](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer), +which hooks into [the TestBase class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and +retrieves the intermediate DataFrame expressions thanks to +our "explainer" compiler plugin [:plugins:expressions-converter](../plugins/expressions-converter). + +We can also generate "normal" DataFrame samples for the website. This can be done using the +[OtherSamples class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated +HTML files will be stored in [docs/StardustDocs/snippets/manual](./StardustDocs/snippets/manual). + +### KDoc Preprocesor +KDocs can also be exported to HTML, for them to be reused on the website. +Elements annotated with `@ExportAsHtml` will have their generated content be copied over to +[docs/StardustDocs/snippets/kdocs](./StardustDocs/snippets/kdocs). + +### Korro code samples +Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +and they are copied over to Markdown files in [docs/StardustDocs/topics](./StardustDocs/topics) +by [Korro](https://github.com/devcrocod/korro). + + diff --git a/examples/README.md b/examples/README.md index 277dc0ef6..feb0158e2 100644 --- a/examples/README.md +++ b/examples/README.md @@ -44,3 +44,5 @@ ___ --- * 0.14.0 Dev Feature Overview [notebook](notebooks/feature_overviews/0.14.0/new_features.ipynb) Overview of new features available in 0.14.0-dev + +These example notebooks are tested by the [:core module](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/jupyter). diff --git a/generator/README.md b/generator/README.md new file mode 100644 index 000000000..32e2ec994 --- /dev/null +++ b/generator/README.md @@ -0,0 +1,14 @@ +## :generator + +This module holds a little Gradle plugin whose sole purpose is to provide +[:core](../core) with the `generateKeywordsSrc` task. + +This task, generates three enum classes: `HardKeywords`, `ModifierKeywords`, and `SoftKeywords`. +These enums together contain all restricted Kotlin keywords to be taken into account when generating our own +code in Notebooks or any of our [plugins](../plugins). Words like "package", "fun", "suspend", etc... + +As the Kotlin language can change over time, this task ensures that any changes to the language +will be reflected in our code generation. + +This module can probably be moved under [:plugins](../plugins): +[Issue #899](https://github.com/Kotlin/dataframe/issues/899). diff --git a/plugins/README.md b/plugins/README.md new file mode 100644 index 000000000..4626e5e94 --- /dev/null +++ b/plugins/README.md @@ -0,0 +1,28 @@ +## :plugins + +This folder holds all our Gradle- and Kotlin Compiler plugins: + +### [:plugins:dataframe-gradle-plugin](./dataframe-gradle-plugin) +The Gradle plugin for DataFrame that can generate data schemas from a data sample using the simple +task `dataframes {}`. It uses [:plugins:symbol-processor](./symbol-processor) to generate column accessors. + +### [:plugins:symbol-processor](./symbol-processor) +The KSP plugin that can generate data schemas from a data sample using the `@file:ImportDataSchema` annotation. +It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate +column accessors for in the form of extension properties. + +### [:plugins:kotlin-dataframe](./kotlin-dataframe) +The Kotlin 2.x Compiler plugin of DataFrame. A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704) +plugin for your Gradle project that can generate on-the-fly column accessors for the compiler and IDE even without +having to provide data schemas! + +### [:plugins:expressions-converter](./expressions-converter) +A small Kotlin Compiler plugin that provides intermediate expressions of DataFrame +operation chains, used internally by [:core](../core) to generate "explainer dataframes" on the documentation website. + +### [generator](../generator) +A small Gradle plugin that is used internally to generate enums with restricted Kotlin keywords for the +[:core](../core) module. + +This module can probably be moved under [:plugins](../plugins): +[Issue #899](https://github.com/Kotlin/dataframe/issues/899). diff --git a/plugins/dataframe-gradle-plugin/README.md b/plugins/dataframe-gradle-plugin/README.md new file mode 100644 index 000000000..6b4a56f24 --- /dev/null +++ b/plugins/dataframe-gradle-plugin/README.md @@ -0,0 +1,11 @@ +## :plugins:dataframe-gradle-plugin + +This module holds the Gradle plugin for DataFrame, published as "org.jetbrains.kotlinx.dataframe" on the +[Gradle Plugin Portal](https://plugins.gradle.org/plugin/org.jetbrains.kotlin.plugin.dataframe). + +This plugin can let the user generate data schemas from a data sample using the simple +Gradle task `dataframes {}`. It also provides an automatic dependency on [:plugins:symbol-processor](./symbol-processor) +to generate column accessors and support the `@file:ImportDataSchema` notation. + +Read more about how to use the Gradle plugin at +[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html). diff --git a/plugins/expressions-converter/README.md b/plugins/expressions-converter/README.md new file mode 100644 index 000000000..de341947f --- /dev/null +++ b/plugins/expressions-converter/README.md @@ -0,0 +1,12 @@ +## :plugins:expressions-converter + +This Kotlin Compiler plugin, used by [:core](../../core), can extract intermediate +DataFrame expressions from `@TransformDataFrameExpressions` annotated functions. + +It is used to generate sample "explainer dataframe" HTML files that can be used as iFrames on the documentation website. + +Annotated functions in [core/.../test/.../samples/api](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +are tested, generated, and copied over to [docs/StardustDocs/snippets](../../docs/StardustDocs/snippets) by +our "explainer" [plugin callback proxy](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer), +which hooks into [the TestBase class](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and +retrieves the intermediate DataFrame expressions thanks to this module. diff --git a/plugins/kotlin-dataframe/README.md b/plugins/kotlin-dataframe/README.md new file mode 100644 index 000000000..7d79c5831 --- /dev/null +++ b/plugins/kotlin-dataframe/README.md @@ -0,0 +1,10 @@ +## :plugins:kotlin-dataframe + +This module, published as "compiler-plugin-all", holds the Kotlin 2.x Compiler plugin of DataFrame. + +A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704) +plugin for your Gradle project that can generate on-the-fly column accessors for the compiler and IDE even without +having to provide data schemas! + +Details of how to build and run this for yourself will follow, but there is a +[demo project](https://github.com/koperagen/df-plugin-demo) you can try already. diff --git a/plugins/symbol-processor/README.md b/plugins/symbol-processor/README.md new file mode 100644 index 000000000..994577561 --- /dev/null +++ b/plugins/symbol-processor/README.md @@ -0,0 +1,12 @@ +## :plugins:symbol-processor + +This module holds the KSP plugin, published as the library "symbol-processor-all" that can generate data schemas from +a data sample using the `@file:ImportDataSchema` annotation. +It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate +column accessors for in the form of extension properties. + +If you use the [Gradle plugin](../dataframe-gradle-plugin), this module and KSP are added as a dependency automatically. + +Read more about how to use this at +[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html) and +[Extension Properties API](https://kotlin.github.io/dataframe/extensionpropertiesapi.html). diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 000000000..a4ed4db17 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,8 @@ +## :tests + +Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics) +by [Korro](https://github.com/devcrocod/korro). + +This module should likely be moved into [:core](../core): [Issue #898](https://github.com/Kotlin/dataframe/issues/898). From 846c9ad92f599f80946680b8a33166c0720cc3a2 Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Wed, 2 Oct 2024 12:09:55 +0200 Subject: [PATCH 3/6] small explanations about the samples/api split --- core/README.md | 4 ++-- docs/README.md | 5 +++-- tests/README.md | 7 ++++--- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/core/README.md b/core/README.md index 00ba2029b..60c079688 100644 --- a/core/README.md +++ b/core/README.md @@ -17,8 +17,8 @@ Operations likely to be split off in the future include: ### Korro code samples Code samples for the documentation website reside -in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), -and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples that depend solely on `dataframe-core`), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules) and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics) by [Korro](https://github.com/devcrocod/korro). diff --git a/docs/README.md b/docs/README.md index 81a9b13d7..5698d3486 100644 --- a/docs/README.md +++ b/docs/README.md @@ -28,8 +28,9 @@ Elements annotated with `@ExportAsHtml` will have their generated content be cop [docs/StardustDocs/snippets/kdocs](./StardustDocs/snippets/kdocs). ### Korro code samples -Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), -and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +(for samples that depend solely on `dataframe-core`), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules) and they are copied over to Markdown files in [docs/StardustDocs/topics](./StardustDocs/topics) by [Korro](https://github.com/devcrocod/korro). diff --git a/tests/README.md b/tests/README.md index a4ed4db17..ecaa2fe9d 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,8 +1,9 @@ ## :tests -Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api), -and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) +(for samples that depend solely on `dataframe-core`), +and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules) and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics) by [Korro](https://github.com/devcrocod/korro). -This module should likely be moved into [:core](../core): [Issue #898](https://github.com/Kotlin/dataframe/issues/898). +This module might be merged with [:core](../core): [Issue #898](https://github.com/Kotlin/dataframe/issues/898). From b1f6243ce1c401b62d6c40abf6156e0b95a06d85 Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Wed, 16 Oct 2024 14:14:21 +0200 Subject: [PATCH 4/6] updating readmes from feedback --- .github/README.md | 3 ++- core/README.md | 2 +- docs/README.md | 7 +++++-- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/README.md b/.github/README.md index a5d4dd5c9..48ae01de7 100644 --- a/.github/README.md +++ b/.github/README.md @@ -6,7 +6,8 @@ there are some CI operations that are handled by GitHub actions instead. ### Publishing Docs The building of the documentation website in [docs](../docs), and the publishing of it along -with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). +with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). Careful: This action replaces the entire contents +of the documentation website. ### Gradle Wrapper Validation diff --git a/core/README.md b/core/README.md index 60c079688..41517654e 100644 --- a/core/README.md +++ b/core/README.md @@ -7,7 +7,7 @@ I/O operations are split off into other modules, like [:dataframe-excel](../data or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations (see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)). -Operations likely to be split off in the future include: +At the moment, these data sources are still part of the `:core` module: - csv/tsv - html diff --git a/docs/README.md b/docs/README.md index 5698d3486..e33d3cce3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,10 @@ This folder holds the source code of our documentation website: It's built using [WriterSide](https://www.jetbrains.com/writerside/) and published by a [Github Action](../.github/workflows/main.yml). -The file structure mirrors the default WriterSide structure with some small exceptions. +The file structure largely mirrors the default WriterSide structure. +For instance, if you want to add a new page to the website, this needs to be stored as an `.md` file in the +[StardustDocs/topics](./StardustDocs/topics) folder, +and included in the [StardustDocs/d.tree](./StardustDocs/d.tree) file. Images all README files can be stored in [docs/imgs](./imgs). @@ -22,7 +25,7 @@ We can also generate "normal" DataFrame samples for the website. This can be don [OtherSamples class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated HTML files will be stored in [docs/StardustDocs/snippets/manual](./StardustDocs/snippets/manual). -### KDoc Preprocesor +### KDoc Preprocessor KDocs can also be exported to HTML, for them to be reused on the website. Elements annotated with `@ExportAsHtml` will have their generated content be copied over to [docs/StardustDocs/snippets/kdocs](./StardustDocs/snippets/kdocs). From edef57040aa47cd2b55ee718ae5f35ad03067543 Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Tue, 22 Oct 2024 11:39:08 +0200 Subject: [PATCH 5/6] updated openapi readmes --- dataframe-openapi-generator/README.md | 13 +++++++++++++ dataframe-openapi/README.md | 9 +++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) create mode 100644 dataframe-openapi-generator/README.md diff --git a/dataframe-openapi-generator/README.md b/dataframe-openapi-generator/README.md new file mode 100644 index 000000000..f65b1a73c --- /dev/null +++ b/dataframe-openapi-generator/README.md @@ -0,0 +1,13 @@ +## :dataframe-openapi + +This module, published as `dataframe-openapi-generator` contains all logic and tests for DataFrame to be able to import +OpenAPI specifications as auto-generated data schemas. This module is a sister module to +[`dataframe-openapi`](../dataframe-openapi): + +- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate + data schemas from OpenAPI specifications. +- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas. + +See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and +[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html) +for more information about how to use it. diff --git a/dataframe-openapi/README.md b/dataframe-openapi/README.md index 8fda2c24d..29a1341ca 100644 --- a/dataframe-openapi/README.md +++ b/dataframe-openapi/README.md @@ -1,7 +1,12 @@ ## :dataframe-openapi -This module, published as `dataframe-openapi` contains all logic and tests for DataFrame to be able to import -OpenAPI specifications as auto-generated data schemas. +This module, published as `dataframe-openapi` contains some functions to be able to use auto-generated +data schemas from OpenAPI specifications. This module is a sister module to +[`dataframe-openapi-generator`](../dataframe-openapi-generator): + +- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate + data schemas from OpenAPI specifications. +- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas. See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and [Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html) From 4d96ef1cc2400ffb05d692db6f8bd1b5acfc65eb Mon Sep 17 00:00:00 2001 From: Jolan Rensen Date: Fri, 25 Oct 2024 13:11:00 +0200 Subject: [PATCH 6/6] readme fixes based on feedback from reviews --- core/README.md | 2 +- dataframe-excel/README.md | 2 +- dataframe-openapi-generator/README.md | 5 +++-- dataframe-openapi/README.md | 3 ++- docs/README.md | 4 ++-- generator/README.md | 2 +- plugins/README.md | 4 ++-- plugins/kotlin-dataframe/README.md | 4 ++-- plugins/symbol-processor/README.md | 3 ++- 9 files changed, 16 insertions(+), 13 deletions(-) diff --git a/core/README.md b/core/README.md index 41517654e..cb07c1b44 100644 --- a/core/README.md +++ b/core/README.md @@ -7,7 +7,7 @@ I/O operations are split off into other modules, like [:dataframe-excel](../data or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations (see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)). -At the moment, these data sources are still part of the `:core` module: +At the moment, these integrations are still part of the `:core` module: - csv/tsv - html diff --git a/dataframe-excel/README.md b/dataframe-excel/README.md index 4dc59671c..31c6f7aea 100644 --- a/dataframe-excel/README.md +++ b/dataframe-excel/README.md @@ -1,7 +1,7 @@ ## :dataframe-excel This module, published as `dataframe-excel`, contains all logic and tests for DataFrame to be able to work with -Excel files. +`xls` and `xlsx` files. See [Read from Excel](https://kotlin.github.io/dataframe/read.html#read-from-excel) and [Write to Excel spreadsheet](https://kotlin.github.io/dataframe/write.html#write-to-excel-spreadsheet) diff --git a/dataframe-openapi-generator/README.md b/dataframe-openapi-generator/README.md index f65b1a73c..06deff05d 100644 --- a/dataframe-openapi-generator/README.md +++ b/dataframe-openapi-generator/README.md @@ -1,11 +1,12 @@ -## :dataframe-openapi +## :dataframe-openapi-generator This module, published as `dataframe-openapi-generator` contains all logic and tests for DataFrame to be able to import OpenAPI specifications as auto-generated data schemas. This module is a sister module to [`dataframe-openapi`](../dataframe-openapi): - `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate - data schemas from OpenAPI specifications. + data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the + `@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function. - `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas. See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and diff --git a/dataframe-openapi/README.md b/dataframe-openapi/README.md index 29a1341ca..435407a8a 100644 --- a/dataframe-openapi/README.md +++ b/dataframe-openapi/README.md @@ -5,7 +5,8 @@ data schemas from OpenAPI specifications. This module is a sister module to [`dataframe-openapi-generator`](../dataframe-openapi-generator): - `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate - data schemas from OpenAPI specifications. + data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the + `@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function. - `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas. See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and diff --git a/docs/README.md b/docs/README.md index e33d3cce3..633c7d7d0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,9 +3,9 @@ This folder holds the source code of our documentation website: [kotlin.github.io/dataframe](https://kotlin.github.io/dataframe). -It's built using [WriterSide](https://www.jetbrains.com/writerside/) and published +It's built using [Writerside](https://www.jetbrains.com/writerside/) and published by a [Github Action](../.github/workflows/main.yml). -The file structure largely mirrors the default WriterSide structure. +The file structure largely mirrors the default Writerside structure. For instance, if you want to add a new page to the website, this needs to be stored as an `.md` file in the [StardustDocs/topics](./StardustDocs/topics) folder, and included in the [StardustDocs/d.tree](./StardustDocs/d.tree) file. diff --git a/generator/README.md b/generator/README.md index 32e2ec994..fceabae25 100644 --- a/generator/README.md +++ b/generator/README.md @@ -10,5 +10,5 @@ code in Notebooks or any of our [plugins](../plugins). Words like "package", "fu As the Kotlin language can change over time, this task ensures that any changes to the language will be reflected in our code generation. -This module can probably be moved under [:plugins](../plugins): +This module will likely be moved under [:plugins](../plugins): [Issue #899](https://github.com/Kotlin/dataframe/issues/899). diff --git a/plugins/README.md b/plugins/README.md index 4626e5e94..2d1dd1dbd 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -8,8 +8,8 @@ task `dataframes {}`. It uses [:plugins:symbol-processor](./symbol-processor) to ### [:plugins:symbol-processor](./symbol-processor) The KSP plugin that can generate data schemas from a data sample using the `@file:ImportDataSchema` annotation. -It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate -column accessors for in the form of extension properties. +It is also used to generate column accessors for in the form of extension properties for +(both manually written- or generated) `@DataSchema` annotated classes/interfaces. ### [:plugins:kotlin-dataframe](./kotlin-dataframe) The Kotlin 2.x Compiler plugin of DataFrame. A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704) diff --git a/plugins/kotlin-dataframe/README.md b/plugins/kotlin-dataframe/README.md index 7d79c5831..7fbb5a6d2 100644 --- a/plugins/kotlin-dataframe/README.md +++ b/plugins/kotlin-dataframe/README.md @@ -3,8 +3,8 @@ This module, published as "compiler-plugin-all", holds the Kotlin 2.x Compiler plugin of DataFrame. A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704) -plugin for your Gradle project that can generate on-the-fly column accessors for the compiler and IDE even without +plugin for your Kotlin project that can generate on-the-fly column accessors for the compiler and IDE even without having to provide data schemas! -Details of how to build and run this for yourself will follow, but there is a +Details of how to use this compiler plugin for your own project will follow, but there is a [demo project](https://github.com/koperagen/df-plugin-demo) you can try already. diff --git a/plugins/symbol-processor/README.md b/plugins/symbol-processor/README.md index 994577561..83e085043 100644 --- a/plugins/symbol-processor/README.md +++ b/plugins/symbol-processor/README.md @@ -1,7 +1,8 @@ ## :plugins:symbol-processor This module holds the KSP plugin, published as the library "symbol-processor-all" that can generate data schemas from -a data sample using the `@file:ImportDataSchema` annotation. +a data sample using the `@file:ImportDataSchema` annotation. This annotation can be used as an alternative to +the `dataframes {}` syntax of the [Gradle plugin](../dataframe-gradle-plugin) that's declared closer to the source code. It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate column accessors for in the form of extension properties.