Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Choose a format for publishing harmonized datasets #15

Closed
gaurav opened this issue Sep 17, 2020 · 6 comments
Closed

Choose a format for publishing harmonized datasets #15

gaurav opened this issue Sep 17, 2020 · 6 comments
Assignees
Labels
in progress Somebody is working on this right now

Comments

@gaurav
Copy link

gaurav commented Sep 17, 2020

We currently have an initial, incomplete set of harmonized data from some IDC example datasets (#4), and need to chose a format to store and publish this data in a way that maintains provenance information. Here are some possible formats:

Please feel free to add any other formats we should look at!

This issue covers comparing these formats for our purposes and to translate the incomplete harmonized data into that format as a potential exemplar.

@gaurav gaurav added the in progress Somebody is working on this right now label Sep 17, 2020
@gaurav gaurav self-assigned this Sep 17, 2020
@gaurav gaurav added this to the Phase 2 - Quarter 3 milestone Sep 17, 2020
@fedorov
Copy link

fedorov commented Oct 1, 2020

Before we answer this question - do you know where will the resulting files live within the CRDC? What resource is going to be responsible for supporting users in querying the data stored in those files?

@balhoff
Copy link

balhoff commented Oct 2, 2020

I think the answer to that lies in the ultimate architecture for CDA. Definitely the querying part; not sure about storage of the files.

@fedorov
Copy link

fedorov commented Oct 2, 2020

Since CDA was mentioned - adding @DavidPotCanuck who is one of the leads for CDA, in case he is interested to participate in this discussion.

@gaurav
Copy link
Author

gaurav commented Nov 12, 2020

I've made a preliminary conversion from the ISPY1 patient clinical subset to PFB by using a JSON file that records the mappings. You can download the PFB file, or have a look at their internal representation extracted with the PyPFB tool: the Avro schema, PFB metadata node, and data.

The good news is that PFB can be used to store textual, numerical and enumerated data, along with a reference to the caDSR CDE used for the mapping (from which we could map the harmonized values to NCIt concept identifiers). The bad news is that Avro fields and enum values are both restricted to the regex [A-Za-z_][A-Za-z0-9_]*. The PFB developers have come up with a simple representation for inserting Unicode characters (using e.g. _00A9_ to represent U+00A9 = ©), but PyPFB doesn't consistently translate those field names into and out of JSON at the moment. We could probably fix this without too much of a problem, assuming that this is a bug in PyPFB and not a misinterpretation on my part. PFB doesn't currently have a facility for storing verbatim values, so we do lose the actual values recorded in the original data. We might be able to propose a change to PFB to record this information if it is important to us. Another possibility is that we could store all the mapping information in another part of the Avro file, leave the verbatim values as-is and only convert them into harmonized values when exporting it.

I'm going to leave this conversion as-is for now and move on to trying to convert the same dataset into the CEDAR instance format so we can do a side-by-side comparison between these two formats.

@gaurav
Copy link
Author

gaurav commented Jul 6, 2021

We'll have some example output files in cancerDHC/example-data#10 -- we can use that to figure out which format work best for our needs.

@gaurav gaurav added this to the Phase 3 - Quarter 3 (2021) milestone Aug 16, 2021
@gaurav
Copy link
Author

gaurav commented Oct 1, 2021

For our immediate needs, the LinkML instance format (in YAML) seems to be a good representation, and has been developed into some exemplars by the CCDH Data Model Harmonization team as part of the CCDH Pilot. Future formats will probably be supported by adding generators to LinkML, so that data from any LinkML model can be converted into that format (e.g. see the issue tracking an Avro/PFB generator for LinkML). Given that, I think we can close this issue until specific use-cases emerge from CDA and the CRDC nodes.

@gaurav gaurav closed this as completed Oct 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Somebody is working on this right now
Projects
None yet
Development

No branches or pull requests

3 participants