Can not generate 5 patients despite --generate.max_attempts_to_keep_patient=10000

[gthaker]

My keep.json is as shown below. I have simple "Or" of two conditions related to prostate. I want to generate 5 patients. (Later I hope to generate 5 more that would have "And" of these two conditions.

java -jar synthea-with-dependencies.jar -p 5 -a 50-80 -k keep.json -g M --generate.max_attempts_to_keep_patient=10000

takes a very long time and generates 4 patients for me. What might I be doing wrong here? It seems that, given that cancer happens to about 13 out of 100 males (over their lifetime), it should not be statistically too rare for this to be happening in generation logic?

Also, in output/fhir i get huge files for:

15048 lines in file: output/fhir/hospitalInformation1697759621010.json
17018 lines in file: output/fhir/practitionerInformation1697759621010.json

Is it possible to control number of practioners and hospitals that get generated? I have trouble uploading these files into my locally running HAPI FHIR server.

2023-10-20 12:10:34.114 [http-nio-8080-exec-6] WARN c.u.f.r.s.i.ExceptionHandlingInterceptor [ExceptionHandlingInterceptor.java:205] Failure during REST processing: ca.uhn.fhir.rest.server.exceptions.ResourceNotFoundException: HAPI-1091: Invalid match URL "Practitioner?identifier=http://hl7.org/fhir/sid/us-npi|9999985291" - No resources match this search

{
"name": "Generated Keep Module",
"states": {
"Initial": {
"type": "Initial",
"name": "Initial",
"conditional_transition": [
{
"transition": "Keep",
"condition": {
"condition_type": "Or",
"conditions": [
{
"condition_type": "Active Condition",
"codes": [
{
"system": "SNOMED-CT",
"code": "126906006",
"display": "Neoplasm of prostate"
}
]
},
{
"condition_type": "Active Condition",
"codes": [
{
"system": "SNOMED-CT",
"code": "65575008",
"display": "Biopsy of prostate"
}
]
}
]
}
},
{
"transition": "Terminal"
}
]
},
"Terminal": {
"type": "Terminal",
"name": "Terminal"
},
"Keep": {
"type": "Terminal",
"name": "Keep"
}
},
"gmf_version": 2
}

[dehall]

It looks like you're doing everything right given the population you want to create, but Prostate cancer in Synthea is modeled a little differently than other conditions. In this case it's only modeled for the Veteran population. Instead of ~13% of the male population you should be seeing it in a % of the Veteran population, so that's much smaller. For this specific case you may want to use the "veteran population override" to generate a population of only Veterans. To do that, add the flag --generate.veteran_population_override=true to your run command

Re: the number of records in hospital/practitionerInformation files, that sounds like a bug to me. These files are intended to contain only the hospitals and practitioners that actually provided care to the exported patients, but the way the keep module works is it generates a lot of patients and only exports some of them, so the hospital/provider info associated with the records that were thrown away is still present. I'll open an issue for that

[gthaker]

Thank you very much for a quick and detailed response. I ran the following command:

java -jar synthea-with-dependencies.jar -p 5 -a 50-80 -k keep.json -g M --generate.max_attempts_to_keep_patient=10000 --generate.veteran_population_override=true

Program again ran for a long time (> 30 minutes) and appears to have generated 6 patients (though I had asked for 5, perhaps some threads work in parallel so 6 is fine).

The patient records are large and I have not yet loaded them to my local HAPI FHIR server.

By the way, is there an official/approved/recommended python program that I can point to these files and have them uploaded in a 'batch' mode? Files are of course too large to cut/paste into HAPI FHIR server's text windows.

output/fhir:
total 67M
-rw-rw-r-- 1 gthaker adv.users 2.0M Oct 20 09:54 Barton704_Heathcote539_d8317e75-bb53-c439-38a8-d408ff1ea4aa.json
-rw-rw-r-- 1 gthaker adv.users  36M Oct 20 09:54 Caleb651_Daniel959_b9c228a1-58f5-7a38-fa52-566817fe284f.json
-rw-rw-r-- 1 gthaker adv.users 2.7M Oct 20 09:54 Colby655_Walker122_333540d4-3c55-21be-2b3f-7cdd92cb068b.json
-rw-rw-r-- 1 gthaker adv.users 4.1M Oct 20 09:54 Scotty190_Lakin515_6679344d-35ff-0e77-3de5-18648073a290.json
-rw-rw-r-- 1 gthaker adv.users  14M Oct 20 09:54 Sung603_Greenfelder433_095c0bf0-8690-7f49-81b3-f0825d6e0fbc.json
-rw-rw-r-- 1 gthaker adv.users 8.2M Oct 20 09:54 Xavier983_Gottlieb798_499ce35f-4244-982c-fa77-62d4b26a61bb.json
-rw-rw-r-- 1 gthaker adv.users 158K Oct 20 10:20 hospitalInformation1697810072093.json
-rw-rw-r-- 1 gthaker adv.users 165K Oct 20 10:20 practitionerInformation1697810072093.json

ChatGPT says:

"As of September 2021, the percentage of American males between the ages of 50 to 80 who are veterans was approximately 33.4%"

So I am not quite sure how this changes the equation much, but I suppose it does by perhaps a factor of 3?

[dehall]

We don't have an official script but you can run the following shell commands to upload them with cURL (replace SERVER with your FHIR server base url):

for file in hospitalInformation*.json
  do curl -d @$file -H 'Content-Type: application/json' SERVER
done

for file in practitioner*.json
  do curl -d @$file -H 'Content-Type: application/json' SERVER
done

for file in [A-Z]*.json
  do curl -d @$file -H 'Content-Type: application/json' SERVER
done

Re: the prevalence of Veterans, you're right it's a higher % than I remembered but still that 3x should be a meaningful performance improvement.

[gthaker]

Thanks for all your help. I was able to get my 6 patients uploaded to my local HAPI FHIR server and should now be able to move forward. If there is a bug tracker link related to my issues of having too many providers for my 6 patients pool please do share that.

I imagine Synthea does not generate image diagnostics (though probably does generate a radiologist report as if a radiologist had viewed an image.) With generative (image) AI it could be an interesting, may be, avenue to see if synthetic images too can be produced.

[dehall]

Oh I had assumed the issue would link automatically since I referenced this discussion # in it but I guess not -- see #1378

And you're right, out of the box Synthea does not create images though that is an area we explored once a few years ago (prior to the current suite of generative AI tools) and would consider looking into again. That was an effort called the "Coherent Data Set" where we brought together several different data types. If you're interested you can read more at https://doi.org/10.3390/electronics11081199 and the download link is at https://synthea.mitre.org/downloads (bottom of the page)