This repository has been archived by the owner on Dec 16, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6
iCarol Data Mapping (Eden and UWBA)
wojtekbu edited this page Feb 8, 2019
·
7 revisions
The data for two providers - Eden and UWBA are collected by the iCarol API. There were no differences found between those two data models, so there is only one code responsible for both mappings.
As the collected data might have impressive size, only basic information is returned in a "take all" request. It is required to collect the IDs of the entities from the response and execute plenty of new requests, with IDs as parameters to collect the detailed data. 10 is the max number allowed for a single request, so the IDs are grouped into 4 groups - programs, sites, agencies and programs at sites. Then the detailed data is collected for each of the group. Each of the groups is separately mapped, and each of them is described below.
Additionally, the data is persisted in an established order - Sites are mapped to the localizations at first. Then, for each Agency from "Site.related" array is mapped to an Organization. "Agency.related" is used to find related Programs, which are mapped to the Services. Described flow ensures that the relations between entities are persisted as well, but it also might lead to the lack of the data if more than one Site was attached to the Agency - only first one is considered then. It has to be changed later, as it should be possible for One Organization to have more than one Location.
| Provider field | Example value | Mapped to |
|---|---|---|
| contactDetails | An array of objects of different types. Each type is mapped differently (see below). | details below |
| contactDetails.PhysicalLocation | "contact": { "latitude": 11.342769, "longitude": -131.312386, "precision": "Precise", "purpose": "Physical", "line1": "1122 Avenue", "county": "Alameda", "city": "New York", "stateProvince": "CA", "country": "United States", "zipPostalCode": "23541", "type": "PhysicalLocation" }, "isConfidential": false |
PhysicalAddress: line1 - address1 city - city county - region country - country stateProvince - stateProvince zipPostalCode - postalCode and with reference to the Location, where: name is generated with pattern: line 1 + " - " + city + " (" + state + ")" and: longitude - longitude, latitude - latitude |
| contactDetails.PostalAddress | "contact": { "precision": "Precise", "purpose": "Physical", "line1": "1122 Avenue", "county": "Alameda", "city": "New York", "stateProvince": "CA", "country": "United States", "zipPostalCode": "23541", "type": "PostalAddress" }, "isConfidential": false |
PostalAddress: line1 - address1 city - city county - region country - country stateProvince - stateProvince zipPostalCode - postalCode |
| accessibility | { "disabled": "Wheelchair accessible", "public": "Not accessible" } |
AccessibilityForDisabilities: disabled - accesibility, public - details |
| id | 123123 | used to detect relations, but ignored in mapping |
| databaseID | 2376 | ignored |
| uniquePriorId | 3283712 | ignored |
| related | array of objects like: { "id": 111111, "type": "Site", "name": "Site name" } |
used to detect relations, but ignored in mapping |
| cultureCode | "" | ignored |
| searchHints | array of values like "1111111" | ignored |
| type | Agency | used to detect type of the entity, but ignored in mapping |
| modified | 2018-12-19T19:35:47.323Z | ignored |
| lastVerifiedOn | 2017-10-09T07:00:00Z | ignored |
| isFeatured | false | ignored |
| isConfidential | false | ignored |
| status | "Active" | ignored |
| languagesOffered | "English; Spanish" | ignored |
| legalStatus | None or unknown | ignored |
| translations | array of values like "en" | ignored |
| Provider field | Example value | Mapped to |
|---|---|---|
| names | An array of objects like: { "value": "The Name", "purpose": "Primary" } |
the "Primary" element of the array is mapped to the Organization.name, and if any other ones exists, the first of them is mapped to the Organization.alternateName |
| contactDetails | An array of objects of different types. Each type is mapped differently (see below). | details below |
| contactDetails.EmailAddress | "contact": { "address": "[email protected]", "label": "Main", "type": "EmailAddress" }, "isConfidential": false |
Organization.email |
| contactDetails.Website | "contact": { "url": "www.example.org", "label": "Website", "type": "Website" }, "isConfidential": false |
Organization.url |
| status | "Active" | if equals "Active" then Organization.active = true, else Organization.active = false |
| id | 123123 | used to detect relations, but ignored in mapping |
| databaseID | 2376 | ignored |
| uniquePriorId | 3283712 | ignored |
| related | array of objects like: { "id": 111111, "type": "Site", "name": "Site name" } |
used to detect relations, but ignored in mapping |
| cultureCode | "" | ignored |
| searchHints | array of values like "1111111" | ignored |
| type | Agency | used to detect type of the entity, but ignored in mapping |
| modified | 2018-12-19T19:35:47.323Z | ignored |
| lastVerifiedOn | 2017-10-09T07:00:00Z | ignored |
| isFeatured | false | ignored |
| isConfidential | false | ignored |
| languagesOffered | "English; Spanish" | ignored |
| description | "{open some xml tag}This agency is great{close some xml tag}" | ignored |
| descriptionText | "This agency is great" | ignored |
| coverageNote | "CA - Alameda County" | ignored |
| verificationExpired | 2017-10-09T07:00:00Z | ignored |
| coverage | array of objects like: { "purpose": "CoverageArea", "county": "Alameda", "stateProvince": "CA", "country": "United States", "type": "PostalAddress" } |
ignored |
| legalStatus | None or unknown | ignored |
| translations | array of values like "en" | ignored |
| customFields | array of objects like: { "id": 1010, "label": "Transport Eligibility", "selectedValues": { "2303": "No" } |
ignored |
| hours | array of objects like: { "dayOfWeek": "Mon", "opens": "6pm", "closes": "8am" } |
ignored |
| Provider field | Example value | Mapped to |
|---|---|---|
| contactDetails.PhoneNumber | "contact": { "purpose": "Unspecified", "number": "123-321-333", "label": "Phone", "description": "Main Office", "type": "PhoneNumber" }, "isConfidential": false |
number - Phone.number label - Phone.type description - Phone.description |
| languagesOffered | "English; Spanish" | separated by "; " and each mapped to Language.language |
| names | An array of objects like: { "value": "The Name", "purpose": "Primary" } |
the "Primary" element of the array is mapped to the Organization.name, and if any other ones exists, the first of them is mapped to the Organization.alternateName |
| contactDetails | An array of objects of different types. Each type is mapped differently (see below). | details below |
| contactDetails.EmailAddress | "contact": { "address": "[email protected]", "label": "Main", "type": "EmailAddress" }, "isConfidential": false |
Service.email |
| contactDetails.Website | "contact": { "url": "www.example.org", "label": "Website", "type": "Website" }, "isConfidential": false |
Service.url |
| descriptionText | "This service is great" | Service.description |
| fees | "No fees" | Service.fees |
| applicationProcess | "By phone" | Service.applicationProcess |
| eligibility | "Anyone in need" | Eligibility.eligibility |
| id | 123123 | used to detect relations, but ignored in mapping |
| databaseID | 2376 | ignored |
| uniquePriorId | 3283712 | ignored |
| related | array of objects like: { "id": 111111, "type": "Site", "name": "Site name" } |
used to detect relations, but ignored in mapping |
| cultureCode | "" | ignored |
| searchHints | array of values like "1111111" | ignored |
| type | Agency | used to detect type of the entity, but ignored in mapping |
| modified | 2018-12-19T19:35:47.323Z | ignored |
| lastVerifiedOn | 2017-10-09T07:00:00Z | ignored |
| isFeatured | false | ignored |
| isConfidential | false | ignored |
| status | "Active" | ignored |
| description | "{open some xml tag}This service is great{close some xml tag}" | ignored |
| coverageNote | "CA - Alameda County" | ignored |
| verificationExpired | 2017-10-09T07:00:00Z | ignored |
| requiredDocumentation | "None" | Added to Service.applicationProcess, with prefix " Required items: " |
| taxonomy | array of objects like "AB-3233 * CD-3333.3330" | ignored |
| hours | array of objects like: { "dayOfWeek": "Mon", "opens": "6pm", "closes": "8am" } |
each mapped to OpeningHours: dayOfWeek - proper number of weekday (assuming that possible values are Mon, Tue, Wed, Thu, Fri, Sat, Sun) opens - opensAt closes - closesAt |
| coverage | array of objects like: "purpose": "CoverageArea", "county": "Alameda", "stateProvince": "CA", "country": "United States", "type": "PostalAddress" } |
ignored |
| Provider field | Example value | Mapped to |
|---|---|---|
| id | 123123 | used to detect relations, but ignored in mapping |
| databaseID | 2376 | ignored |
| uniquePriorId | 3283712 | ignored |
| related | array of objects like: { "id": 111111, "type": "Site", "name": "Site name" } |
used to detect relations, but ignored in mapping |
| cultureCode | "" | ignored |
| searchHints | array of values like "1111111" | ignored |
| type | Agency | used to detect type of the entity, but ignored in mapping |
| modified | 2018-12-19T19:35:47.323Z | ignored |
| lastVerifiedOn | 2017-10-09T07:00:00Z | ignored |
| isFeatured | false | ignored |
| isConfidential | false | ignored |
| contactDetails | [] | ignored |
| status | "Active" | ignored |
| languagesOffered | "English; Spanish" | ignored |
| coverageNote | "CA - Alameda County" | ignored |
| verificationExpired | 2017-10-09T07:00:00Z | ignored |
| isLinkOnly | false | ignored |
A parsed data should respect all restrictions described below. If a part of the data does not apply to these rules, it should not be persisted.
| Field | Restrictions | Mapped to |
|---|---|---|
| agency.name | not blank | organization.name |
| agency.contactDetails.contact.number | not blank | phone.number |
| agency.hours.day.dayOfWeek | not blank | openingHours.weekday |
| agency.status | not blank | organization.active |
| program.contactDetails.contact.line1 | not blank | postalAddress.address1 |
| program.contactDetails.contact.city | not blank | postalAddress.city |
| program.contactDetails.contact.stateProvince | not blank | postalAddress.stateProvince |
| program.contactDetails.contact.number | not blank | phone.number |
| program.hours.day.dayOfWeek | not blank | openingHours.weekday |
| program.eligibility | not blank | eligibility.eligibility |
| site.contactDetails.contact.line1 | not blank | postalAddress.address1 |
| site.contactDetails.contact.city | not blank | postalAddress.city |
| site.contactDetails.contact.stateProvince | not blank | postalAddress.stateProvince |
| site.accessibility.disabled | not blank | accessibilityForDisabilities.accessibility |
To ensure all required relationships between entities are set, each of them is listed below.
- Each Language has a reference to the Service and Location
- AccessibilityForDisabilities has a reference to the Location
- Eligibility has a reference to the Service
- Phone has a reference to the Service and Location
- PhysicalAddress has a reference to the Location
- PostalAddress has a reference to the Location
- Service has a reference to the Organization
- Organization has a reference to the Location and the Source Document
- Each OpeningHours has a reference to the RegularSchedule
- RegularSchedule has a reference to the Serivce
If the record is marked as confidential, it will not be persisted in the database. However for some fields, the record is never marked as a confidential, but its parent is instead. Below it's described when a record is considered as a confidential, based on already parsed ServiceNet objects:
- AccessibilityForDisabilities - won't be saved if ICarolAccessibility is confidential
- PhysicalAddress - won't be saved if ICarolContactDetails is confidential
- PostalAddress - won't be saved if ICarolContactDetails is confidential
- Location - won't be saved if ICarolContactDetails is confidential
- Phone - won't be saved if ICarolContactDetails is confidential
- Organization - won't be saved if ICarolAgency is confidential
- Service - won't be saved if ICarolProgram is confidential
- Eligibility - won't be saved if ICarolProgram is confidential
- OpeningHours - won't be saved if ICarolDay (element of ICarolHours.days) is confidential or if ICarolHours is confidential
- Language - won't be saved if ICarolProgram is confidential
Additionally, if a record is not marked as confidential, but its fields are:
- Organization.name - will be null if the mapped element of ICarolAgency.names is confidential
- Organization.alternateName - will be null if the mapped element of ICarolAgency.names is confidential
- Organization.email - will be null if the mapped element of ICarolAgency.contactDetails is confidential
- Organization.url - will be null if the mapped element of ICarolAgency.contactDetails is confidential
- Service.name - will be null if the mapped element of ICarolProgram.names is confidential
- Service.alternateName - will be null if the mapped element of ICarolProgram.names is confidential
- Service.email - will be null if the mapped element of ICarolProgram.contactDetails is confidential
- Service.url - will be null if the mapped element of ICarolProgram.contactDetails is confidential