search-api endpoint enhancement: What values can be returned by the search-api? (SCHEMA)

[AlanSimmons]

Story

As a consumer of the search-api, I want to know all of the attributes that can be returned from a search against an ElasticSearch index. In other words, I want the equivalent of a data dictionary or annotated schema of the search response body.

Target

Developers of applications that would call the API.

Justification

Before being able to build search URLs with parameters, I need to know what can be searched.

Technical Translation

Publish a schema that describes the attributes of the response body.

The schema document should be the single source of truth, but published in multiple locations, including:

1. SmartAPI
2. GitHub Pages

Comments

1. The tab index_attributes_fields in the analysis spreadsheet in the Epic lists attributes for the entities, files, and antibodies indexes.
2. The tab index_attributes_fields maps attributes to YAML files distributed throughout repos in the organization. Currently, there are two types of YAML files:

• YAML files that describe the attributes--e.g., the meaning of organ
• YAML files that enumerate categorical values for the attributes--e.g., the possible values for organ

4. The YAML tab shows full file paths to the YAML files.
5. Some attributes do not map to YAML files. For example, some enumerations are actually hard-coded.
6. Because of the provenance hierarchy, attributes have scope. It will be necessary to map descriptions to all levels of scope for an attribute--e.g., "description", "ancestors.description", "descendants.description", "immediate_ancestor.description", etc.
7. It may be beneficial to filter out some attributes that may not be useful for search--e.g., uuid or entity_type.
8. The notebook list_index_attributes.ipynb in the analysis folder of the search-api repo splits and organizes index attributes based on specified path.

[AlanSimmons]

Analysis

Schema information related to responses from the search-api are available in other repositories.

The question is more one of source of truth: can the search-api documentation refer to schemas associated with other APIs (i.e., distributed sources of truth), or should the documentation in the other APIs refer to search-api?

The entity-api specification , which derives from this YAML file, describes schema elements that appear in responses from the search-api.

Issues with entity-api specification

1. The entity-api specification uses static sets of enums for some elements (e.g., assay_types) instead of YAMLs with enums (e.g. via $ref) to URL references.

2. Although the entity-api specification refers to a file component, the content appears to be out of synch with the corresponding components in both the search-api response and the files-api response.

Proposal

1. Update the entity-api specification for:

• enums
• the file component

2. Refer to the entity-api specification in the search-api specification.

[AlanSimmons]

The SmartAPI documentation for for search-api source YAML has been updated.

1. The search and search_by_index endpoints have clearer explanations of what is required to call the endpoint.
2. References to Elasticsearch DSL queries were updated.
3. A stub schema named SearchResponse contains a link to the entity-api SmartAPI documentation. Almost all of the elements in the response body of the endpoints will be entities from the provenance hierarchy. The SmartAPI documentation for entity-api describes the schema of entities in detail.