526ez | Pull disability list from Lighthouse API

[Mottie]

Description

Lighthouse provides a list of disabilities (/disabilities). We need to switch from loading in the static disabilities labels list (27KB) to that endpoint which has the most up-to-date list.

Open questions

• What feature is this associated with?

  Used by 526ez's add new condition autocomplete input (/disability/file-disability-claim-form-21-526ez/new-disabilities/add)

• What backend work needs to be done to complete this?
• Is this a paginated API?

Acceptance Criteria

• Hardcoded list is removed
• List from Lighthouse is being pulled
• List is searchable

Tasks

• Set up an action to load in the list of disabilities from Lighthouse
• Remove disabilityLabels.js file
• Add unit tests
• Add request spec tests
• Update e2e tests

Definition of done

• All tasks complete
• All tests passing

[jerekshoe]

@jacobworrell after talking to @Mottie I'm realizing that there needs to be backend work done before we can start this ticket. vets-website doesn't interface with Lighthouse directly, but uses vets-api as a middle man instead.

[saderagsdale]

@Mottie do we need to make a second story for the backend work that this depends on?

[Mottie]

Yes, we would need another ticket for backend work to set up communication with Lighthouse.

[tblackwe]

@RakshindaAslam This looks like a good candidate for Team Carbs (DBEX Squad 2) to take on.

[RakshindaAslam]

@tblackwe - This could be blocked by the ongoing LH migration work. Good item for 16th minute today.

[NB28VT]

Noting I removed the blocker on this since the referenced issue has been closed

[NB28VT]

Posting some initial findings from the website repo here. I haven't directly investigated the API endpoint as @micahaspyr was going to look into that; I also don't have a sandbox API key yet but will request one.

Findings Summary

There are actually a few places that reference the static disabilities list .

1. The autocomplete form input for adding a new disability, as referenced in the original ticket above.

    // Ideally, this would show the validation on the array itself (or the name
    // field in an array item), but that's not working.
    'ui:validations': [requireDisability],
    items: {
      condition: autosuggest.uiSchema(
        autoSuggestTitle,
        () =>
          Promise.resolve(
            Object.entries(disabilityLabels).map(([key, value]) => ({
              id: key,
              label: value,
            })),
          ),
        {

I validated this by hardcoding my own values in place of the mapping over disabilityLabels:

2. A mapping of the disability list values to lower case strings, called LOWERED_DISABILITY_DESCRIPTIONS.

This is referenced in the following validation code, which appears to happen on form submit. As documented in the code, we have a check that prevents a user submitting a disability name longer than 256 characters; if the name is present in our existing list of disabilities, we ignore that restriction

 // We're using a validator for length instead of adding a maxLength schema
  // property because the validator is only applied conditionally - when a user
  // chooses a disability from the list supplied to autosuggest, we don't care
  // about the length - we only care about the length of unique user-entered
  // disability names. We could've done this with `updateSchema` but this seems
  // lighter-touch.
  if (
    !LOWERED_DISABILITY_DESCRIPTIONS.includes(fieldData.toLowerCase()) &&
    fieldData.length > 255
  ) {
    err.addError('Condition names should be less than 256 characters');
  }

3. Another form pre-submittal callback which appears to update the form metadata submitted to the backend to include the code for the disability. I believe this happens anyway if the user selects one of the existing disability names, but presumably we still need this check in some cases

  // new disabilities that match a name on our mapped list need their
  // respective classification code added
  const addClassificationCodeToNewDisabilities = formData => {
    const { newDisabilities } = formData;
    if (!newDisabilities) {
      return formData;
    }

    const flippedDisabilityLabels = {};
    Object.entries(disabilityLabels).forEach(([code, description]) => {
      flippedDisabilityLabels[description?.toLowerCase()] = code;
    });

    const newDisabilitiesWithClassificationCodes = newDisabilities.map(
      disability => {
        const { condition } = disability;
        if (!condition) {
          return disability;
        }
        const loweredDisabilityName = condition?.toLowerCase();
        return flippedDisabilityLabels[loweredDisabilityName]
          ? _.set(
              'classificationCode',
              flippedDisabilityLabels[loweredDisabilityName],
              disability,
            )
          : disability;
      },
    );

    return _.set(
      'newDisabilities',
      newDisabilitiesWithClassificationCodes,
      formData,
    );
  };

4. In the "follow up" dropdown a user can use to select a disability that caused another disability

        causedByDisability: {
          'ui:title':
            'Please choose the disability that caused the new disability you’re claiming here.',
          'ui:required': (formData, index) =>
            formData.newDisabilities[index]?.cause === 'SECONDARY' &&
            getDisabilitiesList(formData, index).length > 0,
          'ui:options': {
            labels: disabilityLabels,
            updateSchema: (formData, primarySchema, primaryUISchema, index) => {
              const disabilitiesList = getDisabilitiesList(formData, index);
              if (!disabilitiesList.length) {
                return {
                  'ui:hidden': true,
                };
              }
              return {
                enum: disabilitiesList,
              };
            },
          },
        },

5. Two validation tests which references the static file directly. Note there are likely other tests/new tests we would write that cover this workflow as well

My Open Questions Remaining

• Is it within the scope of this ticket to update all references to the static file? The issue description seems to indicate so, although it doesn't appear the original author was aware of the other places the reference that file.
• What is the preferred mechanism for querying the Lighthouse API (via the vets-api) from within the website repo? The issue mentions setting up an "action". Does this refer to actions in the Redux sense?
• Is there a way we can load this list from the API once and reference that state in all of the places documented below which now reference the hardcoded file?
• How many entires are returned in total from the Lighthouse endpoint? It appears to be paginated, so we may need to make multiple calls based on the number of entries/pages/max entries per page (wondering if there is an existing wrapper for that type of data load in the front end somewhere)
• Many of the disabilities are capitalized in the hardcoded file are they capitalized in the response from the API?
• Still haven't investigated which existing tests cover this behavior/where we would put new tests, beyond the tests mentioned above which reference the disabilityList file directly.

[Mottie]

• Last time I looked, all references were pointing to the local disabilityList file. I believe the other places that use this list can be modified to use the form data instead of the list, e.g. we shouldn't need to validate an issue that was selected from the list
• I don't think there is an established method to query lighthouse. For the service branches, we query and store the short list in memory to limit API requests. There is also a fallback to a local service branches list when the API fails (invalid token?). But in this case, would it be better to actively interact with the API to get recommendations? Would this overload the endpoint?
• The disability list is very large. Loading it all in at once may not be ideal for mobile or slow Veteran with slow wifi.
• The lighthouse endpoint pagination is optional, you can get a full list
• Last time I checked, the list from Lighthouse is exactly the same as the disabilityList
• Since autosuggest is utilized, the tests for this feature should be in the platform code - I haven't looked in a long time

[micahaspyr]

Yep this list isn't queryable for matching text as the disabilities endpoint just returns everything or the scope of however many you want. I think this doesn't really solve much by hitting the lighthouse endpoint. I'm not sure how often the disabilities list changes, but we could just have an endpoint that returns a selected subset of data from the large json set. Seems the most practical way to me

[Mottie]

The list has changed once in the 3.5 years I've been working on VA products.

[Mottie]

Actually, I forgot that the separation location code loads in the list of locations and uses it in the autocomplete input

[austinjprice]

Backend: ready to move to in progress from initial spike - medium complexity due to unknown PR/Testing requirements, but simple from a code implementation perspective / plus front end and backend deployment

Frontend: need @NB28VT input

[NB28VT]

@austinjprice front end insight:

Ready to move forward on the front end, also medium timebox due to unknown PR/Testing requirements. The testing paradigm is the one part I didn't really get to explore.

Overall could either be complex or relatively straightforward I think it will depend on what kind of feedback I can get early.

That being said, from the demo check-in today it sounds like we may be blocked on this pending guidance from other teams.

[austinjprice]

Reviewed in Sprint review with feedback regarding dependencies around other teams working on this problem along with potential UX concerns. @austinjprice to follow up and connect the right parties

[austinjprice]

same as with this ticket: https://app.zenhub.com/workspaces/disability-benefits-experience-team-carbs-6470c8bfffee9809b2634a52/issues/gh/department-of-veterans-affairs/va.gov-team/59815

Moving to icebox for now until conversation around next steps

[austinjprice]

To remove from bug board and move to technical debt, pending discussion with team