Mirabuf Caching

[a-crowell]

Description

Utilizing browser storage for mirabuf caching: local storage for a map of keys and metadata, and OPFS for storing mirabuf data

Objectives

• Cache from fetchLocation or ArrayBuffer (if not already in)
• Get assembly from ID
• Remove specific entry from local storage and OPFS
• Clear storage
• SpawningModal separates mirabuf files in cache vs remote as well as uses the names of the files.

Testing

Two new temp buttons on the MainHUD:

• Print Mira Maps -> prints the maps of robot/field fetch locations and ids in console
• Clear Mira -> clears both these maps stored in local storage and any mirabuf files stored in OPFS

Other than making sure robots and fields import correctly (no user difference from before), you can also check:

• Pulling from cache when it can and remote when it needs to (via console "Mira in cache" vs "Caching new mira")
• Saving correctly (via "Print Mira Maps" for the local storage maps and an OPFS explorer for the mirabuf files)

Warning

Clear browser cache completely before testing. Storage structure is different now, so past data is breaking.

JIRA Issue

[autodesk-chorus]

Chorus detected one or more security issues with this pull request. See the Checks tab for more details.

As a reminder, please follow the secure code review process as part of the Secure Coding Non-Negotiable requirement.

[BrandonPacewic]

Couple of fixes after the merge dev commit that I had to make to run the branch.

[HunterBarclay]

@BrandonPacewic Could you commit those fixes?

[BrandonPacewic]

@BrandonPacewic Could you commit those fixes?

@HunterBarclay, committed.

[PepperLola]

This looks good and seems like it works well. The only nitpicks I have are to add some documentation to the UnzipMira function (what do the magic numbers mean?) and then to use undefined instead of null per the official TypeScript style guidelines.

[a-crowell]

@PepperLola Done on the undefined. I didn't write the UnzipMira though. @HunterBarclay do you know how it works?

[PepperLola]

@PepperLola The caching usually compares the url/fetchLocation to determine whether or not the file is in the cache already. The ImportLocalMiraModal uses createObjectURL to pass in a fetchLocation though, giving keys of "blob: ... localhost ...", so instead, we now use the hash as a key.

If you have to load, unzip, and decode the mira to hash it before checking if it's already in the cache, wouldn't that defeat the point of caching it in the first place? Or is there something I'm missing?

[a-crowell]

If you have to load, unzip, and decode the mira to hash it before checking if it's already in the cache, wouldn't that defeat the point of caching it in the first place? Or is there something I'm missing?

You're right what is the point of this.

In case Hunter meant hash of the url, I tried that too, but the whole thing is that it's different every time so it doesn't help either.

The two options I see are to either use the file name or just accept a new file from that modal and not cache it. @HunterBarclay You were against cutting down the fetchLocation to the name for other files--do your reasons still apply here?

Is there another way to check for same files I'm not thinking of?

[BrandonPacewic]

Thanks for making all my suggestions. It sounds like there is still some internal discussion that needs to be had on this feature so I'll wait to properly review it till after everything has been finalized.

[PepperLola]

Instead of caching the file and then getting it and decoding there in the CacheAndGetLocal/Remote functions, why not unzip and decode there or create a new function to do it that is called everywhere else instead? If the end result of caching and then getting the cache is just getting the mira assembly, and if they're both using the same file buffer, this would skip however much time it takes to (unnecessarily?) read the file back from OPFS. Obviously let me know if I'm missing something, but to me it looks like a redundant step that could be skipped to save time.

[a-crowell]

@PepperLola Thanks for pointing that out! I removed the remote version since it was never used anyway and adjusted CacheAndGetLocal to avoid grabbing from OPFS while taking advantage of having the assembly already to cache the name.

[PepperLola]

Looks good

[BrandonPacewic]

I'm not 100% up to date on this code and I just want to make sure that @HunterBarclay 's comment was address / is no longer relevant.

I used a URL.createObjectURL() for the ImportLocalMirabufModal to convert the blob I get from the file input to a downloadable URL. We should probably customize the key for those, maybe a hash of the blob?

[a-crowell]

@BrandonPacewic Yep, all covered! That modal now uses a hash of the file as key instead of url, so you won't find another localhost in the map.

[BrandonPacewic]

I tried to push formatting fixes but this branch is still not passing validation for some reason. I mentioned this in another PR review.

From what I can tell looks good, will be easier to test everything when we have end to end functionality.

[HunterBarclay]

Fixed the unit tests and added a few clean ups. Didn't format because it still doesn't work on my desktop.

[HunterBarclay]

Merging for now. This Prettier issue is going to be a continuous problem