ECMWF Real-Time Data Availability

[am94ghiassi]

Hello everyone,

I am currently working with ECMWF real-time data and have encountered an issue regarding the timeliness of the forecast data. Despite the "real-time" designation, the most recent forecast I can access, regardless of the model stream, has an issue time of 10 days ago from the current date.

Any insights or suggestions on how to access more up-to-date forecast data would be greatly appreciated.

Thank you in advance for your help!

Best regards,
AmirMasoud

[TomAugspurger]

Can you show a sample STAC search returning stale data? With this Azure CLI command, I can see some files from today:

az storage blob list --account-name ai4edataeuwest --container-name ecmwf --sas-token (curl "https://planetarycomputer.microsoft.com/api/sas/v1/token/ai4edataeuwest/ecmwf" | jq -r .token) --prefix 20240603 -o table
Name                                                              Blob Type    Blob Tier    Length      Content Type              Last Modified              Snapshot
----------------------------------------------------------------  -----------  -----------  ----------  ------------------------  -------------------------  ----------
20240603/00z/aifs/0p25/oper/20240603000000-0h-oper-fc.grib2       BlockBlob    Hot          68293075    application/octet-stream  2024-06-03T07:59:17+00:00
20240603/00z/aifs/0p25/oper/20240603000000-0h-oper-fc.index       BlockBlob    Hot          19580       application/octet-stream  2024-06-03T07:59:14+00:00
20240603/00z/aifs/0p25/oper/20240603000000-102h-oper-fc.grib2     BlockBlob    Hot          63164041    application/octet-stream  2024-06-03T07:59:35+00:00
20240603/00z/aifs/0p25/oper/20240603000000-102h-oper-fc.index     BlockBlob    Hot          19370       application/octet-stream  2024-06-03T07:59:38+00:00
20240603/00z/aifs/0p25/oper/20240603000000-108h-oper-fc.grib2     BlockBlob    Hot          62570444    application/octet-stream  2024-06-03T07:59:40+00:00
...

There is an issue with some of the STAC metadata generation failing because the ECMWF recently started publishing AIFS forecasts. The stactools package we're using doesn't yet handle that properly, and we end up trying to ingest items with duplicate IDs.

[am94ghiassi]

By using the Azure CLI command, I retrieved the same files. However, when I use the following sample code (which is a sample of STAC I assume):

import pystac_client
import planetary_computer

catalog = pystac_client.Client.open(
    "https://planetarycomputer.microsoft.com/api/stac/v1",
    modifier=planetary_computer.sign_inplace,
)
search = catalog.search(
    collections=["ecmwf-forecast"],
    query={
        "ecmwf:stream": {"eq": "oper"},
        "ecmwf:type": {"eq": "fc"},
        "ecmwf:step": {"eq": "12h"},
    },
)
items = search.get_all_items()
item = max(items, key=lambda item: item.datetime)
item

The result shows the latest time as ecmwf-2024-06-02T18-oper-fc-12h-0.25 . However, yesterday the latest one was ecmwf-2024-05-17T18-oper-fc-12h-0.25.

Do I need to change the catalog? If so, what should the collection and query be?

Many Thanks!

[TomAugspurger]

That's most likely the issue from the new AIFS forecasts I mentioned. We'll get that fixed eventually, and there shouldn't be any changes required from your side.

If necessary, you can use Blob Storage APIs as a workaround for now.

[am94ghiassi]

I checked the blob through the Azure CLI command, and I noticed that for a single forecast only the lead_time of 0-hr, 6-hr, 12-hr, and so on for oper stream are available there. However the lead_time of 1-hr, 2-hr would be interesting for us. I was wondering if there is possibility for accessing them.

Thanks in advance for your help again!

[TomAugspurger]

I'm not sure, that would depend on what all data the ECMWF generates and publishes.