2025 Week 12
Hope everybody has been enjoying the nice weather this week so far, here’s this weeks FDRI developments!
Highlights
- Real FDRI Sensor data is being ingested -> API -> UI
- Phenocam gallery UI is almost out
- New Gridded Data section by Matt Brown
🌳 Work Package 1 FDRI Sensor UI
This week we stopped spending any more time looking into opensearch as an option to provide a quick user interface for the new FDRI sensor data after being unable to get it configured correctly. And instead switched gears into quickly ingesting the FDRI data, adding API endpoints and a UI to view the raw data in table form. It’s not quite live yet, but above is a sneak preview of our in development work.
✈️Timeseries Model
Work has begun on documenting our time series module, which is generic time series processing python module built on top of polars. There’s still a lot more to document and we don’t have the sphinx docs deployed anywhere, but they are available with the code above.
We’ve been having in-team discussions on how to aggregate flags in data . E.g if start with 1 minute resolution data and a single data point is flagged as suspect, if we aggregate that data into daily resolution, how do we mark the day against the flags in the underlying data. Still ongoing discovery on what should do here, more details available here
Driving the time series processing via metadata currently our time series processing pipeline has been driven via hardcoded config. We are working on actually using the metadata created by epimorphics , defined here to drive our processing pipeline.
Subperiod aggregation management, added restrictions to how aggregation can be applied to make sure, the agreggation period always fits cleanly into the original period. For example agreggating 1 hour data into 2 hour data fits, but aggregating 3 hour data into 4 hour doesn’t it leaves an overhang. The second example we have decided to explicitly not support (at least for now).
❇️Home page date selector
Small update on the COSMOS UI, we now have a date selector
📷 Phenocam
This week we were able to get the phenocam api live, available here . It works by search through the s3 bucket for the requested objects and generates a list of pre-signed urls for each image. The image urls can then be taken and presented in a user interface. Sneak peek of an in-development UI using our API above.
Planning of what to do next with phenocam is still ongoing. Our original plan is almost all green!
📖 Gridded Data
Written by Matt Brown
This week I thought I’d give a quick intro/reminder into what the gridded side of FDRI WP2 is all about before diving into the progress and details next week:
- Converting existing large gridded datasets into zarr AKA ‘ARCO’ and uploading them to a central object storage ‘repository’
- Having some-sort of catalogue/API/version control system (such as ArrayLake/IceChunk?) sitting over this to allow easy discovery and access, and make sure everyone works with the same central verified copy of the data. This will hopefully integrate with EIDC.
- Developing lots of user guides and examples for working with data on object storage instead of disk
Fuller details can be found in the product description document for brave adventurous souls, whilst day-to-day progress updates/me thinking via a keyboard are on a GitHub project board for the repo.
📚 Dev Book Club
Round 2 of the Dev Book Club is on Monday. If anybody is interested in joining, we are reading designing data intensive applications , next meeting will be about chapter 2: Data Models and Query Languages, can be joined at any time, no need to have been to previous sessions and no need to read the chapter, just a want to discuss the subject.