2025 Week 31

weeknote

EA and SEPA data is ingested!

We’ve now got EA and SEPA api data being regularly ingested into our systems and store in s3. This is a huge milestone in getting ready to process the NRFA data in our new system.

Processing Updates

A few more of us in the team have been getting involved and focusing on the time series processor.

mergin the ts_def and ts_id
implementing derivations
implementing aggregations
passing the network via the cli

Data Publishing

Work is starting on getting the FDRI data into our publishing systems. Mostly like source of truth data format will be parquet, with options to request CSV.

Expected data available (per catchment)

FDRI Chess catchment meteorological station data
FDRI Chess catchment rainfall station data
FDRI Chess catchment evaporation station data
FDRI Chess catchment soil moisture station data
FDRI Chess catchment groundwater station data
FDRI Chess catchment river level/flow station data
FDRI Chess catchment water quality station data

30 minute data now available in the UI

Thirty minute data can now be viewed and plotted in our UI.

Phenocam database indexing, sqs etc

Ongoing work is happening to start indexing the phenocam images as they come in, this is essential to allow performant queries to fetch the phenocam images.

Rough sketch of how this fits together:

Cluster version update

EKS cluster version has been updated from 1.30 -> 1.33

WP1 are using our UI

Attending the data checking last week we saw, WP1 using our UI instead of excel, this is really great to see and very happy to have our tools used in the early stage.

Here’s some links WP1 are using:

FDRI Product showcase

Last week we had our FDRI WP2 product showcase, this was of a really high standard, giving an overview of what each product has done so far.

Fun UI bug

Recording 2025-08-04 102228(1)

Dithering in above gif is due to compression, the bug is when multiple rows are removed from the select table, this is a race-condition bug related to the query params being updated. Still on going on how best to fix.