2026 Week 23

Lots of developments this week as things start to ramp up for the September soft launch - and we have welcomed back Dom after his paternity leave.

Metadata Service

There is alot of work on the metadata side of things atm. Notable developments are:

• EA and SEPA station metadata has started to be collated and the metadata model setup to ingest these. This will allow us to deploy our new EA ingesters (we now have 4!) into the new accounts, alongside the existing SEPA one, and start collecting some data.
• Some fundamental changes to how our data processing configurations are handled. As a reminder, any dataset (e.g. temperature, rainfall, potential evapotranspiration etc.) can have any number of “data processing configurations” that define “what to do” to generate that dataset. This can be QC methods, infilling, corrections, derivation calculations, or aggregations. We now have explicit “data processing plans” saved to a dataset, so our pipeline has an explicit ordered list of operations it needs to carry out. This has also required some work on the pipeline itself to handle these changes. But the benefits are now we can be more flexible with ordering of configurations, and no longer need “interim” datasets defined for intermediate steps of processing.

UI

Comparison plots are now available in the UI. There is a new button next to the “add/edit variables” which allows switching the plot type from the default to a comparison chart and back again.

Flux

The refactor from Richard mentioned last week has been merged, and I managed to do a run of the flux data all the way through the new metadata-driven timeseries-processor! In theory, all that now remains is to get it deployed onto AWS (although there is a tiny bit more post-processing to implement). Of course things are never quite that simple, and we will have to update the metadata API with information about the FDRI flux sites instead of the COSMOS ones we’ve been using so far.

The work we handed over to IT Infrastructure on a scheduled pipeline to collect 20Hz flux data directly from Campbell devices over a VPN has now been deployed - kudos to Andrew Sun for setting it up without needing guidance and support from the dev team! It’ll need some observation and small changes, but the hard parts are done.

AWS Migration

Not loads left to do on the migration so been a slightly quieter week, but lots going on in the background. We are getting ready to migrate the Phonocam ingestion service (once everything is setup in the database), we have been getting the infrastructure setup to allow Flux data to be ingested into the new accounts and we have been discussing the best way forward to switching the sensors over to the new accounts (and how we test this).

Data API

In preperation for the soft launch in September we have been adapting our API to restrict certain endpoints (Phenocam and NRFA) and networks (NRFA and others) for users outside of UKCEH. As an intermin solution whilst longer-term company wide solutions are reviewed, we are restricting access based on the IP address of the user.

Geospatial API and UI

We have been busy getting PostGIS setup onto our RDS instance, and at the same time, allowing the geospatial API to connect and find the data. Fingers crossed we can test out all the components next week.

Monitoring and logging

Work has started on building a logging and monitoring stack for the project. As part of this, we have been restructuring our Kubernetes cluster to be more resiliant when deploying services. Alot more work is required, especially on standardising our logs accross services, but we now have a basic UI where we can see them.

A brief update from the gridded data team

The gridded data engagement online workshop report has been produced and circulated internally for approval, with an aim to publish early June. Work is ongoing to make more datasets available in Zarr format including:

• CEH-GEAR-1hr: An observations-derived hourly 1km resolution gridded rainfall dataset over the UK from 1990 to 2016 in two separate chunking regimes to allow for more performant access
• CHESS-SCAPE: Projected daily 1km resolution gridded meteorological driving data aimed at hydrological models for the UK under different climate change scenarios for 1980-2080.
• HadUK-Grid rainfall