2026 Week 8

Welcome Clare Jones

We have a new member! Clare is here to help out with project management while Mollie goes on maternity leave. Welcome!

🥳 COSMOS Processed data is live

This is a huge milestone! We’ve got an end to end pipeline working! Data is ingested, processed, stored, available via an API and plottable in our ui!!

Checkmarks across the whole pipeline, next up FDRI data! (Plus much more work to add on)

Below is a screenshot of the raw vs processed wind direction data at the Alice Holt site. See it here

⛈️ AWS migration update

Hit a few blockers in the past few weeks but we’ve managed to move past them, the above completed issues tracker isn’t a great measure of progress in itself but clear shows us getting blocked and unblocked.

Milestones for this project have been written up here.

❇️ Everything broke and was unrelated to the AWS migration

We hit a really odd issue last week were all our deployment pipelines broke and we couldn’t push to ECR from github actions anymore, we assumed this must have been related to our ongoing AWS migration. It was fun digging into Service Control policies, NACL, NAT gateways, subnet configs etc but turns out something got moved under our feet in the github action, and it now trys to get the images before pushing a new one. So needed a one line change to update the IAM policies to allow read and not just write.

🗣️ Error handling in pipeline discussion

There is a question as to whether the timeseries processing pipeline should stop if a dataset dependency fails to be processed, or if some data should be saved. Please take a look at the notes from https://github.com/NERC-CEH/fdri_words/pull/35.

👾 NMDB ingester update

NMDB metadata is now available from the epimorphics API. Hardcoded sites have been removed from the NMDB config, and sites are being read from the API instead. Ongoing discussion between Leanne and Nathan about some architecture/testing decisions.

🎛️FDRI QC Metadata update

QC tests for battery voltage have been added to the metadata for some FDRI datasets to test QC with FDRI as a proof of concept, before adding more QC tests.

🔌Flux pipeline

Since the last update, we’ve had a lot of meetings with various people to figure out where we’re going to fit into other places, like telemetry coming from the field, the metadata service, and adding our code to the timeseries processor. This has meant that a lot of things are becoming much clearer. We’re aiming to get an initial “spike” working in the next week or so, where we can run data through EddyPro in AWS with some hardcoded metadata and some manual putting of the data in the right place. Hamayoon has also opened our first PR to the timeseries-processor repo.