2025 Week 35

Let’s gooooo

15 minute data is available!

https://dri-ui.staging.eds.ceh.ac.uk/fdri/sites/CBUW-UPZZ-FK7J

NRFA data is available!

https://dri-ui.staging.eds.ceh.ac.uk/nrfa/sites

Flux Data discovery (SFTP S3 Trasfer)

Initial discovery was with using presigned urls to upload objects to an s3 bucket via https, but this is restricted to 12 hours and rotating the urls wouldn’t be straight forward. We have since setup a temporary sftp connector for s3 to upload data via sftp. Hopefully we can get this working with the campbell loggers, sftp has come up a few times on the project so we see this as something that might be used for other data sources too.

Productionising the ingestion

Started work on improving our ingestion piece in the pipeline, it’s currently working but missing some core infra features, we are looking at adding:

• structured logging (+setting up loki to search application logs)
• performance testing the app against the expected full site install load (up to 500 loggers producing 50 variables per minute)
• backups for the data on s3
• Improve the grafana dashboard with metrics for ingestion

RDS user setup

• As part of the phenocam ingesting which uses RDS we are setting up IAM authentication for our postgres database users, which will allow us to use IAM roles/policies to control application access to the DB.

Some FDRI/NRFA metadata poking through

Metadata has started to appear in our metadata systems and should be ready to drive the FDRI/NRFA ui soon! Which means less id’s everywhere.

Ability to include private github repos in github action runners CI

Some internal changes to allows github projects at CEH to access other private repos during CI runs.

General UI bug fixes

• Quickly adding/removing plots on explorer
• Arrow keys cycling between sites when adding corrections
• query params change

Processing updates

• handling metadata pagination
• refactoring and improvements to timestream (check out the timestream code intro here)
• many more…..

Started daily wheel spinning

The new highlight of the teams day is watching our wheel of names spin and announce a winner (they get to chair the standup).