2026 Week 6

As always lots happening, including a new team launch!

The first week note went out Week 6 2025, it’s been a whole year of weeknotes!

🔐 Username/password MQTT IOT Core Comms

New domain that supports username/password MQTT has been setup, this is to support so low powered IOT devices, that don’t support certificate based MQTT communication. AWS IOT Core only supports certificate based MQTT by default, so we setup a custom authorizer that enables this new flow.

🌳 COSMOS Loggers coming on board

We are starting to support the COSMOS team in sending data to us via MQTT, to take advantage of the data pipeline we have produced. We are currently setting up the first test logger, but getting another MQTT sensor network into our systems will be a nice milestone!

⛈️ AWS Migration

Lots happening with our AWS migration team! For those inside UKCEH the project board can be seen here. We are 11 tickets in so far.

Highlights:

• Reviewing our existing infra, we’ve managed to reduce costs on the old account by around 26%
• We have noticed a number of things that are misconfigured (sometimes in multiple places as config had been copied)
• New accounts are now configured to use terraform
• We are in progress spinning up newly named s3 buckets and our EKS cluster

🛍️ Processed Data is being produced

The end goal of the timeseries processor product is to generate “processed” datasets of the variables recorded at each monitoring site - taking the raw data and applying QC, gap filling, derivations and aggregations. We are finally at a place where we have data flowing through our pipeline on a scheduled task, and being saved out to “processed” data files! Next steps are to show these datasets on the UI. These datasets will be separate from the “raw” version, so you’ll be able to plot both alongside each other to compare raw vs. processed.

🐶 DAG update

We’ve made some tweaks to our directed acylclic graph (DAG) builder to be more efficient with the calls to the metadata API. This was driven by a unification of where each dataset’s processing methods stores it’s “dependencies” (i.e. which other datasets are needed for a given calculation). They all come from the same place now, which simplifies the dependency gathering phase of the DAG.

🪛 NRFA

A working MVP was completed last week which allows the NRFA team to upload data into batches, view the data in a UI for manual QC and then delete batches as required. We will be working with them over the next few weeks to test the tools alongside the upcoming data ingestion.

💼Flux pipeline

Work continues on the eddy covariance pipeline. Josh presented the initial architecture diagram at the All Things FDRI product showcase on Thursday. Hamayoon has finished putting together a Docker image that is uploaded the the ECR repo, which will be necessary to run EddyPro in the AWS Kubernetes cluster. Josh is still working through deciding the archictecture to extend the time series processor, what metadata is required to run the pipeline (both sensor information and processing configuration). Current priorities are identifying the best way for the 20Hz data to be telemetered back into the system, deciding on the how to integrate into the timeseries-processor code, establishing the compute requirements to run EddyPro, and figuring out a data retention policy.

⏲️ Updates to TimeStream

Implemented new AngularMean aggregation method, which calculates the average angle over a given aggregation period. Used for calculating e.g. average daily wind direction. See [TimeStream Documentation][https://nerc-ceh.github.io/time-stream/user_guide/aggregation.html#angular-mean]

📽️ Updates to TimeSeries-Processor

• Added new aggregation methods to calcualte the average wind direction and average radiation.
• Developing new correction method Clip to set data that is above or below a given threshold to that threshold value. This will be used to correct PE values below 0 to 0. This required updates to the metadata.

🤘Updates to Metadata

• We are making changes to metadata to accommodate new datasets and parameters required for the updates to the time series processor, e.g.aAdded new datasets and methods to process (correct) PE.

🐜 Bug with metadata and cosmos/nrfa not available in the UI

During the week, some changes to the metadata API were implemted in order to better support the timeseries processor. This involved refactoring the datasets metadata api endpoint to remove timeseries definition information. Unfortunately an unexpected side effect of the refactoring was that some of the fields which the data API and UI rely on were accidentally removed, disabling the ability to plot COSMOS and NRFA variables in the UI. FDRI was unaffected for the most part as it has backup logic to scan S3 directories to list available variables for logger data that has not yet been entered into the metadata api.

The bug has now been addressed and it is possible to plot COSMOS and NRFA data in the UI again. Although it should be noted that there are still some outstanding issues where some variables appear to be duplicated and work is underway to resolve this.

In order to avoid this happening again we are looking into removing mocking of the metadata api responses so we can test any changes to the metadata api directly against dependent codebases (data api, timeseries processor, ingestion etc). This should allow any issues to be identified and fixed before the metadata api changes are merged in.

🪨Geospatial API and UI

A lot of work has been happening in the background for the geospatial UI. Functionality in development at the moment includes the ability to zoom on the map to a layer once it has been loaded, and to view the layer in 3D.

Work is also underway to identify and begin processing all initial FDRI and supporting data that will be needed to be displayed in the UI for the September release.