2026 Week 3
First week note of 2026!
🤖 New IOT Domain for FDRI
We have new domain for MQTT FDRI devices mqtt-fdri.eds.ceh.ac.uk it’s confirmed fully working and is being used by 3 loggers so far (CH-CHOPS-01-RG, CH_MILFA_EC_01, Generic-Precipitation-1). All loggers need to be migrated to this domain to continue to work, using a domain CEH controls instead of *.amazonaws.com allows us the ability to migrate to the new AWS accounts (see below). Or even control to switch the DNS to send the data whereever we want, allowing a form of cloud agnosticism.
⛰️ Cosmos MQTT coming soon
The COSMOS team are in the process of testing out new MQTT comms the same as fdri. Since it’s the same campbell loggers as FDRI, it shouldn’t be too much work on our side, but we will need to make some changes to support, most likely:
- setup a new iot domain for cosmos (mqtt-cosmos.eds.ceh.ac.uk)
- setup the iot rule to write to the level 0 bucket under /cosmos/cosmos-sensors (instead of /fdri/fdri-sensors)
- setup api and ui to support new data
☁️ AWS ADR
We’ve writen up a new Architecture Decision Record (ADR) on how we plan to setup our AWS account structure going forward. This is for FDRI and future DRI projects that fit. https://github.com/NERC-CEH/fdri_words/blob/main/adrs/013-AWS-Accounts.md
(Although written up, discussed and agreed with all parties amny times, a different approach has been forced through, update in next weeknote)
🎹 Flux data
This week we’ve kicked off the flux/eddy covariance work. The aim is to take 20 Hz data measuring water vapour fluxes, telemeter them back into an S3 bucket, then run processing software (EddyPro) on it to produce 30-minute output. We’re hoping to reuse as much as possible of the ingestion and processing code. There are a few additional challenges with some of the metadata and data that the processing software needs to run, as well as dealing with the sheer volume of 20 Hz data from lots of sites (although only 1 is running so far).
This is a diagram of how the architecture plan looks at the moment:
So far, we’ve got the team (Hamayoon, Josh, Matt C, Matt DP, Ola, Robin with Hollie as Product Owner) together and up to speed on what we are doing, and set up some initial investigatory tickets. Some important early jobs are:
- speak to other groups doing eddy covariance analysis with the hope that we can build something that can fit into their workflows too
- decide how we want to arrange our buckets: whether we share the existing time series ones or set up our own
- work out the best way of integrating EddyPro (which is usually used with a GUI) into a headless pipeline
🐶 Data processing pipeline
Back in 2025’s weeknote 48, we mentioned about the reworking of how our time series processor resolves dependencies using a Directed Acyclic Graph (DAG). This has now been deployed onto AWS, with the first bits of processed data coming through! A bit more work to resolve some changes to the metadata API, but a big step in getting a robust and extensible pipeline implemented.
Next steps will be to start showing the processed data on the web UI, with thought need about how we show the various flagging for each data point (e.g. to tell you about what QC, infill, etc. has occured during processing).