2026 Week 25
Welcome to last weeks week note, plenty happening with new deployments going out with new sensor types (ecolog 1000). Flux data transfer connection is starting to look promising, new campbell dataloggers are sending data directly to our new production AWS account and some sleek designs and progress on our networking monitoring dashboard.
Ecolog 1000
We have data! We have received the first bits of data from the first ecolog 1000 device in production.
This puts us at the green Raw s3 bucket box in this diagram. Next up is ingesting the data into the ingested bucket, so it can be easily queried.
Flux
After what looked like a promising start using the built-in Campbell Web Service API to collect flux data directly from field devices, we found it was frequently returning empty results but that was happening more consistently over time. So we’ve switched to collecting data hourly, via FTP.
The existing code in the ingester to read and transform the binary tob3 format was really useful for this. IT Infrastructure quickly deployed the new version, and it’s ready to add more flux sites as they go out into the field - Mill Farm next. These are hour-long snapshots of 20Hz flux data, ready for the next step in the ingestion pipeline which is being broken up into the format that EddyPro wants to read.
AWS Migration
We’re working to switch across the Campbell dataloggers to use the new AWS production system. (At the moment they are connecting to the MQTT service in the testing AWS account and the data is then replicated to the new account).
A round of testing is needed and there’s always overlapping annual leave, but there’s nice progress on this. It shouldn’t cause any extra effort on the part of the WP1 field team, but appear transparent.
We now have some new campbell dataloggers going directly to our new AWS account!
Network monitoring
In the network monitoring dashboard, we now have live battery status information for each site. When drilling into a single site this is displayed on gauges together with a threshold value. Also you can see that we have gauges for Data Completeness and Quality Control, but they are yet to be powered by live data…
…however, we have tested using Apache ECharts’ heatmap with live data to create a working heatmap to visualise Data Completeness and Quality Control beneath timeseries graphs. This will be used in both the dashboard and public UI. Here is the mockup followed by the very raw test against live data - its not supposed to look pretty yet:
Mockup