2025 Week 25
As always lots of things happening, hope everybody has a sunny weekend
🧑🤝🧑 New Developer
This week we had a new developer join us in the Lancaster office. Samantha Hewkin has joined the FDRI team to help us out across the board. In her first week she has already smashed out 2 pull requests! Very impressive. Please say hello if you see her around.
📸 FDRI field cameras
There’s a new observation site being set up in Wales next week, and as part of this we’re deploying Raspberry Pis with an attached camera module.
This effort has been led by Jack Hambridge in the field engineering team, with programming support and coaching by research software engineers in WP2. The cameras aren’t yet being used for Phenocam-style observations, only to provide some monitoring for conditions on the site. The devices have a custom housing designed by Will in the UKCEH Engineering Workshop.
They’re capturing images on a schedule and uploading them directly to AWS object storage. The aim is to re-use the same interface development work that was done for COSMOS to provide the engineering team with a feed of camera images.
This is a trial installation and the Pis are not registered as Things in AWS IoT Core yet, but that’s part of the intention for managing them in the future - with plenty of ideas kicking around for what can be done with tiny Linux computers in each FDRI site.
🌩️ Production Data on our Test Environment
With the hefran forest site going live and the other site going live next week we are now receive production data, which is heading directly into our staging/test environment. We are in the (very slow) process of procuring a production environment to help rectify this. One potential solution is to rename our test environment to production and start treating it as such, then the newly procurred environment can become our new test environment. Althought this is how things are for now, we have good practices around our data and it’s stored securely and safely in an s3 bucket, which only admins having access to delete it (our deployed code doesn’t have delete permissions).
In future we are looking at other ways to backup this data:
- Backup for S3
- AWS DataSync to our on-prem s3 ceph object storage
- Some cronjob or something to backup to the on-prem SAN which is where vast major or other project data is
🖐️ Image data partitioning
Lots of back and forth on how we partition the incoming image data, this is important to get right since it’s tricky to change onces the cameras get deployed since we are going live without any remote device management. The partitioning structure if setup correctly should allow our query use cases to performantly query the image data in the bucket e.g give me images from site X between date y and date z without us having to store an alongside index. We didn’t get time to test the performance before going live but think the following should allow that and we can always add an index in if needed without needing to update the raspbery pi camera code.
/catchment=SE/site=CARGN/compound=01/type=PCAM/direction=E/date=2025-06-18/SE_CARGN_01_PCAM_E_20250618_154719.jpg
🔁 DRI-Replayer
In fixing some of the ingestion bugs found when looking at the production data we created a new git repo containing helper scripts for debugging and replaying stuck messages.
🦭NRFA data ingestion
Huge ongoing effort to pull together different data sources into a single format to prepare for processing has been ongoing. There’s a mix of CSV, XML and custom structured text files being processed which represent the same data in various ways, some examples above, hopefully help show how much work this is.
🌃 Truncating Data
Some interesting discussion on how in a time series we handle the period boundries. For example a 15 minute timeseries with the following data
00:00, 1
00:15, 2
00:30, 3
00:45, 4
01:00, 5
The value at 00:00 could be referring to the 15 minute period up to midnight or the 15 minutes between 00:00 and 00:15 so should it be included in the current day or before.
🏞️ Chilterns National Landscape
The wider FDRI project had a day with the Chilterns National Landscape and we presented our existing UI work. They seemed impressed and interested! Which is a great place to be.
🤔 Ongoing workstreams in the team
Diagram of our current ongoing workstreams in the team, changes almost daily but it was useful for me to draw it out.