2025 Week 18

Skipped a few weeks because of bank holidays and taking some annual leave, but week notes are now back! As always lots happening, kicking off with a gridded update from Matt Brown

🪟 Gridded Data Update (Matt Brown)

This week in gridded-land it was a planning week with lots of work going in to figuring out where to spend time and effort this year. That’s not wonderfully exciting for what we’re going for here though, so instead here’s a couple of things that have happened over the past month or so:

Some example of working with datasets on object storage are now available on datalabs* for users to have a play around with. The idea is to develop these further into a gallery of well-documented examples users can adapt for the analysis they wish to do. For now though, they focus on the 1hrly GEAR Gridded Rainfall dataset. Take a look and let me know what you think - suggestions for further examples or improvements very welcome, along with any bugs you find!

I’m also slowly adding datasets to the FDRI object storage tenancy on JASMIN. As of this week the Daily version of the GEAR dataset is available, and soon the CHESS-MET dataset will be too. The code snippet allowing you to access this dataset from anywhere is:

import xarray as xr
import fsspec

mapper = fsspec.get_mapper('s3://geardaily/GB/geardaily_fulloutput_yearly_100km_chunks.zarr', 
                           anon=True, 
                           endpoint_url="https://fdri-o.s3-ext.jc.rl.ac.uk")

ds = xr.open_zarr(mapper, consolidated=True)

*You’ll need to have a Datalabs account and be a member of the FDRI project. Let me know if you’re not and I can add you.

🧑‍🤝‍🧑 Hiring process improvements and offer sent

A few weeks ago we redesigned our interview process in the Digital Research group and got to try it our in our latest round of interviews for a Software Engineer. It wasn’t perfect and there’s definately ways we could improve it but we are confident it helped us assess the candidates in the ways that we wanted to and are confident it’s helped us narrow down the best candidate which we have extended an offer to (hopefully joining soon!).

☁️ AWS <> UKCEH Meeting

This week a few of us met with AWS in Manchester to start the basis of a closer strategic working relationship. It was very insightful to learn about the AWS way of approaching technical problems and we were able to kick off by explaining in a somewhat chaotic way everything we are doing on FDRI and plan to do in future (both fdri and non-fdri). The solutions architects we met with were all really switched on and had a background in science and research, and we look forward to working closer with them to get confidence in our design decisions and suggestions for improvements going forward.

🥈 Year 2 Planning

Lots of work has been happening in moving forward with our year 2 plan and breaking it down into implementation pieces. This is still ongoing but mapping out our rough aims for the next 12 months has been a valuable exercise.

🍌 Kanban

The FDRI timeseries dev team have been working using agile sprints for the last 12 months but are now moving to Kanban, we hope this helps us with the unpredictable priorities that arrive during our current sprints due to the greenfield nature of our work and how much of it is discovery and difficultities in planning out the correct priorities.

🏗️ Moving FDRI into our proper COSMOS built data pipeline

We were able to quickly ingest the new fdri sensor data when it came online and quickly hacked together a user interface to view the data , we’ve now be working on rethinking the FDRI sensor data and doing it a bit more proper following the best practices we developed for the COSMOS test data we have been working with. The end result of this is less code overall and the ability in the UI to view charts for the fdri data and the ability to view a data table for the cosmos data.

❇️ UI

There’s been 2 main updates on the UI recently:

• Gaps in the data are now clearly highlighted, see screenshot above or click here
• Custom date ranges can be selected (more than 10 days of data) here’s the last 2 months of data (scroll the x axis to see more)

🛀 Processing Pipeline

Main updates on our processing pipeline are that it’s now properly driven via config living in our metadata service and we are now working on handling dependency data for derived values.

📷Phenocam

More high level planning on what to tackle next for phenocam. Our current plan is to start tackling tagging images, which after some initial investigation into https://labelstud.io/ we have decided it’s not quite suitable for our usecases and will be moving forward with building our customer user interface. A mockup for our plan is above.