2025 Week 20

A workday late, whenever I write these I start by thinking not that much happened this week, then find loads of things crop up, here’s what I remember from last week, most definitely missing some things.

🪴Prep for first field site

Fixed some critical bugs on our FDRI UI, one was blocking new data coming in when a value was rounded to zero decimal places, its was being assumed an int and clashing on the schema. Second bug was with the sorting on our UI where when sorting it would only sort the current column, instead of all rows by that column.

We’ve also added some reliance to our data pipeline and api to support duplicate values (which should never happen, but sometimes does :) ) . This is in the time-stream package and shared across the data pipeline.

We think when WP1 deploy the sensor (K3CN-QJ9X-6HQ8) in Wales next week, everything should just keep working. But this is perhaps too optimistic and we have put time aside to monitor our systems closely during the deployment time.

Cross org Sensor Data Pipeline session

There was a large cross org technical knowledge sharing session this week, where UKCEH, Newcastle Urban Observatory, Forth-Era, NOC and others were attending. Had quite a large turnout and great engagement from everybody there, lots to learn about and we could definitely spend more time discussing everything.

Jo’s excellent notes from this meeting

📆Year 2 Planning

Year 2 planning is being finalised and has been moved into our jira board 149 epics, lots to keep us busy with!

🚪 Goodbye Lewis

This week we said goodbye to Lewis Chambers, one of our FDRI WP2 Development team founding members. He’s had a huge impact on the project and it’s really sad to see him leave, wishing him all the best for his future endeavours. Above is Lewis’ last message on teams 🥹

🚰 NRFA design meeting

As part of Lewis leaving (see above) we had some handover meeting which turned into a design meeting for the NRFA data. Our current thinking for this part of the FDRI project is to not use parquet/s3 like COSMOS/FDRI and instead use Postgres, the reason for this is that the current NRFA flow using an Oracle DB and has a lot of complex manual workflows involve sql scripts/write access to the db, which would take a long time to implement on top of parquet, which would mean the old system would continue to be used for a long time. We think it makes sense to try to transistion to a PostgresDB as quickly as possible to get users using our system, then improve the workflow from their reducing the number of sql scripts being run manually. This plan is still very rough thoughts and likely to change.

🍌 Started Kanban

Kanban has begun! Difficult to tell if it’s working only 1 week in, but seems okay so far.

Merging of the timeseries and phenocam teams

The teams are now merged and the collaboration is working nicely. There’s still some admin moving the jira projects together/bringing in the year 2 stuff.