2025 Week 11
Another week another update of FDRI developments.
❇️ UI
The main change on our UI this week was enhancing our data exploring chart to allow cross-site cross-dataset data to be plotted on the same chart. You can try it out here .
🛁 Processing Pipeline
Don’t think this has been shared on the weeknotes before, there’s been some work rethinking how we design our processing pipeline. It’s captured very well in these slides
🤘Metadata Management
Metadata is a huge part of what we are building and it’s involved in all parts of our digital research infrastructure (DRI), from data ingestion, data processing, data fetching and data publishing.
There’s lots of effort happening in many places to figure out how this all fits together and gets modelled. This week we’ve been working on the using the metadata service created by epimorphics (RDF triplestore with generic api layer on top) more in our data processing pipeline to pick up the required processing config and in our data api to improve our ui (displaying variable names instead of ids and plotting data using variable units for example). This is still ongoing work but crucial to our project and lots of discovery and learning happening as we try to fit these pieces together.
📜 WP1 Second FDRI Logger has been setup
Work Package 1 got a second FDRI logger setup, this was the first time setting it up using our automated terraform setup. The setup is as straight forward as adding the loggers client id here, this automatically creates the IOT core “thing”, creates the certificates for it to communicate securely and all the required permissions policies.
🔍 IOT raw message browsing/searching
We’re still trying to figure out a nice solution to allow easy browsing of the new fdri sensor data, we were hoping that setting up an iot rule to forward data to AWS opensearch would be a quick win. The ongoing work is here it’s proving quite tricky to get it setup correctly, we have it deployed as in the screenshot but the iot data isn’t coming in, we will continue for now, but are starting to think this might not be the quick win we thought it might be.
🫘 Times Series Plotter
Part of what we are doing with our data exploration ui is replacing the need for the time series plotter. Each time we look at the time series plotter, I’m always amazed at how flexible it is and how much it can do. It’s a daunting task
📷 Phenocam
We have been thinking about what to do next in the phenocam project once we have the images visible in our ui. Currently we are thinking we can start on the processing pipeline for the images using some masks to create the RGB timeseries data, this is still being discussed but sounds reasonable to follow what we are doing on the timeseries team.
One thing that came up this week is the iFDO standard being led by the marine research community but has some clear parallels to the phenocam image processing work we want to do.
🧊 Kubecon 2025
https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/
Kubecon is in london this year, Josh Holland and myself (dom ginger) are planning on attending, please reach out if you are attending :)
📚 Dev Book Club
The Dev Book Club started this week! Thanks to everybody that came, I found it very useful and I think we had some great discussions. If anybody is interested in joining, we are reading designing data intensive applications , next meeting will be about chapter 2: Data Models and Query Languages, can be joined at any time, no need to have been to previous sessions and no need to read the chapter, just a want to discuss the subject.