2025 Week 24
Oh no it’s a bit late, here’s last weeks round up.
🖌️Updated Diagrams
Overview
Here’s an overview of what we are building
| Colour | Section |
|---|---|
| green | Data Ingestion |
| red | Data Storage |
| yellow | Metadata management |
| blue | Data Processing (QC etc) |
| purple | Data Access and Visualisation |
| orange | Data Publishing |
Technologies
Not terribly useful unless you recognise all the hidden icons.
🐘 Using a database
We have a new Architecture Decision Record (ADR) on how we will be managing the database migrations.
Which has led to a repo setup here: https://github.com/NERC-CEH/dri-database-models
🏗️ Architecture Decision Records
Following on from our together time ways of working discussions, one action item was to document more things in the fdri_words repo (which is where these are updates are written).
All of our ADRS have been migrated from sharepoint and hopefully using markdown/github will help encourage us to write them more.
💻 Productive talks with IT about working together
Some enthusiastic and encouraging talks with IT on how we can better work together on the FDRI project. Leading to a list of areas where we could work together/ get some help from IT.
Listed here.
| Project | Item | Notes |
|---|---|---|
| FDRI | AWS IAM User Account Management | Active directory setup, user groups defined and allocated |
| FDRI | AWS terraform/atlantis setup | |
| FDRI | Managed K8S maintainance | K8s yaml/flux setup, Good k8s experience in DR already |
| FDRI | Managed RDS maintainance | As above, it’s a managed service so mostly dealt with. Might not be worth getting IT involved. Any IT DB experience might help us |
🦇 NRFA processing “batches”
A new concept of a “batch” is being discussed, this is when new data is recieved for NRFA (could also apply to other networks) and it needs checking before it can be merged into the main timeseries. This is a common part of the existing NRFA workflow and an important point for us to conceptualise consistently across the team and with stakeholders
⚾ DRI-Replayer
As part of our debugging of the ingestion piece of our pipeline we have created a dri-replayer scripts repo. It helps with querying the ingestion message status via dynamodb and allows replaying messages that failed, along with deleting and tidying up stuck messages. This work was mostly done months ago but has been improved and used a lot recently with some current bugs.
📛 How to name an image
It’s one of the two hard things in software engineering and has led to lots of discussion between WP1 and WP2 on how to name the incoming images from the raspberry pi based camera, that’s going to get deployed next week. Naming things is really important to make the context clear and in our case make sure the partitioning is as such we can retrieve the images in the ways we want to efficiently.
🌲 Ongoing work to merge the COSMOS and FDRI UIs
We are marking progress on getting the live FDRI data so it can be plotted like the COSMOS data . The work is ongoing, but we are making progress!