2025 Week 9
Another weekly update of FDRI developments
🏗️ Infra
https://github.com/NERC-CEH/dri-infrastructure
Atlantis: All our infrastructure is managed with terraform and atlantis, this week we had problems with atlantis where it seemed to just stop working because CPU usage of the underlying ECS task was hitting 100% There were no errors it simply stopped, this was difficult to debug but scaling up the ecs task fixed it for now.
Atlantis CPU usage:
AWS Solutions Architect: There are some ongoing talks around our use of the AWS at UKCEH led by Gordon and Mike Brown about if it should be used for other projects outside of FDRI. As part of these discussion we were able to catch the ear of a solutions architect at AWS and now have them available as a resource to ask any questions we have around our solutions and use of AWS tech. There is also plan to run have a more dedicated session in person where we can run through everything we have done so far.
🥫 Ingesting data
https://github.com/NERC-CEH/dri-ingestion
We have an ingester service that listens to incoming COSMOS data and ingests it into nice to use parquet files. The performance of it was working well, until the COSMOS network went down and a huge amount of messages came in at once when it came back up.
We were able to speed it up ~50x! The speed has increased from ~19 messages/min to ~960 messages/min
Here’s the messages stuck on our COSMOS ingester queue, reducing after our fixes
🏁 Time series model Flagging
https://github.com/NERC-CEH/dri-timeseries-processor/tree/main/src/time_series
We have been building a generic timeseries data module in python to handle common timeseries data problems. Part of what this supports is flagging data (e.g flagging a value as suspect, missing etc), this week we have been adding convenience methods to the TimeSeries class for interacting with flag columns, and adding additional checks when creating flag columns for added robustness of the flagging systems.
❇️UI
https://github.com/NERC-CEH/dri-ui/
First Users: We had our first users of the UI last friday! Since we don’t have FDRI sensors or data yet, we have been using the COSMOS data and have been able to provide a value add to the existing COSMOS field engineers data checking process. We received very positive feedback overall and it’s great to get this high level of back and forth.
Land Cover Map: Today we got the first pull request into the UI from another team, it’s great to see another project contributing to shared DRI resources. We get a bunch of benefits collaborating across teams like this into a shared codebase.
Here’s our DRI-UI intro slides if you would like to learn a bit more about the tech we are using.
📃 API
https://github.com/NERC-CEH/dri-data-api/
Ongoing talks around how a specification led API would look like. Our investigation has been showing that the Sensor things OGC api spec may not be a good fit for what we are doing. The only real implementation of this standard is [FROST](https://github.com/FraunhoferIOSB/FROST-Server which has a number of limitations, main one being all data and metadata must be in a postgres db. There’s a number of talks happening with other orgs and epimorphics on what our options are and how we can provided a standards led API for our data and metadata.
📸 FDRI Phenocam
More progress from the tiny phenocam team this week too, the phenocam images are now streaming into a linux server where we can write code and we have managed to upload some images to s3 ready to be served by and API.
📚 Dev Book Club
We are kicking off a dev book club! The first book is designing data intensive applications. Please reach out if you would like join. We will be meeting every 2 weeks, starting 10.03.2025 to discuss a chapter of a book with no requirement to have read the chapter.