2025 Week 32
As always there’s lots going and it’s never all covered in the weeknotes, here’s some of the latest!
NRFA data ingested and plottable!
this isn’t quite released but it’s a huge milestone for getting a third network of data into our systems. The COSMOS and FDRI data systems are both very similar, NRFA is quite a bit different in the way it’s ingested and introduces some new concepts to our code base. There was a huge amount of work involved in getting the NRFA data to this stage, it’s still in development but as you can see from the screenshot we are very close to getting this released.
Hartree Visit
Some of us from UKCEH travelled to STFC Hartree last week for some cross org knowledge sharing/networking. It was a great day in warrington, some key insights
IOT HUB
Hartree worked on a similar project with SMDH to FDRI’s IOT management, where they are managing 100’s of IOT devices for different users using AWS IOT core. They felt at that scale (same as FDRI scale) the need to build custom management platforms ontop of IOT core, we were thinking we might need this and it was great to get another viewpoint.
AWS Landing Zone/Control Tower Setup
Hartree currently mostly work with corporate clients, building projects in AWS accounts and then handing over the AWS account at the end of the project. CEH are quite different but they had some good insights into using control tower effectively across many accounts.
New Supercompute capacity
Hartree Centre have a new super computer available soon with H100’s, it’s paid for and aimed at industry users but could be utilised by UKCEH if we have the business case.
Data Lake Cataloging
They heavily use parquet and s3 buckets, not quite in the same way, but they have had success using it across different projects. They also use iceberg on top of it and had great things to say about apache polaris which sits on top of iceberg REST apis to provide a single place to query and handle data access. It can be used across clouds/s3 buckets as well as to access any blob binary data as long as an iceberg rest api can be put it front of it. Something for us to look into.
Simplified diagram
As part of the visit, we presented some of the FDRI work here’s a simplified overview from the slide deck. Original overview.
End to End Testing of the processor
We now have data passing through our processor and being written out the otherside. Many things are still missing and work in progress but with this milestone, we are now able to write end to end tests of the pipeline. This should help us on the way to create a world class network agnostic sensor data pipeline.