Week 19 - Gridded Data edition
Matt DP here, back from my weeknotes-hiatus! Lots happening in the gridded-data-area of FDRI, some of which briefly hinted at in last week’s note.
Datasets 🌧️
UKCEH has lots of great datasets that are widely used in important science. We’re working to make some of these more accessible and easier to work with by convering them from a format of often hundreds or thousands of NetCDF files to a contiguous, cloud-friendly format known as ‘Zarr’, presumably invented by pirates. Once converted these datasets can be made available for access from anywhere in the world with an internet connection, and don’t need to be downloaded in order to be analysed and visualised. Sweet! Here’s a list of datasets that we’ve converted, and those that are work in progress for this year:
- Available Now: CEH-GEAR-1hr: An observations-derived hourly 1km resolution gridded rainfall dataset over the UK from 1990 to 2016
- Partially available: CHESS-SCAPE: Projected daily 1km resolution gridded meteorological driving data aimed at hydrological models for the UK under different climate change scenarios for 1980-2080. RCP8.5 currently available.
- Coming soon: CHESS-Met: Observations-derived daily 1km resolution gridded meteorological driving data aimed at hydrological models for the UK
- Under consideration: HadUK-Grid Rainfall: Daily 1km resolution observations-derived gridded rainfall datasets over the UK, produced by the Met Office
- Under consideration: Potential EvapoTranspiration: There are various versions of PET data, which is often a key input for hydrological models
- Under consideration: Abstraction Data: Gridded observationally-derived estimates of monthly water abstracted from the hydrological system for various uses, helpful for more accurate modelling of river flows.
Any further datasets we could consider for making accessible in this way, let us know! See the notebooks below for how to easily access and work with these datasets!
Notebooks notebooks NoteBooks! 📘📖📑
Converting the datasets is all very well and good but they ain’t much use if they’re a pain in the proverbial to access! Therefore we’ve made some example code notebooks in Python or R that walk you through how to access the CEH-GEAR-1hr dataset in either it’s new fancy format or NetCDF and perform simple analyses on it, without needing to download the data. We’ve made these notebooks available on Google Colab, Binder and the JASMIN Notebook Service, the first two allowing one-click access-and-run use, no faffy package installations and environment setup needed! (We’re working on this for the JASMIN Notebook Service too). Feel free to have a play around with them and feedback what went well or not, or what more you’d like to see. The idea is to have one of these for all of the datasets we make available in this way, so this is very much just the start!
Insert Engaging Content here 🧐
These notebooks were demoed at a end-user engagement session run by Kit and I to get formal feedback on experience of our target end-users with Python/R, Notebooks, and various data formats, as well as what they thought about our current example data-access notebooks. The formal report is currently being written up but here are some early notable takeways:
- Users were most familiar with NetCDF as a format, Zarr relatively unknown, but a perceptive comment highlighted that users will largely work with whatever format the data comes in
- Data access hurdles were cited as a common frustration, along with the scattered nature of data repositories and the amount of wrangling needed to work with chosen datasets
- The majority of users were familiar with notebooks and how they work. There was a lot of familiarity with the Google Colab platform, much less for Binder and other, newer notebook access options such as Marimo and Jupyter Lite. Very few were aware that notebooks also support R, not just Python.
What does the future hold?
Well, at least for gridded data products in FDRI, this is what’s on our ralidar:
- Access to LIDAR datasets and terrain/elevation models via notebooks
- More large gridded datasets converted to Zarr and made accessible via notebooks
- Potential direct integration with the EIDC (TBC)
- Integration with the spatial data explorer, or other means of easy visulisation of the datasets (watch this space!)
Think that’s probably enough to be getting on with for one week xD
We’ll be back soon with more from the other awesome FDRI products being developed. Until then!