Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

The Connected River (Part 1) - Building

A reusable workflow for joining environmental monitoring data through river networks

UK Centre for Ecology & Hydrology

A reusable workflow for joining environmental monitoring data through river networks

This work was carried out as part of the UK Environmental Exposure Hub (EEx Hub), developed through the UKCEH National Capability for UK Challenges (NC-UK) programme and funded by NERC (NE/Y006208/1).

What this notebook does

Biology, chemistry and pressure data are collected by different monitoring programmes, often at different points along the river network and through time. This notebook presents a reusable workflow for bringing those records together as an England-wide dataset at biological-survey level.

The main problem is deciding which chemistry station can reasonably represent each biological survey. The workflow traces possible connections through the river network, compares the available evidence and records why a match was accepted or rejected. The England-wide example demonstrates the method; the reusable matching workflow is the main output.

Question: How can monitoring datasets collected along the river network and through time be joined defensibly?

Before running

Edit the settings below, then choose Restart Kernel and Run All. build_england downloads and processes the national example. The first run can take several hours, but downloads and completed stages are cached. use_existing reopens a finished output.

Two UKCEH inputs need to be obtained separately under their own licence terms:

The workflow needs the topology-enabled UKCEH river-network CSV, including link and node identifiers; a line-only map layer is not sufficient. Set both local paths in the settings cell. The run information records which files were used; the source files themselves are not copied into the output. Once this notebook has finished, The Connected River (Part 2) - Exploring opens the completed output for the maps and analysis.

Settings for most runs

Start here before choosing Restart Kernel and Run All. For the England-wide build, most users only need to choose the run mode, provide the two licensed spatial inputs and set the final year. The workspace and refresh values can usually stay as they are.

  • Use build_england to create or update the national dataset.

  • Use use_existing with a completed output to inspect the method without rebuilding.

  • Paths may be absolute or relative to this repository.

Advanced matching settings

The settings below define the matching recipe. Change them when testing another temporal window, restricting eligible routes, changing the 100-point ranking or supplying a different directed river network. Leave them unchanged to reproduce the default example.

Build the evidence base

The next cell imports the project functions and either builds or reopens the output. Most of it is setup, so the cell is folded. The implementation is kept in ordinary Python modules under connected_river/, leaving the main steps visible in the notebook.

Source
Loading...
<Figure size 1500x420 with 1 Axes>

Source data and refresh strategy

The sources update at different times and arrive as APIs, bulk downloads, tables and spatial layers. The pipeline downloads them where possible and records the source, retrieval date and file size. The table below comes directly from the source catalogue used for this run. required sources is the number in each group needed for the default England-wide joined dataset.

Loading...

Harmonisation before matching

The catalogue lists where each source comes from and the run information records the files actually used. Licensed river-network data are registered locally and are not copied into the output.

Before matching, the code standardises sites, dates, coordinates, identifiers, units and biological survey records. The specialist output keeps the original values and source information. Missing measurements stay missing.

Loading...

Match sites through the river network

The table above comes from the generated data dictionary and describes the current output. The final unit is one biological survey; chemistry is added only after the source checks and matching steps below.

A nearby site is not necessarily on the same connected river. The workflow therefore keeps the stages separate:

  1. check which sites can be snapped to the network;

  2. trace connected routes between plausible pairs;

  3. describe distance, direction, tributaries and available monitoring evidence;

  4. rank the eligible routes; and

  5. apply the scientific acceptance checks.

The ranking step compares possible routes. A route must still pass the evidence checks before it can be accepted.

Candidate construction before selection

All connected alternatives are retained until a station is selected. For each route, the table records distance, direction, snap quality, temporal support, tributaries, pressure context, component scores and the final decision.

The first figure checks whether the supplied monitoring coordinates can be represented on the national network. Later figures deal with whether those connections are suitable for analysis.

<Figure size 1350x670 with 2 Axes>

Ranking and scientific acceptance

The next cell prints a sample of the candidate table and follows one biological site through the ranking step. The comparison keeps both the highest-scoring route and the route selected after the evidence checks.

Loading...
<Figure size 1350x580 with 2 Axes>
Loading...

Adapt the matching recipe

A matching recipe is the set of choices used to move from all connected candidate routes to one chemistry station for each biological survey. It has three parts:

  1. candidate filters decide which routes are relevant to the question;

  2. a 100-point score ranks the remaining routes; and

  3. final checks decide whether the best available route can be included.

FILTER
Which connected routes are eligible?
→
RANK
Which eligible route has the strongest 100-point score?
→
ACCEPT
Does the best route pass the scientific checks?

The default ranking score is:

Evidence used for rankingPoints
River-network distance35
Network snapping quality15
Temporal support20
Chemistry support20
Transport plausibility10
Total100

A higher score changes the ranking, but it does not bypass the acceptance checks. Custom weights must also add to exactly 100 points.

The candidate table can therefore be reused for different questions without rewriting the whole pipeline:

  • require metals with wq_has_metals:eq:true;

  • keep upstream chemistry with network_direction:eq:wq_upstream_of_bio;

  • add route context such as path_permitted_sewage_outlets:gt:0;

  • change antecedent_window_months from 6 to 12;

  • supply CUSTOM_MATCH_RECIPE to change the ranking weights; or

  • use custom_irn_schema to map another directed river network.

Filters use column:operator:value and AND logic. They are applied before a winner is selected, and the number of rows removed by each rule is saved.

Loading...
Loading...

What changes when the weights change?

The example below keeps the candidate filters and acceptance checks unchanged. It moves 10 points from river-network distance to chemistry support, so the total remains 100. This tests the effect of the ranking choice on coverage and on the station selected for each survey.

<Figure size 1350x560 with 2 Axes>
Loading...
Loading...
Loading...
Loading...

Worked candidate pre-filter

This example keeps only chemistry stations upstream of the biological site. Direction is a route property, so it can only be applied after candidate routes have been traced; it is still a pre-filter for ranking and final selection. The example uses a separate result and does not change the main national run.

Loading...
Loading...

What changes when candidate filters change?

These examples keep the 100-point weights fixed and change which routes are eligible. The comparison shows how much coverage is retained and whether the selected chemistry station changes.

<Figure size 1350x560 with 2 Axes>

Target-specific chemistry window

Chemistry is summarised separately for each biological survey date. Samples outside the declared window are excluded, including any taken after the survey date.

The window is a user choice, not a fixed part of the method. Change MATCHING_OPTIONS["antecedent_window_months"] in the settings cell (for example, from 6 to 12) and rebuild to use the previous year. The diagram below always reads the active setting, while the run information records the value used to build the dataset.

<Figure size 1300x390 with 1 Axes>

Replacing the river network

The method is not tied to UKCEH field names. A different directed network can be used by mapping its columns to the required fields below and setting NETWORK_MODE to custom_irn_schema. The replacement needs line geometry in one projected CRS and a consistent upstream-to-downstream node direction.

Loading...

Review the matching outcomes

The decision table includes accepted matches, records held for review and cases where no defensible match was found. The river-network-distance distribution is shown alongside those outcomes.

<Figure size 1300x520 with 2 Axes>

Write and verify the joined product

The configured workspace receives all generated data. The compact CSV is intended for routine analysis; the specialist Parquet keeps the full source schema, route evidence and matching diagnostics.

Files created

The workflow creates three groups of outputs:

  • the compact analysis table and full specialist table;

  • all ranked routes, selected matches and filter/recipe settings; and

  • the data dictionary, source details and output checks.

The next cell writes the current recipe and inspects the products that were actually created.

Loading...
Loading...
Loading...

Check the joined data

Before the output is used, the checks compare record identities, data-dictionary coverage, expected files and stored row counts. The saved matching tables also keep excluded candidates, review reasons, route evidence and the settings used for the run.

Loading...

What has been built?

This notebook turns separate monitoring records into a connected, traceable dataset. The same sequence can be reused with different source data, another directed river network, a different temporal window or a revised matching recipe.

Each run keeps the candidate routes, 100-point score, acceptance decisions, source records, settings, source details, data dictionary and check results. A joined value can therefore be traced back to the evidence and assumptions that produced it.

The Connected River (Part 2) - Exploring starts with this completed dataset and shows what can be learned from it.