Skip to contents

MetQC App User Guide

Peter Levy

2026-07-21

app_user_guide.Rmd

This document explains how to use the MetQC Shiny app bundled with the metamet package. It covers every tab in the app, all major controls, and includes a step-by-step FAQ for common tasks.

Prerequisites

The app requires a set of optional packages that are not installed automatically. Install them before launching:

# Recommended: install metamet plus all optional dependencies in one step
install.packages("pak")
pak::pak("NERC-CEH/metamet", dependencies = TRUE)
# Or install the app packages individually
install.packages(c(
  "shiny", "shinydashboard", "shinyjs", "shinyFiles",
  "shinyvalidate", "shinycssloaders", "ggiraph", "glue"
))

Launch the app from R:

metamet::run_shiny()

The app opens in your default web browser. It will stop automatically when you close the browser tab or click Stop App in the left sidebar.

App layout

The left sidebar contains five items:

Sidebar item Purpose
Create new Metamet object Six-step wizard to build a .rds from raw data files
Open existing Metamet object Browse and load an existing .rds
Select date range and QA/QC Main dashboard for interactive data validation
Download processed data Export processed data as CSV or ZIP
Help and Documentation Links to gap-fill methods and this guide

Your system username (from Sys.info()[["user"]]) is used automatically to label any changes you make. It is not currently configurable within the app.

Tab 1 — Create new Metamet object

This six-step wizard takes you from a raw data file to a saved metamet .rds file ready for QA/QC. Each step must be completed in order; use the Back and Continue buttons to navigate.

Step 1: Load data file

Select the format of your raw data file:

  • Plain CSV — a standard CSV file with a header row.
  • Campbell TOA5 — data logger output in TOA5 (ASCII) format.
  • Old Campbell (.dat + .dld) — older Campbell files; you will be prompted to select both the .dat data file and the .dld metadata file.
  • CEDA BADC-CSV — the BADC-CSV format used by CEDA data archives.

Click Select data file, browse to your file, then click Load & preview. A preview table will appear. If the file contains multiple tables (Campbell TOA5), a drop-down lets you choose which table to use. Select the column that contains the timestamp, then click Continue.

Step 2: Site information

Provide site-level metadata. You can either:

  • Enter manually — fill in the Site ID, site name, latitude, longitude, elevation, and the date range over which this metadata applies.
  • Load from CSV file — select an existing dt_site CSV and choose the row corresponding to your site.

Click Continue to variable mapping when ready.

Step 3: Map data columns to ICOS variable names

Each column in your data file must be mapped to a standard variable name. You can:

  • Map manually — for each data column, select the matching ICOS variable from a drop-down list. Columns not mapped to any variable can be left as — skip —.
  • Load dt_meta from CSV file — if you already have a dt_meta metadata file, load it here and the mappings will be applied automatically.

Click Continue when all required columns are mapped.

Step 4: Set units and QC ranges (optional)

For each mapped variable you can set:

  • Units — the physical units of the raw data.
  • Valid range — minimum and maximum acceptable values for automatic range checking.
  • Imputation method — the default gap-filling algorithm to apply to this variable.

This step can be skipped; settings can be refined later. Click Continue.

Step 5: ERA5 reference data (optional)

Attach ERA5 reanalysis data as a reference dataset (dt_ref). Select an ERA5 CSV file (as exported by add_era5()). The reference data will be plotted alongside observations in the QA/QC dashboard. Click Continue (or skip).

Step 6: Review and save

A summary of the object is displayed. Click Download as .rds to save the metamet object to a file. This file is the input for the QA/QC dashboard.

Tab 2 — Open existing Metamet object

Click Browse for .rds file, navigate to a previously saved metamet .rds file, and select it. The file is loaded and the app switches automatically to the Select date range and QA/QC tab. A notification confirms the file name that was loaded.

The .rds file must be a metamet object in long format (or wide format — the app reshapes it automatically).

Tab 3 — Select date range and QA/QC

This is the main validation interface.

Selecting a date range

The Data Selection box contains:

  • Start date / End date — date pickers pre-filled with the earliest and latest timestamps in the loaded file.
  • Hour / Minute — fine-tune the start and end times within the selected days.

Click Retrieve from database to extract the chosen time window. The Extracted Data panel appears below.

After retrieval, the Compare variables button becomes active. Click it to open a scatter-plot modal comparing any two variables in the extracted window.

Extracted data panel

The panel contains one tab per variable. Each tab shows:

  • Replicate checkboxes (if the variable has multiple replicates) — tick/untick individual sensor replicates to show or hide them on the plot.
  • Interactive plot — a time-series plot produced by ggiraph. Point colours indicate the current QC code (grey = valid, coloured = imputed or flagged). Hover over a point to see its timestamp and value.
  • Rescale reference to observations checkbox — when ERA5 reference data is present, tick this to rescale the ERA5 values to the observation range before plotting.
  • Point size slider — adjust the visual size of plotted points (0.5 – 8).

Imputing (gap-filling) data

  1. Move the mouse over the plot area. A toolbar appears in the top-right corner of the plot. Click the lasso selection button (the left-most icon).
  2. Draw a lasso around the data points you want to replace. Selected points turn red.
  3. Choose a Gap-Filling Method from the drop-down below the plot. Available methods are described in the Help and Documentation tab (and in the gap_fill_methods vignette):
    • Time interpolation — GAM spline through time; a smoothness slider appears.
    • Regression — linear regression against another variable; a covariate drop-down appears.
    • ERA5 — substitute values from the reference dataset.
    • Zero — set selected values to zero.
    • Non-negative — set negative values to zero.
    • Night zero — set night-time values to zero.
  4. (Optional) Type a brief reason for imputation in the comment box.
  5. Use the Do not alter data estimated by checkboxes to protect any previously imputed points from being overwritten.
  6. Click Impute selection. The plot updates immediately.
  7. Repeat steps 1–6 as needed for the same or other variables.
  8. When a variable looks correct, click Finished checking variable for date range. The variable’s tab turns grey to indicate it has been signed off.

Saving changes

Click Save changes when you are satisfied with all variables in the selected date range.

Important: unsaved changes are lost if you click Retrieve from database again for a new date range. Always save before moving to a new window.

The app writes two new files next to the source .rds:

  • <original_name>_qc_by_<username>_on_<date>.rds — full metamet object with updated QC codes and imputed values.
  • <original_name>_qc_by_<username>_on_<date>_ceda.rds — CEDA-formatted output.

A notification confirms the files were created.

Click Restart app at any time to reload the app and start fresh.

Tab 4 — Download processed data

Select the format from the drop-down:

Option Output Contents
Level 1 .zip containing two .csv files Raw data (dt) and QC codes (dt_qc)
Level 2 .zip containing two .csv files QC-processed data and updated QC codes
CEDA Single .csv file CEDA BADC-CSV formatted output

Click Download to save the file.

Tab 5 — Help and Documentation

Sub-items link to:

  • Gap-fill methods — descriptions of each imputation algorithm.
  • App guide — this document.
  • Data process guide — overview of the data processing pipeline.

Frequently asked questions

Q: How do I validate data step by step?

  1. Load a .rds file via Open existing Metamet object (or create one first).
  2. On the Select date range and QA/QC tab, choose your start and end date/time.
  3. Click Retrieve from database.
  4. Work through each variable tab: inspect the plot, impute suspect points as needed, and click Finished checking variable to sign off.
  5. Click Save changes to write the output files.

Q: How do I change the username the app uses?

The username is read automatically from your operating system account (Sys.info()[["user"]]) and is not currently configurable within the app.

Q: I want to download the data

Click the Download processed data tab in the left sidebar. Choose Level 1, Level 2, or CEDA from the drop-down and click Download.

Level 1 and Level 2 outputs are .zip archives containing a data CSV and a QC CSV. CEDA output is a single BADC-CSV file.

Q: How do I create a metamet object from scratch (new site / new year)?

Use the Create new Metamet object wizard (first item in the sidebar). Work through the six steps to load your raw data, enter site information, map variables, set QC ranges, attach ERA5 data, and download the .rds.

Q: The app is slow or has frozen — what should I do?

Click Restart app (visible in the Extracted Data panel after data retrieval) or close the browser tab and re-run metamet::run_shiny().

Q: My question is not listed here, I have noticed a mistake, or I would like a new feature

log a GitHub issue, or email Pete () or Will ().