rainfallqc.checks.gauge_checks

Quality control checks examining suspicious rain gauges.

Gauge checks are defined as QC checks that: “detect abnormalities in summary and descriptive statistics of rain gauges.”

Classes and functions ordered by appearance in IntenseQC framework.

rainfallqc.checks.gauge_checks.check_breakpoints(data, target_gauge_col, p_threshold=0.01)

Use a Pettitt test rainfall data to check for breakpoints.

This is QC6 from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data.

• target_gauge_col (str) – Column with rainfall data.

• p_threshold (float) – Significance level for the test.

Return type:

int

Returns:

:

flagint

1 if breakpoint is detected (p < p_threshold), 0 otherwise

rainfallqc.checks.gauge_checks.check_day_of_week(data, target_gauge_col, p_threshold=0.01)

Perform a two-sided t-test on the distribution of mean rainfall over time slices.

This check performs less well when using less data.

This is QC3 (day of week bias) from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• p_threshold (float) – Significance level for the test (default 0.01)

Return type:

int

Returns:

:

flagint

1 if bias is detected (p < threshold), 0 otherwise

rainfallqc.checks.gauge_checks.check_hour_of_day(data, target_gauge_col, p_threshold=0.01)

Perform a two-sided t-test on the distribution of mean rainfall over time slices.

This check performs less well when using less data.

This is QC4 (hour-of-day bias) from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• p_threshold (float) – Significance level for the test (default 0.01)

Return type:

int

Returns:

:

flagint

1 if bias is detected (p < threshold), 0 otherwise

rainfallqc.checks.gauge_checks.check_intermittency(data, target_gauge_col, no_data_threshold=2, annual_count_threshold=5)

Return years where more than five periods of missing data are bounded by zeros.

TODO: split into multiple sub-functions and write more tests! This is QC5 from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• no_data_threshold (int) – Number of missing values needed to be counted as a no data period (default: 2 (days))

• annual_count_threshold (int) – Number of missing data periods above no_data_threshold per year (default: 5)

Return type:

list

Returns:

:

years_w_intermittency

List of years with intermittency issues.

rainfallqc.checks.gauge_checks.check_min_val_change(data, target_gauge_col, expected_min_val)

Return years when the minimum recorded value changes.

Used to determine whether there are possible changes to the measuring equipment. This is QC7 from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data.

• expected_min_val (float) – Expected value of rainfall i.e. basically the resolution of data.

Return type:

list

Returns:

:

yr_list

List of years with minimum value changes.

rainfallqc.checks.gauge_checks.check_years_where_annual_kth_largest_value_is_zero(data, target_gauge_col, k)

Return list of years where the k-th largest value is 0.

This is QC2 from the IntenseQC framework

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• k (int) – Number of the largest values to take for a given year i.e. k==5 is top 5

Return type:

list

Returns:

:

year_list

List of years where k-largest value is zero.

rainfallqc.checks.gauge_checks.check_years_where_nth_percentile_is_zero(data, target_gauge_col, percentile)

Return years where the n-th percentiles is zero.

This is QC1 from the IntenseQC framework

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• percentile (float) – Between 1 & 100

Return type:

list

Returns:

:

year_list

List of years where n-th percentile is zero.

rainfallqc.checks.gauge_checks.temporal_bias_test(data, target_gauge_col, time_granularity, p_threshold=0.01)

Perform a two-sided t-test on the distribution of mean rainfall over time slices.

This check performs less well when using less data.

This is used in QC3 (day of week bias) and QC4 (hour-of-day bias) from the IntenseQC framework.

Parameters:

• data (DataFrame) – Rainfall data

• target_gauge_col (str) – Column with rainfall data

• time_granularity (str) – Temporal grouping, either ‘weekday’ or ‘hour’

• p_threshold (float) – Significance level for the test

Return type:

int

Returns:

:

flagint

1 if bias is detected (p < threshold), 0 otherwise

Functions

check_breakpoints(data, target_gauge_col[, ...])	Use a Pettitt test rainfall data to check for breakpoints.
check_intermittency(data, target_gauge_col)	Return years where more than five periods of missing data are bounded by zeros.
check_min_val_change(data, target_gauge_col, ...)	Return years when the minimum recorded value changes.
check_day_of_week(data, target_gauge_col[, ...])	Perform a two-sided t-test on the distribution of mean rainfall over time slices.
check_hour_of_day(data, target_gauge_col[, ...])	Perform a two-sided t-test on the distribution of mean rainfall over time slices.
check_years_where_annual_kth_largest_value_is_zero(...)	Return list of years where the k-th largest value is 0.
check_years_where_nth_percentile_is_zero(...)	Return years where the n-th percentiles is zero.