Tests for Quality Control¶

These are the tests available in CoTeDe. Most of the QC recommended guides follow simmilar procedures with small variations, as described below when relevant.

Valid Date¶

Check if there is a valid date and time associated with the measurement.

For Argo, the year also must be later than 1997.

For underwater gliders, the year must be later than 1998.

Valid Position¶

Check if there is a valid position associated with the measurement. It should have a latitude between -90 and 90, and a longitude between -180 and 360.

GTSPP restricts the longitude range to between -180 and 180 degrees.

Location at Sea¶

Check if the position is at sea by using a bathymetry database. If not specified, it is used ETOPO1, a bathymetry with resolution of 1 minute. This test implies approval on Valid Position.

If the position is determined at sea, GTSPP also evaluates the sounding, if present, but that is redundant to the Sounding Test, so this part is neglected in CoTeDe.

Global Range¶

This test evaluates if the measurement is a possible value in the ocean in normal conditions. The thresholds used are extreme values, wide enough to accommodate all possible values and do not discard uncommon, but possible, conditions.

Regional Range¶

This test is equivalent to the Global Range but with a horizontal domain where it is applicable. Good examples are the Mediterranean Sea and the Red Sea, where the feasible range is more restrictive than required for the Global Range.

Regional Range was first introduced in the GTSPP manual 1990.

Note

This test requires the Python package Shapely to read a polygon geometry and evaluate which positions are within that domain.

GTSPP also requires the maximum depth to be less than 5200m in the Mediterranean Sea, and less than 3500m in the Red Sea.

Profile Envelop¶

This test is equivalent to the Global Range but with a vertical domain where it is applicable, i.e. it defines the acceptable range per layer. For instance, GTSPP defines that deeper than 1100m up to 3000m the acceptable temperature range is from -1.5C to 18C.

Profile Envelop was first introduced in the GTSPP manual 1990.

Gradient¶

This test compares

$X_i = \left| V_i - \frac{V_{i+1} + V_{i-1}}{2} \right|$

Gradient Depth Conditional¶

Spike¶

$X_i = \left| V_i - \frac{V_{i+1} + V_{i-1}}{2} \right| - \left| \frac{V_{i+1} - V_{i-1}}{2} \right|$

Spike Depth Conditional¶

Tukey 53H¶

This method to detect spikes is based on the procedure initially proposed by Goring & Nikora 2002 for Acoustic Doppler Velocimeters, and similar to the one adopted by Morello 2011. It takes advantage of the robustness of the median to create a smoother data series, which is then compared with the observation. This difference is normalized by the standard deviation of the observed data series after removing the large–scale variability.

For one individual measurement $x_i$ , where $i$ is the position of the observation, it is evaluated as follows:

$x^{(1)}$ is the median of the five points from $x_{i-2}$ to $x_{i+2}$ ;
$x^{(2)}$ is the median of the three points from $x^{(1)}_{i-1}$ to $x^{(1)}_{i+1}$ ;
$x^{(3)}$ is the defined by the Hanning smoothing filter:
$\frac{1}{4}\left( x^{(2)}_{i-1} +2x^{(2)}_{i} +x^{(2)}_{i+1} \right)$
$x_i$ is a spike if $\frac{|x_i-x^{(3)}|}{\sigma} > k$ , where $\sigma$ is the standard deviation of the lowpass filtered data.

The default behavior in CoTeDe is to flag 4 if the test yields values higher than $k=1.5$ , and flag 1 if it is lower.

Climatology¶

Compares the measurement with the climatology. The difference can be scaled with the standard deviation of the measurements used to create the climatology, thus normalizing by the expected local variance. For instance, the deep ocean is usually more stable, thus the measurements tend to be more similar, resulting in a smaller standard deviation, so small differences are amplified. In comparison, near the surface the high variability and standard deviation minimizes the differences.

I believe that the oldest reference for this test is from GTSPP, but I need to confirm that.

CoTeDe allows to use the standard error to compensate for regions with few measurements. Some regions in the oceans, mostly in deep layers, the available measuremnts are scarce. To distinguish a climatology estimate based on hunderds of measurements versus another with 5 or less measuremnts, the standard error can be used as a tolerance for that estimate. This compensation was first proposed here, in CoTeDe.

$X_i = \frac{V_{it} - <V_t>}{\sigma}$

GTSPP refer to 4 different climatologies to test, which are not implemente in CoTeDe::

Levitus Seasonal
Emery and Dewar
Asheville
Leviturs Monthly

QARTOD climatological test is based on range

World Ocean Atlas (WOA)¶

Use the World Ocean Atlas as reference.

CSIRO Atlas of Regional Seas (CARS)¶

Use CSIRO Atlas of Regional Seas as reference.

Rate of Change¶

For QARTOD, the delta change is normalized by the standard deviation.

Density Inversion¶

This test looks for density inversions in the water column, i.e. higher density above lower density.

Since density inversion is unstable it is not expected to be observed in nature in normal conditions. Note that weak inversions migth be observed near the surface under special conditions of sea surface heat fluxes. Sometimes a small negative threshold is used.

Density Inversion was first introduced in the GTSPP manual 1990.

Constant Cluster¶

For Argo …

This test evaluates a cluster of adjacent measurements that are identical or nearly-identical. This was implemented in CoTeDe as a generalization of the tests: Constant Profile, Stuck Value, Flat Line.

The Constant Profile tests was first introduced in the GTSPP manual 1990.

GTSPP call it Constant Profile test, and requires the full profile to be identical 3 or more measurements.

Deepest Pressure¶

Check for each measurement if the reference pressure (depth) is deeper than the operational limit for that sensor/platform. For instance, the Argo Solo-II operates up to 2000m while the Deep Solo goes up to 6000m. Measurements deeper than that suggest a bad vertical position.

Reference: Argo QC manual 2.9.1

Digit Rollover¶

Every sensor has a limit of bits available to store the sample value, with this limit planned to cover the possible range. A spurious value over the bit range would be recorded as the scale rollover, resulting in a misleading value inside the possible scale. This test identifies extreme jumps on consecutive measurements, that are wider than expected, suggesting a rollover error.

The difference on consecutive measurements must be smaller or equal to the threshold to be approved.

Monotonic Vertical Displacement¶

Evaluates the vertical movement of a profiling platform. If the vertical movement stops, like in a constant depth, all measurements without vertical displacement, except the first one, are flagged bad. If there is an inversion, all measurements in the inverted sequence are flagged bad.

The vertical movement is usually infered by the records of pressure or depth along the time.

The argument from this test might come from the assimetry of every sampling platform. A CTD in a rosette usually samples in the downcast so the CTD itself measures a water parcel less disturbed by the sampling bottles and the rest of the rosette. Spray underwater gliders have the CTD intake in the upper part of the hull, hence the ascencion phase of the dive leads to cleaner samples. Therefore, a change in the “flight” pattern wouldn’t mean a bad measurement per se, but an inconsistent measurement with he previous and following values.

GTSPP evaluates depth.

Argo evaluates pressure and is named ‘pressure increasing’.