4.4 Confidence in Report Card findings

The degree of certainty in the Report Card results depends on two conceptual questions: whether good indicators were chosen and how well the data presented for each indicator accurately reflect the real status or trend in the metric(s).

The first of these questions pertains to the indicators themselves and how well they address the objectives or attributes they are meant to represent. Certainty about the indicators depends on four main factors:

  • Importance.  the degree to which a linkage (functional relationship) controls the outcome relative to other drivers and linkages affecting that same outcome,
  • Understanding.  the degree to which the performance indicator can be predicted from the defined linkage (functional relationship) and its driver(s),
  • Rigor.  the degree to which the scientific evidence supporting our understanding of a cause-effect relationship (linkage) is contested or confounded by other information, and
  • Feasibility.  the degree to which input data necessary to calculate the proposed performance measure can be delivered in a timely fashion (without external bottlenecks) and the amount of effort (relative to other possible indicators) needed to implement the cause-effect linkage in a computer model.

Where possible, confidence findings for each indicator are mentioned in the corresponding sections as they form an important component of overall confidence in the Report Card.

The second question pertains to statistical confidence in the data presented for each indicator. The available data may contain a variety of sources of uncertainty including:

  • Measurement error. Random or systematic errors introduced during the measurement process, sample handling, recording, sample preparation, sample analysis, data reduction, transmission and storage (USEPA 2006; Thompson 2002)
  • Uncertain/inappropriate interpretation of sampling frame. Errors in inference resulting from opportunistically mining the available data without knowledge of the sampling frame[1]. For example, macro-invertebrate data may have been collected by several different studies with different objectives and target populations (e.g. they could have focused on different stream orders). Without this knowledge, we must make assumptions about the probability of selecting each site and the appropriate weighting of the observation.
  • Sampling error. The error resulting from only examining a portion of the total population (Cochran 1977; Lohr 1999; Thompson 2002), if a census of the population is taken (e.g., school lunch enrolment) then there is no sampling error.
  • Process error. Actual variability between spatial or temporal units in the population. This source of variability exists even if a census is taken with no measurement error. This is often referred to as natural variability.

Any of the above sources of uncertainty affects confidence in the estimates of status and reduces the ability to detect trends over time. For some indicators quantification of different sources of uncertainty in the data may be possible, but in many cases there are limitations to providing a qualitative description of the likely sources of error and associated magnitude.

For each indicator, the best available data were aggregated to produce an estimate for each subwatershed. The 95% confidence interval for the metric statistics are presented, along with the minimum, maximum, and number of observations (n). Finally, when possible, the estimates and associated confidence intervals were transformed to a 0-100 scale (as described in section 4.2).