Appendix B: Indicator Selection Criteria

Availability of high-quality data

One of the main obstacles many face when selecting indicators is the lack of available data. Frequently the data for an indicator that may be important are not available. Alternatively, the data might only be available for random points in time or for limited geographical areas. The data might have been collected for one purpose in a particular way that served the original purpose, but for your purposes, it may be inadequate. If new data are needed, the feasibility of collecting them might be limited by the amount of effort required to accurately make the measurement (e.g., actual salmon escapement). Alternate indicators may be considered that have significantly lower cost (e.g., remote-sensing based habitat assessment). For certain indicators, it may be very cost-effective to collect the required metrics (e.g., habitat assessment for a species of concern), but the indicator may not represent the process of concern compared to more expensive indicators (e.g., actual population trends in the species of concern).

Data collection and analysis costs (further described as a separate criterion below) have to be evaluated in relation to the potential cost and societal implications of a proposed action or inaction, i.e., the greater the expected tradeoffs between societal goals, the greater the need for certainty in the environmental outcome. When choosing indicators, it is essential to carefully consider the current availability of data for the indicator, as well as how much data will be available in the future from our own collection and from the efforts of others. The availability of metadata is one criterion for selection of particular data for corresponding indicators. Finally, indicators will be useful and useable in the long-run if there is a process for updating the corresponding database, metadata, and data collection & QA/QC procedures.

Data affordability

One factor to consider in evaluating indicators is the costs associated with collecting and analyzing data. One consideration in evaluation the costs and benefits is the usefulness of the information for evaluation of management and ecosystem condition. Indicators that are cost-effective, while accurately representing ecosystem characteristics are preferable. The primary guide is that the amount of data required to adequately report on condition and change in condition can be and are being collected with the resources available. The data should also be collected in a standardized way for which there are QA/QC procedures described. For critical indicators (those reflecting important system conditions for which there is no viable alternative), more resources may need to be made available if they are currently inadequate.

System representation

Another factor to consider in indicator selection is how well the indicator reflects the issue for which it was selected. Frequently, certain indicators are widely recognized to be a useful measure for an issue. Selecting these indicators is usually a ‘safe bet’. For example, percent riparian canopy cover is considered a good indicator of riparian conditions because it has been extensively studied and shown to have a good relationship with stream temperature and the detection of changes can be made easily. Selecting indicators that have been carefully evaluated for their scientific validity means they usually have wider acceptance than those that haven’t been studied very much, and they are more likely to allow you to make confident inferences about system condition.

Indicators that are representative of large aspects of system condition and trends are preferable for those that have narrower utility, all else being equal.

Sometimes the condition is itself an important ecosystem driver. For example, surface water temperature is an important ecological variable for understanding the condition of aquatic ecosystems. It is also the target of management actions to benefit these ecosystems, which is another criterion described below. Indicators that can provide important information at both broad and fine spatial scales are likely to be more useful as they can help inform both strategic and site-specific decisions.

Ability to detect change over time

The ability to report on trends over time is a key function of an indicator. The availability of a data set collected over a period of many years is ideal. Indicators that respond relatively quickly to management intervention and can effectively be used to measure change over time may be preferable to those that require data over long periods of time to observe changes due to management actions. This is especially useful in reference to short-term grants and contracts, or short-term program evaluation, which require performance measures to demonstrate the success or failure of the project. If possible, select indicators whose range of natural variation can be quantified and that permit change detection over short periods of time (2-3 years). At the same time, recognize that many of the processes that we try to improve with restoration programs take decades or longer to change or recover (e.g., salmon population recovery). Indicators for these projects and programs should be stable over these longer timeframes (i.e., decades).

Independence of indicators from one another

Independence refers to how related indicators are to each other. Road density and %impervious surface are related indicators because roads are often impervious. Indicators that are relatively independent are preferable (e.g., rate of ground water use for irrigation and migration barriers), while recognizing that some critical indicators are related and somewhat dependent on each other (e.g., surface water temperature, flow, stream shading, hydraulic connectivity to groundwater, salmon rearing habitat suitability). The concern about independence is important for designing efficient indicator systems, but is secondary to choosing easily-measured and representative indicators. You may choose related indicators, but you would be constrained in your attempts to use them together to explain condition of a system. For example, if (a) surface water temperature, (b) flow, (c) stream shading, (d) amount of groundwater withdrawal, and (e) salmon rearing habitat were indicators of success for a restoration program, then you could not report changes in these indicators without acknowledging that (a) depends on (b), (c), and (d); (e) depends on (a), (b), (c), and possible indirectly on (d) through (b); and (c) may depend on (b) and (d). If restoration of riparian shade (c) was a goal in order to benefit salmon rearing (e), then the inter-dependence of some of the other parameters would need to be acknowledged and potentially controlled-for in order to measure the true effect of increased riparian shade on salmon rearing.

Supports management decisions and actions

Measuring conditions in the environment and in communities can inform policy development and social/fiscal investments. Indicators should be informative in evaluating environmental/social/economic conditions, as well as the influences on these conditions. Another useful characteristics of indicators is that they can be used to evaluate the effects or effectiveness of management actions.  be it a state or federal agency or the goals and objectives of a watershed council. Whatever the business of the organization is, indicators should provide information that can be used to assess the effectiveness of the work and efforts of the group. In the past, activities were seen as a measure of the effectiveness of an organization. The number of grants awarded, the number of pamphlets distributed, or similar “bean counting” has been used extensively to evaluate an organization’s productivity. Environmental performance measures, on the other hand, look at the environmental and social outcomes of these activities to determine an organization’s effectiveness. This is the reason it is so important to select indicators that are closely linked to management actions and decisions and that can be reported and understood in public arenas

The point of most indicators is to inform a wide audience about conditions in the environment and communities. Indicators should be science-based and easily understood by various kinds of decision-makers (e.g., scientists, public, elected officials). They should be equally presentable in summary form in newspapers and on web sites. Finally, indicators should be based upon reportable technical & scientific information and links easily made between summary presentations and the source data and knowledge.

Citations

Shilling, F.M., S. Sommarstrom, R. Kattelmann, B. Washburn, J. Florsheim, and R. Henly. California Watershed Assessment Manual, Volume II (2007). Prepared for the California Resources Agency and CALFED.

Stalberg, H.C., Lauzier, R.B., MacIsaac, E.A., Porter, M., and Murray, C. 2009. Canada’s policy for conservation of wild pacific salmon: Stream, lake, and estuarine habitat indicators. Can. Manuscr. Fish. Aquat. Sci. 2859: xiii + 135p.