Managing national hydrometric data from data to information
- 1. Global Change: Facing Risks and Threats to Water Resources (Proc. of the Sixth World FRIEND Conference, Fez, Morocco, October 2010). IAHS Publ. 340, 2010. 451 Managing national hydrometric data: from data to information HARRY DIXON Centre for Ecology and Hydrology, Maclean Building, Crowmarsh Gifford, Wallingford OX10 8BB, UK harr@ceh.ac.uk Abstract The availability of high quality hydrometric data is fundamental to meeting the continually growing challenge of water management across the globe. River flow data, and environmental monitoring more widely, are acutely important during periods of actual or anticipated change. As such, the beginning of the 21st century has seen a heightened scientific demand for hydrometric information to help further understanding of variability in the global water cycle. This paper presents the experiences of the United Kingdom National River Flow Archive in managing large hydrometric datasets of international significance. The role of a National Hydrometric Information Service in relation to all aspects of the river flow monitoring cycle, from user requirements and network design, to information dissemination and decision making is considered. Current operational practices are outlined, including their applicability to other national hydrometric networks and the implications for end users of river flow information. Key words hydrometric information; data management; monitoring; gauging network; data dissemination; river flow data; UK INTRODUCTION River flow records represent the integrated output of all hydrological processes acting upon a catchment, and underpin informed decision-making in areas such as flood risk estimation, water resources management, hydro-ecological assessment and hydropower generation. Policy decisions across almost every sector of social, economic and environmental development are driven by the analysis of hydrometric information. Its wide ranging utility, coupled with escalating analytical capabilities and information dissemination methods, have seen a rapid growth in the demand for hydrometric data over the first decade of the 21st century. This trend looks set to continue in the near future and, as such, National Hydrometric Information Services (NHIS) and international data sharing initiatives are central to providing access to coherent, high quality river flow information to a wide and growing community of end users. This paper outlines key area of consideration in the operation of National Hydrometric Information Services, detailing operational practices of the UK National River Flow Archive that are transferable to other hydrometric networks around the world. NATIONAL HYDROMETRIC DATA IN THE UK In a global context the UK hydrometric gauging station network is dense – largely a response to both the high drainage density and the marked climatological, geological, land-use and water utilisation diversity of the British Isles (Marsh, 2002). Although considerable spatial variations exist across the country, in global terms, water courses are generally short, shallow, and in places subject to considerable artificial disturbance. A total of around 1500 primary gauging stations are currently operated by three major publically-funded measurement agencies. Partially as a result of a well developed raingauge network, hydrometric monitoring remained sparse in the UK until the 1960s, when increased water resource demands and legislative changes lead to a rapid expansion of the gauging station network. Consequently the national hydrometric database currently holds around 50 000 station-years of flow data, with an average record length of 34 years per station. The UK National River Flow Archive (NRFA) is a publically funded national focal centre for hydrometric information storage, analysis and dissemination. While the archive exists in a separate organisational structure to the major national and regional hydrometric measuring authorities, the NRFA is delivered through close collaboration between data providers, the scientific research community, other data users, and national government. Like many National Hydrometric Information Services around the globe, the NRFA provides both access to river flow data and Copyright © 2010 IAHS Press
- 2. Harry Dixon452 associated information, as well as to knowledge, advice and decision support on a range of national hydrological issues (WMO, 2006). The archive serves a wide user community, incorporating water management professionals, scientific researchers, educational users, national government bodies, and international organisations. In the context of this paper the term hydrometric data is used to primarily refer to river discharge records. THE HYDROMETRIC INFORMATION LIFECYCLE The management of hydrometric data can be considered using a data lifecycle approach (Fig. 1), whereby the different processes of data sensing, manipulation and use are stages in the development and flow of information (Marsh, 2002). The analysis of information, and ultimately the associated decision-making, provide end-points in the hydrological information lifecycle. Marsh (2002) highlighted that in an effective system these later stages of information use should also provide continuous feedback influencing the overall design and structure of the system. National Hydrometric Information Services play a vital role in the development of hydrometric information, both fulfilling functions within the cycle and acting as key feedback loops between data users and those responsible for data collection. The NRFA maintains involvement in all stages of the information lifecycle, interacting with data providers, analysts and policy makers to maximize the utility of the national datasets. This paper will consider the key stages in the lifecycle from monitoring network design and data measurement to information dissemination and reporting. Monitoring Network Design Data Sensing and Recording Data Validation and Archival Data Synthesis and Analysis Information Dissemination Information Usage and Decision Making National Hydrometric Information Service Fig. 1 Hydrometric information lifecycle (after: Marsh, 2002). Monitoring network design and development Globally surface water monitoring networks have been designed and subsequently evolved to meet a range of needs, most commonly water resources assessment and hydrological hazard mitigation. The assessment and development of monitoring networks is often driven by operational data needs for water management (e.g. monitoring of water abstractions, flood warning) and controlled by short- to medium-term economic constraints. Within the UK the national archive advocates the consideration of national and international strategic data needs as key drivers for network development. The maintenance of strategic networks of gauging stations suitable for the identification and interpretation of long-term hydrological change or the impacts of artificial
- 3. Managing national hydrometric data: from data to information 453 catchment influences are key capabilities of a national hydrometric network and should be considered alongside operational needs (Marsh, 2002). Subsets of the UK gauging station network have been identified to meet a number of key national strategic needs. Since its initial designation the network of UK “Benchmark” catchments (Bradford & Marsh, 2003) have assumed particular importance in strengthening national capabilities in assessing climatically-driven changes in river flow patterns. The network of around 130 gauging stations across the UK are designated based on hydrometric performance, the degree of artificial influence to the flow regime, homogeneity of the time series, and length of flow record. The network has formed the data selection basis for studies of hydrological change (e.g. Hannaford & Marsh, 2006, 2008). Internationally, a number of similar sub-networks of national surface water monitoring programmes exist for the detection of climatically-driven hydrological change, e.g. the Canadian Reference Hydrometric Basin Network (Harvey et al., 1999). Other sub-network initiatives can be developed to meet other requirements, such as designation of gauging stations with good hydrometric performance at high flows for use in flood estimation methodologies (e.g. Institute of Hydrology, 1999). While the specific strategic needs will vary internationally, the role of National Hydrometric Information Services in designating and promoting such initiatives is central to ensuring that network evolution incorporates assessment of the long-term strategic benefits of existing, or potential future gauging stations, as well as immediate operational needs. Within the UK, as in many areas of the world where surface water monitoring networks are well established, there is an ongoing need to assess the current utility of the network in order to optimise the information it delivers (Marsh, 2002; Hannaford et al., 2010). Such assessments are often a necessary response to historical reductions in network density and the pressure to maximise the cost–benefit of publically-funded monitoring investment (Mishra & Coulibaly, 2009). Data users are often concerned with the degree to which a gauged catchment has similar characteristics to a wider reference area or identifying those stations suitable for use with regionalisation techniques, which frequently form the basis of methodologies for assessing flow regimes at ungauged sites. To this end, the NRFA has led development of techniques such as the “Representative Catchment Index” (Laizé, 2004) and the “Catchment Utility Index” (Laizé et al., 2008). The latter provides an assessment of a catchment’s value in relation regionalisation and capitalises on procedures developed for the Flood Estimation Handbook (Institute of Hydrology, 1999), a UK industry standard used for flood studies. Recently these catchment assessment techniques have been combined with others measures of catchment utility and gauging station performance to develop and apply a new methodology for optimization of networks for regionalisation (Hannaford et al., 2010). National Hydrometric Information Services are uniquely placed to develop such methodologies and conduct end-user focused assessments of hydrometric networks at national or international scales and ensure scientific needs are considered in the early stages of the information lifecycle. Data sensing and recording Responsibility for operation and maintenance of hydrometric networks is frequently devolved to a regional or sub-regional level. National Hydrometric Information Services are able to ensure consistency in data collection and processing methods across different measuring authorities and promote best practice. In the UK, operational procedures in relation to hydrometry and initial data processing (including station design, flow gauging and rating development) are generally set by the individual measuring authorities, who have primary responsibility for operation and maintenance of surface water monitoring networks. These procedures are developed in line with appropriate national (BSi) and international (e.g. CEN, ISO) standards (Herschy, 2009) and involvement in the development of standards represents one route to influencing best practice. The NRFA acts as a central source of expertise for measuring authorities in regard to all areas of hydrometry, especially those related to the processing, management and quality control of data. National Hydrometric Information Services are constantly required to maintain a balance of knowledge between a national scale overview and regional/sub-region hydrometric awareness. To
- 4. Harry Dixon454 data providers an NHIS can provide feedback from information end-users at national and international levels, while to data analysts it is important that the service support the transfer of detailed local level information regarding the hydrometry and catchment characteristics of specific gauging stations. In the UK the maintenance of local and national knowledge is facilitated through a network of NHIS staff who are given specific responsibility of a geographically-defined subset of the network. These designated regional representatives build up important local network knowledge through involvement in data quality control, the production of user guidance information, and maintenance of the detailed hydrometric metadata on the national database. Regular liaison is maintained between regional measuring authorities and the national archive through a combination of field site visits, written guidance, collaborative projects and reporting, in order to maintain strong working relationships, provide feedback and influence day-to-day working practice. Data validation and archival storage The quality control and long-term archiving of hydrometric data represent a central function of National Hydrometric Information Services. Under an organisational system where the core responsibility for quality control of river flow data rests with dispersed data providers, independent national level appraisal of data can provide significant benefits to the final information product. This is particularly true where the national assessments take a user-focused approach to improving the information content of datasets, placing strong emphasis on maximising the final utility of data, e.g. through efforts to improve data completeness, provide user guidance and promote network stability. With the aim of controlling the process of data transfer between organisations and coordinating data validation and improvements, a Service Level Agreement (SLA) was developed in the UK between the national archive and hydrometric measuring authorities. The agreement, which covers the annual transfer and subsequent process of validation of river flow data, is designed to maintain and improve the utility, continuity and fitness-for-purpose of nationally archived data. Through parallel development with network assessment initiatives the SLA promotes long-term stability in the core national hydrometric network, focusing data validation and dissemination activities on the more strategically valuable gauging stations. The SLA provides a framework to manage the annual data transfer to the NHIS and its subsequent validation (Fig. 2). River flow data submitted by measuring authorities are loaded and validated by NRFA staff using a suite of in-house developed software applications. Un-validated data are kept separate from the main national archive until a series of manual and automated checks have been conducted. This process of data validation is conducted by the experienced NRFA regional representatives as diverse hydrological environments and significant anthropogenic disturbance to flow regimes means the hydrograph appraisal undertaken by personnel familiar with the expected flow patterns of individual rivers is the most effective means of quality control (Marsh, 2002). Visual appraisals use versatile hydrograph plotting and manipulation software to enable comparisons between different near-neighbour or analogue flow measurement sites, assessments of catchment rainfall input hyetographs, calculation of water balances and assessment of time series statistics. Additionally, users are able to consult station metadata records detailing the history of the site and its hydrometric performance, along with catchment characteristic maps and previous quality control logs. Where potential anomalous flows are identified these are queried with data providers under an SLA procedure using auditable enquiry logs. By following a standardised data assessment and improvement procedure such as these, NHIS are able to safe-guard against reduced quality data reaching the national archive. Integral to the SLA are quantitative assessments of the utility of data provided to the NRFA. An objective scoring mechanism assesses the completeness and timeliness of data submissions to the archive and quantity of data where queries confirm problems in the measurement or processing or flows (Hannaford, 2004). This allows monitoring of the performance of data providers, individual gauging station or groups of gauging stations. The development of a reporting structure
- 5. Managing national hydrometric data: from data to information 455 Fig. 2 Schematic representation of NRFA data validation, analysis and dissemination procedures. Fig. 3 UK average Service Level Agreement performance indicators for data completeness (2001–2008). based on these key performance indicators encourages improvements in underperforming parts of the country and focuses attention on data quality improvements. The SLA has been in place in the UK since 2002 and over this period marked improvements have been seen in those areas of data quality assessed by the system. Fig. 3 graphs improvement in the overall national completeness of daily mean flow records (the key river flow dataset archived at a national level in the UK) and percentage of stations providing complete series to the NRFA over the operational period of the SLA. Data completeness is particularly important as a data quality indicator as erroneous or missing data severely impacts on the usability of hydrometric time series. In particular, even short segments of missing data points in gauged daily flow series have considerable implications for the calculation of derived series that are widely used in the analysis of hydrometric data (e.g. regional runoff, 7-day minima). The implications of incomplete data series can be considerable if they occur during flow extremes, impacting on series such as the annual maximum and thus impinge on analysis of changes in the frequency/magnitude of extreme flow. It is important to have a complete flow series wherever possible, and the informed use of appropriate techniques to provide estimated flows (flagged as such) is preferable to missing data points. Original data providers are often best placed to derive realistic estimates of flows during unrecorded periods due to their detailed knowledge of individual station performance and catchment responses. Through the operation of a controlled data provision, validation and archival process and a combined focus by those maintaining monitoring networks and NHIS to maximise data quality
- 6. Harry Dixon456 and completeness the utility of national datasets can be improved. Such systems could easily be transferred to help control hydrometric information lifecycles in other part of the world. However, Marsh (2002) highlights the danger of data quality appraisal systems that operate too mechanistically, concentrating on the separate indices of data quality rather than the overall information delivery. To mitigate against such issues the SLA system in the UK is closely connected with other initiatives, such as the dedicated measuring authority liaison and promotion of best practice mentioned above, which aim to improve boarder data utility. Data synthesis and analysis National Hydrometric Information Services play a key role in the delivery of large scale assessments of hydrological data. This is especially true in the active monitoring of the national hydrological situation or the assessment of conditions at times of widespread flood or drought. Since 1988 the NRFA has, in conjunction with the UK National Groundwater Level Archive, operated the National Hydrological Monitoring Programme (NHMP). The programme provides routine monthly reports and statistics on the current national hydrological situation, including assessments of rainfall, river flow, groundwater and reservoir stocks (NERC, 2010). The reports are aimed at a wide audience, providing a snapshot of the current national situation and its historical context. In addition to monthly situation reporting, the NHMP conducts annual reviews of the hydrological year, providing users with published assessments of the key elements of the hydrological cycle. In the light of increased public environmental awareness the demand for high quality scientific information about extreme hydrological events is increasing. NHIS are able to play a key role in providing reactive analysis of flood or drought episodes, capitalising on long term situation monitoring and analysis to provide scientifically-based assessments in the aftermath of events – for example the assessment of the widespread England and Wales flooding in 2007 (Marsh & Hannaford, 2007). The NHMP services a wide range of UK users providing input to scientific reports and research projects, as well as informing policy decisions, media briefings, and increasing public understanding of the state of the water environment. Globally, concerns about recent hydrological changes are increasing prompting increased scrutiny of hydrological time series. In the UK, river flow data from the national archive, often focusing on the Benchmark strategic network, have been used extensively by the NRFA and others in the assessment of trends over the instrumental record (e.g. Dixon, et al., 2006; Hannaford & Marsh, 2006, 2008). NHIS are often well placed to conduct or inform scientific analysis of river flow change at a national or international level and act as a source of advice on data use and guidance on interpretation of flow patterns. Data dissemination One of the primary functions of a National Hydrometric Information Service is to provide comprehensive, spatially coherent access to information at a scale and resolution appropriate for a wide range of end-users. At global, regional and local scales, efforts to meet complex environmental challenges are demanding ever-increasing access to inter-operable observations of freshwater systems from a wider and more diverse community of users. Meeting these requirements necessitates almost constant adaptation and updating of data management practices by NHIS. For large parts of the 20th century the primary data dissemination route for national hydrometric data was via annual hard-copy publications of data tables (e.g. UK Surface Water Yearbooks produced between 1935 and 1995). However, the last 15 years has seen a shift towards more dynamic web-based data dissemination to meet the requirement for shorter lag-time between observation and data publication and ease of data re-use. A key dissemination route for UK river flow data and associated metadata is via an online web-portal which provides users with dynamic access to a wide range of information to allow selection of gauging stations for analysis and, for a sub-set of stations which have been subjected to a higher level of quality control over the full period-of-record, download flow data
- 7. Managing national hydrometric data: from data to information 457 (http://www.ceh.ac.uk/data/nrfa). This facility is complemented by a dedicated enquiry and data retrieval service (Fig. 2). Considerable value is added to the service if NHIS are able to provide data users with access to additional datasets that can help interpret the baseline hydrometric records. The availability of catchment averaged rainfall series allows users to interpret catchment inputs, while spatial datasets can provide an understanding of the nature of the catchments, e.g. land-use, hydro-geology, topography and drainage networks. A key stewardship role for NHIS is the maintenance and dissemination of user guidance information. Along with basic gauging station discovery metadata (e.g. geographical location, station type, catchment area), contextual information about the station hydrometry and flow record provide the user with vital guidance which should be considered during analysis and interpretation. As data storage capabilities increase and information management systems allow ever more detailed metadata collection, the potential quantity of contextual information available to the end- user has burgeoned. Through careful quality control and application of informed judgement, NHIS are able to produce composite user guidance summaries that draw users’ attention to key information and record caveats (e.g. marked artificial influences on the flow regime, high levels of uncertainty regarding specific flood event accuracy, major changes in hydrometric setup). Web- based dissemination of user-guidance and discovery metadata can usefully be complemented by publication of registers of hydrometric stations (e.g. Marsh & Hannaford, 2008). At national and international levels the application of large scale multi-site analysis techniques of river flow records is growing as computing power and data access improves, such projects are vital to meeting the global water challenges of the 21st century. However, this issue represents a key problem for NHIS in balancing ease of access to data with a promotion of responsible data use (where the data context and monitoring limitations are considered). Data management practices must facilitate large scale data analysis and allow widespread data dissemination while maintaining end-user access to important contextual hydrological information. Technological data dissemination solutions and international metadata standards should allow NHIS to continue to exercise important data stewardship and promote sensible data use. Scientific users can, through NHIS, play a key role in improving hydrometric data, providing feedback highlighting important issues in relation to records, helping establish network requirements and adding to a centralised knowledge base regarding national data. Through embracing feedback from the end-user community the overall information delivery of a system can be improved. CONCLUSIONS The Third World Water Development Report highlighted the importance of reliable hydrological data in meeting the world’s water resource management needs, stressing the current global data inadequacies and the need to resolve current barriers to data sharing and access (Grabs, 2009). Where currently established, National Hydrometric Information Services play a key role in the delivery of hydrometric information to scientists and water practitioners. This paper has discussed the management and delivery of hydrometric information through the use of a data lifecycle approach, in particular emphasising the importance of maintaining feedback routes between data users and those responsible for data sensing and processing. Freshwater management decisions and policies should be based upon quantitative knowledge of the hydrological system. The importance of long term monitoring and central contribution of data providers should be recognised at local, national and international scales. As both the demand for and complexity of hydrometric data increases, the key principles of strategic network design, systematic data quality control, responsible data stewardship and effective information dissemination should be promoted throughout hydrometric observation networks. Hydrometric information systems should be designed such that the utility of data to end-user at all levels is a central consideration. The development of international data sharing initiatives such as the European Water Archive (and other similar FRIEND initiatives) and the WMO Global Runoff Data Centre facilitate
- 8. Harry Dixon458 regional and global use of hydrometric information. NHIS are central to the success of such systems, providing reliable access to, and guidance on, national data contributions. While internationally networks may vary with respect to organisation set-ups, gauging methods, data management and data dissemination policies, useful synergies can be developed in all stages of the hydrometric information lifecycle to improve the overall utility of the information output. National and international hydrometric information services are critical to meeting global water management needs. Observational frameworks should be reviewed at all levels to ensure the fitness for purpose of river flow information available to users. In the light of ever-increasing global water management challenge, the widespread decline of hydrometric network and continued barriers to data sharing, there is a pressing need to improve understanding of the value of hydrological data. REFERENCES Bradford, R. B. & Marsh, T. J. (2003) Defining a network of benchmark catchments for the UK. Proc. Inst. Civil Eng.-Water Marit. Engng 156(2), 109–116. Dixon, H., Lawler, D. M. & Shamseldin, A. Y. (2006) Streamflow trends in western Britain. Geophys. Res. Lett. 33(19), 7. Grabs, W. (2009) Bridging the observational gap. In: The United Nations World Water Development Report 3: Water in a Changing World, 226–236. World Water Assessment Programme, UNESCO Publishing/Earthscan, Paris, France. Hannaford, J. (2004) Development of a strategic data management system for a national hydrological database, The UK National River Flow Archive. In: Proc. 6th Int. Conf. on Hydroinformatics (ed. by Shie-Yui Liong, Kok-Kwang Phoon & Vladan Babovic), (21–24 June 2004). World Scientific Publishing. Hannaford, J. & Marsh, T. J. (2006) An assessment of trends in UK runoff and low flows using a network of undisturbed catchments, Int. J. Climatol. 26(9), 1237–1253. Hannaford, J. & Marsh, T. J. (2008) High-flow and flood trends in a network of undisturbed catchments in the UK. Int. J. Climatol. 28(10), 1325–1338. Hannaford, J., Holmes, M. G. R., Laizé, C. L. R., Marsh, T. J. & Young, A. R. (2010) Optimising hydrometric networks for regionalisation: a new methodology and its application to England and Wales. J. Hydrol. (submitted). Harvey, K. D., Pilon, P. J. & Yuzyk, T. R. (1999) Canada's Reference Hydrometric Basin Network (RHBN). In: Proc. CWRA 51st Annual Conf. (Nova Scotia, June 1999). Herschy, R. W. (2009) Streamflow Measurement. Taylor & Francis, London, UK. Institute of Hydrology (1999) Flood Estimation Handbook, 5 Volumes. Institute of Hydrology, Wallingford, UK. Laizé, C. L. R. (2004) Integration of spatial datasets to support the review of hydrometric networks and the identification of representative catchments. Hydrol. Earth System Sci. 8(6), 1103–1117. Laizé, C. L. R., Marsh, T. J. & Morris, D. G. (2008) Catchment descriptors to optimise hydrometric networks. In: Proc. Institution of Civil Engineers-Water Management 161(3), 117–125. Marsh, T. J. (2002) Capitalising on river flow data to meet changing national needs – a UK perspective. Flow Meas. Instrum. 13(5–6), 291–298. Marsh, T. J. & Hannaford, J. (2007) The Summer 2007 Floods in England and Wales – A Hydrologcial Appraisal. Centre for Ecology & Hydrology, NERC, Wallingford, UK. Marsh, T. J. & Hannaford, J. (eds) (2008) UK Hydrometric Register. Centre for Ecology & Hydrology, NERC, UK. Mishra, A. K. & Coulibaly, P. (2009) Development in hydrometric network design: a review. Rev. Geophys. 47, 24. NERC (2010) Hydrological Summary for the United Kingdom. National Hydrological Monitoring Programme, Monthly Pub., NERC: Centre for Ecology & Hydrology/British Geological Survey. WMO (2006) Guidelines on the Role, Operation and Management of National Hydrological Services. Geneva, Switzerland.