Item 5: Introduction to the Global Spectral Calibration Library
Download as PPTX, PDF0 likes201 views
The document discusses the launch of the GloSolan initiative, aimed at creating a global soil spectral calibration library and estimation service, utilizing mid-infrared spectroscopy. The initiative invites countries to contribute benchmark soil samples for characterization, promoting international collaboration and open access to data for soil property estimation. The library will support the development of calibration algorithms and provide tools for various laboratories while ensuring that data ownership remains with the contributors.
Item 5: Introduction to the Global Spectral Calibration Library
- 1. Item 5: Introduction to the global spectral calibration library Ms. Lucrezia Caon, FAO Mr. Richard Ferguson, United States Department of Agriculture Ms. Fenny van Egmond, ISRIC
- 2. April 2020 : Launch of the GLOSOLAN initiative on spectroscopy In order to open the discussion and start building collaboration with institutes and experts on spectroscopy, two concept notes were prepared. One of these focuses on the establishment of a global soil spectral calibration library and estimation service …
- 3. Objectives 1. Build a globally representative soil spectral calibration library (database) based on soil midinfrared diffuse reflectance (MIR) spectra with accompanying soil property reference data recorded in one gold-standard reference laboratory. 1. Provide a freely available and easy-to-use soil property estimation service based on the global spectral library. 1. Support countries to contribute to the global spectral calibration library and use the soil property estimation service.
- 4. Global spectral calibration library hosted at USDA-NSSC Kellogg Soil Survey Laboratory (KSSL) Main characteristics: • Library developed under the GSP framework - the GSP will host a copy of the library • Partners will work together on the development of the initiative and its architecture • Countries contribute samples of their benchmark soils, which KSSL will characterize for a standard suite of soil properties together with MIR, at no cost, and then add the data into the global calibration library • The GSP and other regional bodies should assist countries in preparing and shipping the samples to the US Original Proposal Global Soil Information System (GLOSIS) The two systems will be linked Global Soil Property Estimation Service Main characteristics: • Based on MIR spectra and soil property data recorded at KSSL • Uses in country measured spectra of new samples according to standard protocol • Partners to work together on the development of the best calibration algorithms and serve them up through use friendly web-based interfaces • The service would provide alternative models including downloadable desktop versions, for countries with limited internet connectivity • All code will be open source and FAIR (findable, accessible, interoperable and re-usable)
- 5. Global spectral calibration library hosted at USDA-NSSC Kellogg Soil Survey Laboratory (KSSL) Original Proposal Global Soil Information System (GLOSIS) Global Soil Property Estimation Service Countries (laboratories, individuals, etc.) Samples of benchmark soils • Characterization of samples for a standard suite of soil properties • Accept volunteers for doing the analysis at KSSL • Data (public available) • Possibility to use the KSSL ref. measurements to benchmark or calibrate their laboratories Assist shipping samples Developmentofalgorithmsand guidelinesformeasurement Partners (int. organizations, projects, etc.) • Can use it as long as they have compatible instrument specifications and sample preparation procedures • Possibility to use their existing soil spectral libraries in conjunction with the estimation service Same as per the countries Contribute to the development and architecture Possibility to better manage their soil data and to use the estimation service Existinglibraries
- 6. The spectral library at the Kellogg Soil Survey Laboratory National Soil Survey Center, Natural Resources Conservation Service, USDA
- 7. Kellogg Soil Survey Laboratory National Soil Survey Center Natural Resources Conservation Service
- 8. Themes The USDA NRCS Kellogg Soil Survey Laboratory maintains a large and growing library of mid- infrared (MIR) spectra of soils Calibrations are being developed to estimate soil properties from MIR spectra Goal: using MIR to estimate forms of soil properties for soil survey, soil health assessment and monitoring, etc.
- 9. The USDA-NRCS Kellogg Soil Survey Laboratory has a large and growing MIR spectral library • Over 80,000 archived samples MIR scanned since 2011, allowing calibrations at useful scales. Data collection modeled after protocols by Shepherd (CGIAR) Kellogg Soil Survey has > 400,000 samples for calibrating at various scales.
- 10. Blue dots ( )are sites already scanned into the MIR library (2020) Current MIR coverage map
- 11. MIR Spectrometers used by NRCS Single sample
- 12. 1. Pick a property (e.g., Organic Carbon) and region (e.g. Kansas) 2. Query for already-measured samples stored in the archive 3. Collect MIR spectra on the samples 4. Calibrate by modeling the MIR spectra to the measured data 3.2 2.1 12.0 1.2 3.5 1.4 2.8 3.4 8.0 2.3 … … Org C, % Soil TC OC IC CEC Clay pH 1500 kPa W … etc. soil property predictions Building a calibration – current approach at NRCS
- 13. At NRCS, the first Pilot project took place in the USA Central Plains. Blue diamonds ( ) represent sites of samples used to calibrate. Calibration performances were tested with field samples from multiple scattered sites (✔ ). N 417 miles, 671 km
- 14. Alternative technology used by NRCS field offices • Mill • Retsch MM200 mill produces fine grinds that are spectrally similar to grinds prepared using the KSSL production mill. • Spectrometer • A Bruker Alpha spectrometer is sufficiently compatible with the Vertex 70/HTS-XT used at the KSSL to build the MIR spectral library.
- 15. Example calibration performances* at Kansas field office with true test samples (not used to calibrate), predicting: a. total carbon b. organic carbon, c. CEC d. CaCO3 e. clay f. 15 bar water RMSE informs how far predictions (y- axis) are from measured data (x-axis). R2 informs about correlation between predicted and measured data. Good performance statistics make calibrations useful for soil classification, mapping, interpretation, and assessment. *Seybold, C., et al. 2019. Application of Mid-infrared Spectroscopy in Soil Survey. Soil Science Society of America Journal, 83, pp 1746–1759.
- 17. For the GLOSOLAN MIR spectroscopy initiative, in addition to mid-infrared (MIR) spectra, soil properties the KSSL will measure under this initiative are listed below. Except as noted with an asterisk (*), method codes are from “Kellogg Soil Survey Laboratory Methods Manual, SSIR-42, v. 5, USDA-NRCS.” (https://www.nrcs.usda.gov/wps/portal/nrcs/detail/soils/research/guide/?cid=nrcs142p2_054247) Soil property Method Soil property Method Total carbon, nitrogen, and sulfur method 4H2a1-3a1 Cation exchange capacity, pH 7 method 4B1a1a1a1a-b1 Inorganic carbon (if appropriate) method 4E1a1a1a1-2 Ammonium acetate (pH7) exchangeable calcium, magnesium, potassium, sodium method 4B1a1c1-4a-b1* Organic carbon (calculated from total carbon and inorganic carbon) methods 4H2a + 4E1a1a1a1-2 1500 kPa water holding capacity method 3C2a1a-b Gypsum method 4E2b1a1a1-2 Dithionite-citrate extractable iron, aluminum method 4G1b1-4a-b1* pH: 1 : 1 water method 4C1a2a1a-b1 Ammonium oxalate extractable iron, aluminum method G2a1a1-5a-b1 pH: 1 : 2 0.01-M calcium chloride method 4C1a2a2a-b1 Clay, silt, sand method 3A1a1a pH: 1 : 1 1-N potassium chloride method C1a2a3a-b1 Exchangeable aluminum method 4B3b1a1-b1* pH: 1 : 50 1-N sodium fluoride (if appropriate) method 4C1a1a1a-b1 Mehlich III phosphorus method 4D6a1a-b1 Electrical conductivity method 4F1a1a1a1 Olsen phosphorous (if appropriate) method 4D5a1a-b1 Sodium adsorption ratio and exchangeable sodium percentage (derived quantities) methods 4F3b + 4F3a2 *to-be-published in v. 6 of SSIR-42; write-up available on request to: christopher.lee@usda.gov
- 18. Future 1. Continued development of the MIR library 2. Testing calibrations at various scales. 3. Testing alternative calibration approaches: machine learning, etc. 4. Applications development 5. Improving calibration transfer among different instruments.
- 19. Data Sharing and Ownership • The sample submitter must accept that submitted and measured data and metadata become free public information according to US law (see https://www.congress.gov/bill/115th-congress/house-bill/1770) • USDA-NRCS cannot omit the data or metadata from public view. Without exception, the results of all analyses conducted by the KSSL will be distributed without copyright restriction. The data become part of the public domain. Pros: You will get your data. There is no fee to sample submitters for USDA-NRCS KSSL analytical services or data under this project (currently limited to 300 samples per country, doing the spectral and conventional measurements listed in the sample submission guidelines) Also…You will get everyone else’s data AND Everyone will get your data.
- 20. Data publication and citation • How data will be cited in publications? A simple acknowledgement is standard practice, though not required by USDA-NRCS; e.g. “Data are from the USDA National Soil Survey Center – Kellogg Soil Survey Laboratory” • Can those that sent samples to the KSSL be acknowledged somehow? USDA will furnish properly completed metadata (e.g. origin of samples, submitting entity, etc.), but USDA does not require an author to cite the KSSL or acknowledge any contributor.
- 21. Data Review • KSSL customers are asked to review their data before it is made public • During the review, if a customer has questions about data quality, sample re-analyses may be done • Upon data certification, data are made public ⮚ Measured data available on line https://ncsslabdatamart.sc.egov.usda.gov/ ⮚ Currently, spectral data are served via portable memory drive or via cloudvault transfer Web based spectral data delivery in development • Customer data review period: what’s a reasonable consensus among stakeholders? (To be discussed in Item 6)
- 22. Sample submission guidelines – the fine print!
- 23. Revised proposal on the establishment of the Global Spectral Calibration Library
- 24. While organizing the meeting, we asked you all to tell us your concerns, ask questions and to raise any point that would need to be discussed in order to move forward implementing the initiative All your concerns, questions and points for discussion were compiled in a document and preliminary addressed by the working group on soil spectroscopy
- 25. In short, concerns were raised on the following: - Copyright and data policy – Link to data publishing and acknowledgment of samples and data providers - Possibility to work with different spectral ranges (not only MIR) – Link to the inclusion of existing libraries in the global one - Downscaling of activities – no use of a centralized system - Subdivision of tasks – Link to the development and ownership of tools developed under this initiative
- 26. Global spectral calibration library established as a federated system Main characteristics: • Data consultable at a central portal at FAO • Partners will work together on the development of the initiative and its architecture • The ownership of the data remains with the data provider • Possibility to include spectra other than MIR Areas of concern: • Calibration of data and instruments Revised Proposal Global Soil Information System (GLOSIS) The two systems will be linked Global Soil Property Estimation Service Main characteristics: • Based on the global spectral calibration library • Uses in country measured spectra of new samples according to standard protocol • Partners to work together on the development of the best calibration algorithms and serve them up through use friendly web-based interfaces • The service would provide alternative models including downloadable desktop versions, for countries with limited internet connectivity • All code will be open source and FAIR (findable, accessible, interoperable and re-usable)
- 27. Global spectral calibration library established as a federated system Revised Proposal Global Soil Information System (GLOSIS) Global Soil Property Estimation Service Countries (laboratories, individuals, etc.) Data Samples of benchmark soils Data (public available). Countries to decide when to make the data available to the library Developmentofalgorithmsand guidelinesformeasurement Partners & Regional Champion lab/institutes • Can use it as long as they have compatible instrument specifications and sample preparation procedures • Possibility to use their existing soil spectral libraries in conjunction with the estimation service Same as per the countries • Contribute to the development and architecture • Data Existinglibraries
- 28. Revised Proposal STAGE 1 Establishment of the MIR library and calibration service based on KSSL instruments (Estimated duration: 2 years) STAGE 2 Including other spectrometers and spectral data in the library and establishment of the necessary calibration
- 29. Revised Proposal STAGE 1 • We focus on MIR • No need to have the federated system in place Main activities: - Labs that accept USDA open data policy can send subsets of their samples to the US - Labs that do not accept USDA open data policy can work on their own or with their regional champion laboratory/institute and contribute to GLOSOLAN guidelines, protocols, etc. - GLOSOLAN (GSP/FAO and partners) will start building the federated system - GLOSOLAN (GSP/FAO and partners) will start working on calibration and calibration transfer
- 30. Revised Proposal STAGE 2 • Work on different spectra (MIR, NIR, VNIR, etc.) • Federated system in place Main activities: - Labs to continue sending samples to the KSSL or to their regional champion laboratory/institute or to analyse samples by their own and build their own spectral libraries - Access to different spectral libraries through GLOSIS - GLOSOLAN (GSP/FAO and partners) to keep on working on calibration and calibration transfer - Sample providers can be acknowledged as following in the library (for publishing purposes) # … Sample provided by Sample analysed by Name, last name, institution Name, last name, institution
- 31. How can laboratories that are not using spectroscopy or with little experience in the subject collaborate in this initiative?
- 32. Type of lab How they can contribute What they get back (added value) Notes Laboratories using spectroscopy eq. already that also do wet chemistry • They can contribute with their own spectral library and a subset of their samples • They can scan samples for others in the region • Contribute to the development of standards and models • They can get more spectra for improving their calibration • They can get models for building or strengthening their calibration • Common publications • Free alignment of instruments on spectroscopy and wet chemistry (PT costs to be covered by the initiative) Attention: the equipment and range of spectrum should be compatible with the guidelines (guidelines to be prepared) Laboratories that are using spectroscopy eq. but DO NOT DO wet chemistry • They can contribute with their own spectral library • They can scan samples for others in the region • Contribute to the development of standards and models • Use the calibration service • Common publications • Free alignment of instruments on spectroscopy Attention: the equipment and range of spectrum should be compatible with the guidelines (guidelines to be prepared) Laboratories that just got to use spectroscopy eq. • Send soil samples to the KSSL • They can scan samples for other in the region to start building their own spectral library that can contribute to the global one later on • Contribute to the development of standards and models • Use the calibration service • Comparison of the results obtained by KSSL with their own • Capacity building • Common publications • Free alignment of instruments on spectroscopy and wet chemistry (PT costs to be covered by the initiative) Attention: the equipment and range of spectrum should be compatible with the guidelines (guidelines to be prepared) Laboratories that do not have spectroscopy eq. • Send soil samples to the KSSL • Contribute to the development of standards and models • Get their samples scanned by the KSSL or another laboratory (see above) • Use of the calibration service • Common publications • Guidance on how to implement the spectroscopy analysis (they can start approaching the technology)
- 33. The Global Soil Information System (GLOSIS) and Soil Spectral Data •We propose to incorporate soil spectral data in GLOSIS, the Global Soil Information System, which is currently being built in Pillar 4 of the Global Soil Partnership. GLOSIS will be a federated system. Eg. KSSL node Participating lab node
- 34. The Global Soil Information System (GLOSIS) and Soil Spectral Data ● Data owners will keep their own database (and therefore control) but will be able to link to a network of databases. ● The network will provide tools and training and thereby help improve the data owner’s national, local, or lab soil information system. ● GLOSIS will adhere to GLOSOLAN results and common lab practices for reference data. ● The system will facilitate quality control and a standardised data structure that enables plug-ins for high quality spectral predictions and for national reporting. ● Store own spectral libraries and or data locally in a structured and standardised way incl metadata ● Use the estimation service through the network
- 35. The Global Soil Information System (GLOSIS) and Soil Spectral Data •We propose to incorporate soil spectral data in GLOSIS, the Global Soil Information System, which is currently being built in Pillar 4 of the Global Soil Partnership. GLOSIS will be a federated system. Eg. KSSL node Participating lab node % clay % clay % clay
- 36. Other questions related to the establishment of spectral calibration libraries asked before the meeting
- 37. 1. Is it economically worth for a laboratory to calibrate different soil types? Where to get the money to pay for it? If a laboratory works on its own yes, it makes sense to work on different soil types. However, if labs work under a network like GLOSOLAN and contribute with different soil types to the library and calibration service then they do not need to analyse all soil types. Thus, they can save money. However, they still need to validate the prediction. A laboratory/country does not need to undertake specific sampling for contributing to the library, they can send representative samples they have available already only. 2. If everybody contributes to the submission of samples, can the calibration work be done locally? If participants feel they need help with that then other members of the group can help of course. Yes but there should be a global calibration coordinated by the working group. Global calibration work should not limit anyone to work on calibration at the local/regional level. At the local/regional levels there should be an harmonized procedure to work on calibration that includes capacity building as needed For the calibration service we can have an offline service that can be downloaded and ran locally.
- 38. Time for questions Please ask for clarifications only No criticisms, no better proposals...we will get there in a few minutes
- 39. Thanks for your attention The pictures in the covers and background of this presentation are a courtesy of Dr. Fenny van Egmond, ISCRI