Machine Learning for Data Extraction
Download as PPTX, PDF0 likes102 views
The document discusses a machine learning project by Oak Ridge National Laboratory and the Open University aimed at automating the extraction of study descriptors from toxicology research publications, specifically focusing on OECD test guideline 440. It covers the methodologies employed, including supervised and unsupervised approaches to information extraction, and presents results from experiments using deep learning techniques for pdf extraction. The document highlights the challenges faced, current limitations, and future work needed to improve the accuracy of the extraction process.
Machine Learning for Data Extraction
- 1. ORNL is managed by UT-Battelle, LLC for the US Department of Energy Machine Learning for Data Extraction OECD Workshop on Evaluation of Existing Data on EDCs October 24, 2018, Paris, France Christopher Stahl, stahlcg@ornl.gov Dasha Herrmannova, dasha.herrmannova@open.ac.uk Oak Ridge National Laboratory, US & The Open University, UK
- 2. 22 Machine Learning for Data Extraction Acknowledgements • The following people contributed to the research in this presentation: – Steven Young, ORNL – Robert Patton, ORNL – Jack Wells, ORNL – Mary Wolfe, NTP, NIEHS, NIH – Nicole Kleinstreuer, NICEATM, NTP, NIEHS, NIH
- 3. 33 Machine Learning for Data Extraction Outline • Introduction – Who are we and where are we from? • What is Machine Learning? – Introduction to Machine Learning and Deep Learning • ORNL Deep Learning Experiment – PDF extraction with ORNL DeepPDF • Machine Learning for Data Extraction – Extraction of study descriptors in Toxicology research
- 4. Introduction
- 5. 55 Machine Learning for Data Extraction Introduction • Christopher Stahl – Graduated from Florida Southern College with a Bachelor of Science in 2011 and has been working with the Computational Data Analytics group at Oak Ridge National Laboratory since then. He currently is a Data Analytics Software Engineer who works on multiple projects with a focus on data mining, data analytics, and knowledge discovery.
- 6. 66 Machine Learning for Data Extraction Introduction • Dasha Herrmannova – Research Associate in Scholarly Data Mining at the Open University – Visiting Researcher at @ Oak Ridge National Laboratory since 2016 – Goal: Accelerate scientific discovery, help researchers work more effectively by enabling intelligent access to the content of research papers – My research: text-mining for research evaluation (co-founder of semantometrics.org), analysis of research collaboration and trends, information extraction from scholarly publications
- 7. 77 Machine Learning for Data Extraction Introduction • Oak Ridge National Laboratory – Oak Ridge National Laboratory is the largest US Department of Energy science and energy laboratory, conducting basic and applied research to deliver transformative solutions to compelling problems in energy and security.
- 8. 88 Machine Learning for Data Extraction Introduction • Computational Data Analytics Group – The Computational Data Analytics Research Group at the Oak Ridge National Laboratory conducts innovative basic and applied computer science research on challenges of national interest. Our primary research focus is in the areas of large-scale data analytics and architectures. We conduct innovative basic and applied computer science research on challenges of national interest. Our main research area is analysis of very large sets of data. Our research focus is in the areas of large-scale data analytics and architectures, which includes work machine learning, deep learning, text analytics, graph analytics, visual analytics, and biomedical analytics.
- 9. What is Machine Learning?
- 10. 1010 Machine Learning for Data Extraction What is Machine Learning? • Simply put – “Machine Learning is the science of getting computers to learn and act like humans do, and improve their learning over time in autonomous fashion, by feeding them data and information in the form of observations and real-world interactions.” - https://www.techemergence.com/what-is-machine-learning/
- 11. 1111 Machine Learning for Data Extraction Machine Learning • Two common conceptions: – Fully autonomous robots that ultimately destroy all mankind. – Human assisted machines that make us better, faster, and stronger.
- 12. 1212 Machine Learning for Data Extraction But Why Teach Machines? • Big Problems, Big Data
- 13. 1313 Machine Learning for Data Extraction Deep Learning • Deep Learning research is exploding
- 14. 1414 Machine Learning for Data Extraction What is Deep Learning? • Deep learning is data driven feature extraction supported by hierarchy of neuron layers – Lower layers learn local detail – Higher layers learn global concepts http://www.datarobot.com/blog/a-primer-on-deep-learning/
- 15. 1515 Machine Learning for Data Extraction Simple Deep Learning • Software programs trying to mimic the human brain. • So let's talk more about those cats. • What is a cat? “Features” of a Cat Small and fluffy 2 ears 2 eyes Nose 4 legs Tail
- 16. 1616 Machine Learning for Data Extraction Simple Deep Learning Why? Small and fluffy - ✓ 2 ears - ✓ 2 eyes - X Nose - X 4 legs - X Tail - ✓ Based on our training...not a cat. How does our brain know better?
- 17. 1717 Machine Learning for Data Extraction Simple Deep Learning • Convolutional Neural Network – Instead of defining features (eyes, nose, tail, etc.) let's give the machine the entire image and allow it to scan for features dividing the image into smaller blocks. – The computer looks for things such as, is there a horizontal line, this block is mostly black, does this look fluffy etc. – The computer can find tens to hundreds of such features. These are called neurons.
- 18. 1818 Machine Learning for Data Extraction Simple Deep Learning
- 19. 1919 Machine Learning for Data Extraction Simple Deep Learning • Now we can make another layer with larger blocks. Then another and another, etc. Until we finally have learned what defines a cat. http://www.datarobot.com/blog/a-primer-on-deep-learning/
- 20. 2020 Machine Learning for Data Extraction Not Just Robots, Self Driving Cars, and Cat Videos • Researchers from Sutter Health and Georgia Institute of Technology showed they could predict heart failure as much as nine months before doctors using traditional methods. (https://blogs.nvidia.com/blog/2016/04/11/predict-heart-failure/) • Wearables such as Horus help the blind to be able to “see” by using deep learning to analyze their surroundings and provide feedback to the user. (https://newatlas.com/horus-wearable-blind-assistant/46173/)
- 21. ORNL Deep Learning Experiment DeepPDF https://www.osti.gov/servlets/purl/1460210
- 22. 2222 Machine Learning for Data Extraction ORNL Deep Learning Experiment • What current problem exists, that is easy for human but hard for machine?
- 23. 2323 Machine Learning for Data Extraction Background • Test of Grobid on 100 Sponsor Publications
- 24. 2424 Machine Learning for Data Extraction Background • Deep learning is heavily being applied to image analysis. • Publication files (PDF) look like images. • Can we use Deep learning to improve results???
- 25. 2525 Machine Learning for Data Extraction Data Collection • 50 Randomly selected PDF files from PMC_sample_1943 • 12 Different annotation types • 407 total pages
- 26. 2626 Machine Learning for Data Extraction Experiment • Semantic segmentation • The process of assigning a label to each pixel of an image. • U-Net, A popular network for semantic segmentation tasks. • https://github.com/shreyaspadhy/UNet-Zoo • This network was chosen as it typically provides good performance with relatively few training examples.
- 27. 2727 Machine Learning for Data Extraction Results • The per pixel classification accuracy on the validation set was 94.32%, compared to a baseline of classifying each pixel as “not paragraph” which would provide 79.67 accuracy.
- 28. 2828 Machine Learning for Data Extraction Results
- 29. 2929 Machine Learning for Data Extraction Results
- 30. 3030 Machine Learning for Data Extraction Next Steps • Take results and extract to structured text. This should allow for better results of all our work using PDF’s as an input.
- 31. Machine Learning for Data Extraction https://www.aclweb.org/anthology/W18-5609/
- 32. 3232 Machine Learning for Data Extraction Goal • Goal: automated identification/extraction of study descriptors from research publications – Information pertaining to guideline for rodent uterotrophic bioassays (OECD TG 440) Minimal criteria Female weanling rats Six rats per group Four treatment groups
- 33. 3333 Machine Learning for Data Extraction The Data Being Extracted • OECD Test Guideline 440: Uterotrophic Bioassay in Rodents – The guideline consists of 6 minimal criteria (MC) – All six MC have to be met for a study to be guideline-like (GL) – Below: manually annotated (GL) abstract showing the six MC
- 34. 3434 Machine Learning for Data Extraction OECD TG 440: Minimum criteria (MC) • MC 1: Animal model • MC 2: Group size • MC 3: Route of administration • MC 4: Number of dose groups • MC 5: Dosing interval • MC 6: Necropsy timing Source: Kleinstreuer et al. (2016). A Curated Database of Rodent Uterotrophic Bioactivity.
- 35. 3535 Machine Learning for Data Extraction Dataset • 670 research publications with results for 2,615 uterotrophic bioassays – A curated database of rodent uterotrophic bioactivity. (2015). Kleinstreuer et. al. Environmental health perspectives. – ~120 publications (~18%) contain GL studies (~540 out of 2,615)
- 36. 3636 Machine Learning for Data Extraction Information extraction • Standard approach to document annotation: train a prediction model from labeled data • Requires fine-grained sentence/word level annotations – Obtaining training data can be very costly – Depending on task, data being extracted can vary significantly – What if training data isn’t available? ✓ ✗ VS.
- 37. 3737 Machine Learning for Data Extraction Human learning vs. machine learning Document “pets” It talks about dogs It talks about cats … “pets” document label dog cat buy Which words were in the document?
- 38. 3838 Machine Learning for Data Extraction Human learning vs. machine learning • A human non-expert can correctly annotate criteria given just a description of the target information
- 39. 3939 Machine Learning for Data Extraction Human learning vs. machine learning • Machine Learning (ML) algorithm learns to associate certain linguistic patterns with the criteria – These patterns may be hard to spot for a human • For example an ML algorithm can learn to correctly predict if a document met criteria based short text snippets not mentioning the criteria (such as abstracts)
- 40. 4040 Machine Learning for Data Extraction Our approach • Goal: extract information for the purpose of identifying GL/non-GL documents • Two approaches: 1. Classify, then extract: Can we train a good classifier? If so, extract learned patterns (fully supervised) 2. Extract, then classify: Extract text segments relevant to criteria descriptions (unsupervised), then classify the extracted segments “pets” Why? document label dog cat buy Which words were in the document? Sentence 1: Pre-pubertal (day 18 of life) female mice were randomized to receive placebo, 200 mg/kg CTX, or 120 mg/kg CTX. Similarity to MC1: 0.5478 Sentence 2: The dosages of CTX used were based on previous studies, which demonstrated a significant dose- dependent ovarian toxicity. Similarity to MC1: 0.3887 ✓
- 41. 4141 Machine Learning for Data Extraction Classify, then extract • Train a classifier (e.g. Logistic Regression, Convolutional Neural Network) to distinguish between publications that met/didn’t meet criteria • Extract the linguistic patterns the classifier learns to use to make the decision – Occurrence/lack of certain words in text, importance of words • Pro: theoretically a more accurate method • Con: requires labeled data – what is the minimal amount of data needed to make this approach work reliably?
- 42. 4242 Machine Learning for Data Extraction Extract, then classify • Utilize criteria descriptions/sample sentences/taxonomy (=a “query” into the document) • Extract parts of the document most relevant to the “query” (e.g. parts which are the most similar to the query) • Pro: Does not require labeled data, works on individual documents • Con: What is a good ”query” and a reliable extraction (similarity) method?
- 43. 4343 Machine Learning for Data Extraction OECD TG 440: Minimum criteria (MC) • MC 1: Animal model • MC 2: Group size • MC 3: Route of administration • MC 4: Number of dose groups • MC 5: Dosing interval • MC 6: Necropsy timing Source: Kleinstreuer et al. (2016). A Curated Database of Rodent Uterotrophic Bioactivity.
- 44. 4444 Machine Learning for Data Extraction Example abstract
- 45. 4545 Machine Learning for Data Extraction Results for MC 1: Animal model • Top: supervised (classify, then extract), bottom: unsupervised Correct answer (underlined)
- 46. 4646 Machine Learning for Data Extraction Results for MC 2: Group size • Top: supervised (classify, then extract), bottom: unsupervised
- 47. 4747 Machine Learning for Data Extraction Results for MC 3: Route of administration • Top: supervised (classify, then extract), bottom: unsupervised
- 48. 4848 Machine Learning for Data Extraction Results for MC 4: Number of dose groups • Top: supervised (classify, then extract), bottom: unsupervised
- 49. 4949 Machine Learning for Data Extraction Results for MC 5: Dosing interval • Top: supervised (classify, then extract), bottom: unsupervised
- 50. 5050 Machine Learning for Data Extraction Results for MC 6: Necropsy timing • Top: supervised (classify, then extract), bottom: unsupervised
- 51. 5151 Machine Learning for Data Extraction Unsupervised (similarity-based) approach • Top sentences extracted from full text (correct answer underlined) • Similarity scores: 70.61, 65.31, and 63.69 1. After weaning on pnd 21, the dams were euthanized by CO2 asphyxiation and the juvenile females were individually housed. 2. Six CD(SD) rat dams, each with reconstituted litters of six female pups, were received from Charles River Laboratories (Raleigh, NC, USA) on offspring postnatal day (pnd) 16. 3. This validation study followed OECD TG 440, with six female weanling rats (postnatal day 21) per dose group and six treatment groups.
- 52. 5252 Machine Learning for Data Extraction Current limitations and future work • “Semantic implication” • Tables/figures • Multiple studies • Answer not in text
- 53. 5353 Machine Learning for Data Extraction Current limitations • “Semantic implication” – MC 4: Number of dose group – Minimum of two dose groups, must have positive and negative control
- 54. 5454 Machine Learning for Data Extraction Current limitations • Tables/figures
- 55. 5555 Machine Learning for Data Extraction Current limitations • Multiple studies – We currently don’t have a definitive way of recognizing publications with multiple studies or matching sentences to studies
- 56. 5656 Machine Learning for Data Extraction Current limitations • Document 24096037 (describes 4 studies), top sentence from full text for each MC 1. Animal model: (I) Female Sprague-Dawley rats and C57BL/6 mice, ovarectomized on PND 20 and all within 10% of the average body weight (b.w.), were obtained from ... 2. Group size: (I) An equal number of time-matched Veh animals (n=5) were treated in the same manner. 3. Route of administration: However, o,p-DDT induced Ca2 in the rat, with a clear difference between EE and o,p-DDT treatment. 4. Number of dose groups: (I) For graphing purposes, the relative expression levels were scaled such that the expression level of the time-matched control group was equal to one. 5. Dosing interval: Female Sprague-Dawley rats and C57BL/6 mice, ovarectomized on PND 20 and all within 10% of the average body weight (b.w.), were obtained from ... 6. Necropsy timing: (HI) All animals were sacrificed 24 h after the last treatment (72 h after the initial dose).
- 57. 5757 Machine Learning for Data Extraction Current limitations • Answer not in text – We currently don’t have a definitive way of recognizing if the answer not there
- 58. 5858 Machine Learning for Data Extraction Other improvements • Making extraction more accurate – Applying robust evaluation metrics toward the extracted text – Combining both (supervised and unsupervised) approaches • Labeling extracted text • Using ensembles to improve classification / joint modelling of MC labels (e.g. using Deep Learning)
- 59. 5959 Machine Learning for Data Extraction Conclusions • A hard problem given the lack of annotations • Similarity-based approaches may provide a way forward
- 60. 6060 Machine Learning for Data Extraction Funding • Support for this research was provided by an Interagency Agreement with the National Institute of Environmental Health Sciences (AES 16002-001) and the U.S. Department of Energy at Oak Ridge National Laboratory. • This research was supported in part by an appointment to the Oak Ridge National Laboratory ASTRO Program, sponsored by the U.S. Department of Energy and administered by the Oak Ridge Institute for Science and Education. • This manuscript has been authored by UT-Battelle, LLC and used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory under Contract No. DE- AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan.
- 61. Thank you for listening!