Elastic as a Fundamental Core to Pfizer’s Scientific Data Cloud
Download as PPTX, PDF1 like39,500 views
Pfizer's Scientific Data Cloud (SDC) aims to enhance research and development by leveraging advanced data analysis and predictive modeling, facilitating faster decision-making while meeting regulatory requirements. It utilizes an elastic core for efficient data management, including metadata and audit logs, which support the analytics layer. The document also outlines design principles and an upgrade plan for their data processing infrastructure.
![7
Configuration of Search Interface
PUT sdc-config/masterconfig/invivo
{"metadata": [{
"metadataName": "experimentID",
"caption": "Experiment ID",
"searchable": true,
"filterable": true,
"filterType": "CheckBoxList"
}],
"resultsPage": [{
"metadataName": "experimentID"
}],
"apps": [{
" link": "https://sdctickets.pfizer.com",
"display": "Submit Ticket"
}]
}](https://image.slidesharecdn.com/boston-kimberly-jones-pfizer-fundamental-core-pfizers-sdc-191004011322/85/Elastic-as-a-Fundamental-Core-to-Pfizer-s-Scientific-Data-Cloud-7-320.jpg)
Elastic as a Fundamental Core to Pfizer’s Scientific Data Cloud
- 1. 1 Jim Silva – Director Kimberly Butler – Lead Data Engineer 3 October 2019, Pfizer, Inc. Elastic as a Fundamental Core to Pfizer’s Scientific Data Cloud
- 2. 2 Our Purpose Our Company $53.6BILLION in revenue in 2018 40MANUFACTURING sites worldwide MORE THAN 125COUNTRIES in which Pfizer sells products 8PRODUCTS with sales greater than $1 billion in 2018 MORE THAN 180NEW R&D COLLABORATIONS in 2018 90,000 COLLEAGUES around the world
- 3. 3 Worldwide Research and Development Research and Development is at the heart of fulfilling Pfizer’s purpose as we work to translate advanced science and technologies into therapies that matter for patients in need. We focus our resources in disease areas with the best chance of scientific and commercial success: Oncology Internal Medicine Inflammation & Immunology Vaccines Rare Disease
- 4. 4 Introduction to the Scientific Data Cloud (SDC) Goals • Accelerate timelines through exploratory analysis and predictive modelling • Modeling helped to accelerate a previous project discovery and development project. • Physical experiments can be avoided through data reuse enabling faster decisions • Satisfy regulatory requirements for secure data retention • Regulators expect immediate response to data requests and demonstration of audit trail
- 5. 5 SDC’s Elastic Core • File Metadata • Line-Specific Metadata • Line-Specific Configuration • Search • Usage Reports • Audit Reports • Files and Metadata to Populate Analytics Layer• File Download Logs • Agent Downtime Logs • Search Logs • S3 Logs Config Search Amazon Redshift Amazon S3
- 6. 6 Search Interface App Menu File Details and File Download 2 Filter on Metadata 1 3
- 7. 7 Configuration of Search Interface PUT sdc-config/masterconfig/invivo {"metadata": [{ "metadataName": "experimentID", "caption": "Experiment ID", "searchable": true, "filterable": true, "filterType": "CheckBoxList" }], "resultsPage": [{ "metadataName": "experimentID" }], "apps": [{ " link": "https://sdctickets.pfizer.com", "display": "Submit Ticket" }] }
- 8. 8 Design Principles "dynamic": "strict", "analysis": { "filter": { "id_word_delimiter": { "catenate_all": "false", "preserve_original": "true", "split_on_numerics": "false", "catenate_numbers": "false", "type": "word_delimiter_graph" } } }
- 9. 9 Audit & Usage Reports SDC S3 Bucket Audit Reports Custom Plugin Amazon CloudWatch Pfizer Logging S3 Bucket File Activity Logs Tomcat Webserver BEATS KIBANALOGSTASH ELASTICSEARCH Usage Reports Native Kibana File Download Logs Search Logs +
- 10. 10 Elastic Upgrade Remediation Plan Upgrade from ELK version 5.3 to 6.4 1 2 3 4 5 Build ELK 6.4 Sandbox Rebuild Indices Run Queries Install Custom Kibana Plugin and Import Visualizations Point Applications to Sandbox