Efficiently Building Machine Learning Models for Predictive Maintenance in the Oil & Gas Industry with Databricks
0 likes636 views
The document discusses the development of machine learning models for predictive maintenance in the oil and gas industry using Databricks, detailing the processes of data collection, cleaning, feature engineering, model training, and performance monitoring. Halliburton aims to improve operational efficiency and reduce costs by leveraging a vast array of historical and real-time data. The document highlights the integration of various digital processes and the use of MLflow for model management to enhance data quality and streamline operations.
Efficiently Building Machine Learning Models for Predictive Maintenance in the Oil & Gas Industry with Databricks
- 1. Efficiently Building Machine Learning Models for Predictive Maintenance in Oil & Gas Industry with Databricks Daili Zhang, Varun Tyagi Data Scientists Halliburton
- 2. Introduction ▪ Halliburton Digital Solution (HDS) ▪ Support and Consolidate Digital Transformation across all PSLs ▪ Provide common platforms/architectures from data warehouse/data governance, analytics development, to BI reporting ▪ Streamline and consolidate various digital processes across all PSLs ▪ Provide and build a strong talent pipeline for software/digital development ▪ Data Science Team ▪ Develop analytics/ML models to ▪ Improve operational efficiency ▪ Increase productive uptime ▪ Reduce operational cost ▪ Provide insights at the right time to the right people to help make business-level decisions
- 3. Analytics Life Cycle in Halliburton Rig Edge SQL Files Raw Data Ingestion & Integration Data Cleaning & Aggregation & Transformation Feature Engineering Model Training & Testing Model Selection Model Deployment Results Visualization Model Performance Monitoring Model Management ML model training & testing takes less than 5% time
- 4. What Data Do We Have? ▪ Operational Data ▪ Historical data (ADI (proprietary format), Parquet, csv) ▪ One Product Service Line (PSL): 500,000+ ADI files (3GB per file in Parquet format) for fracturing jobs collected over 12+ years (1,500TB+ data) ▪ Real-time data (edge device, growing significantly over time) ▪ Hardware Configuration/Maintenance/Event Data ▪ SAP (for example, 5M maintenance orders in less than 2.5 years) ▪ SQL database ▪ Files ▪ External Data ▪ Weather data ▪ Geological & geophysical data No lack of data, but lack of data with QUALITY
- 5. Predictive Maintenance Project (example) ▪ Objective ▪ Reduce annual maintenance cost by 10% through field operation optimization based on avoiding failure modes identified by big data analytics for transmissions
- 6. Data Cleaning & Aggregation ▪ Marry the operational data and the configuration/maintenance data in a consistent way ▪ Different sample frequency (from 1hz to 1000Hz) ▪ Free text input in maintenance records ▪ Mixed equipment identification ▪ Data discontinuity ▪ Missing data ▪ Wrong data ▪ Use Databricks cluster and run time ▪ Leverage Delta Lake ▪ Use pandas_udf functions to gain 10-100x speed
- 7. Feature Engineering (example) Select High Load Windows with Continuous Data • Load Threshold • Window Size • # of Windows Welch Fourier Transform for Each Window • # of points for each section Create Features from Windowed Data • Lag Window Size for Correlation • # of peaks to select in frequency domain • Etc. Combine Features and Cleaning • Classification/Regression • Time window to prediction failure • Balance Data or Not
- 8. Model Training/Testing/Selection (example) ▪ Explored various methods ▪ Spark ML ▪ Deep Learning ▪ Azure AutoML ▪ XGBoost ▪ Sklearn ▪ Evaluated the models with various metrics ▪ Recall rate ▪ F1 score ▪ Accuracy ▪ Business impact with dollar value
- 9. Model Deployment and Visualization ▪ Modularize the whole process from extracting data to model prediction to results visualization into different notebooks ▪ Use the Notebooks workflows to synchronize different notebooks runs ▪ Use user-set widget parameters to pass the parameters used in different notebooks ▪ Schedule the job to run on daily basis through the notebook UI ▪ The results are visualized in PowerBI for end users
- 10. Model Performance Monitoring ▪ Store the prediction into blob storage continuously ▪ Store the actual results into blob storage continuously ▪ View the discrepancy along the time in PowerBI ▪ Alerts are set to send emails to users based on specified thresholds ▪ Investigate the model drifting and re-train the models
- 11. Model Management ▪ Prior using MLflow, manually wrote the model specific information into a .csv file and stored the models into a blob storage with certain name conventions ▪ MLflow greatly simplifies the process with consistency and quality