From DBA to DE: Becoming a Data Engineer
0 likes72 views
This document outlines the transition from a Database Administrator (DBA) to a Data Engineer (DE), highlighting the essential skills and knowledge required for this career shift, such as statistics, Python, and data cleaning. It emphasizes the importance of understanding data governance, model accuracy, and the human factors contributing to project success or failure, while offering insights into machine learning tools and practices. Additionally, it suggests continual learning and professional engagement through certifications and industry events to maintain relevance in the evolving field of data engineering.
From DBA to DE: Becoming a Data Engineer
- 1. From DBA to DE: Becoming a Data Engineer Jim Czuprynski Zero Defect Computing, Inc.
- 2. Who Am I, and What Am I Doing Here? ➢E-mail me at jim@jimthewhyguy.com ➢Follow me on Twitter (@JimTheWhyGuy) ➢Connect with me on LinkedIn (Jim Czuprynski) Traveler & public speaker Summers: Wisconsin Winters: Illinois Cyclist XC skier Avid amateur bird watcher Oldest dude in martial arts class
- 3. Jim Czuprynski Liron Amitzi https://www.beyondtechskills.com The podcast that talks about everything tech – except tech.TM
- 4. What Does a Modern Oracle DBA Spend Her Time On? Protecting database health, recoverability and security Tuning queries for optimal performance and efficiency Building flexible yet resilient data models, thus ensuring data is accurate and trustworthy Keeping data sources as pristine as possible to refresh data domains efficiently
- 5. Not Everyone Can Be A Data Scientist. Thank Goodness. Data scientists report that they typically spend as much as 90% of their time cleansing data … … and that’s when they’re not searching for relevant data, in numerous places, in different formats … … while ensuring their selected data is sufficiently anonymized to protect subjects’ privacy What they’d rather be doing: Training models and interpreting results for useful insights
- 6. Data Science Is Just Like Application Development. (Not!) DevOps: CI/CD Process Flow • Focus: Capturing, retaining, and reporting on data • Errors are relatively, if not immediately, apparent • Worst case: Roll back to a prior version of the application and its objects within the database* * Assuming you’ve planned for that eventuality! Credit: Giulano Ligouri (@ingliguori) Data Science: Data > Useful Model(s) • Focus: Accurate (and thus useful) models • Machine Learning / AI involves extremely complex mathematics that devour computing cycles • Worst case: A perfect model is now utterly inaccurate! • Underfit: Poor initial training data results in bad model precisely when it’s most needed • Overfit: Good initial training data yields a good model initially … and then new, never-before-seen data screws up everything
- 7. Who Said AI/ML Was Easy? From a recent seminar with Intel’s AI/ML team It turns out that most of the time, technology isn’t the cause of project failure; rather, the human dimension is often the root cause Of course, it’s more complicated than this. Check out my recent blog post for deeper insights
- 8. CREATE TABLE t_patients ( pa_id NUMBER NOT NULL ,pa_first_name VARCHAR2(40) NOT NULL ,pa_last_name VARCHAR2(40) NOT NULL ,pa_middle_initial CHAR(01) NOT NULL ,pa_sex CHAR(01) NOT NULL . . . ); The Scourge of Bad Data (1) What should be the CHECK constraint for this column? (M)ale and (F)emale are obvious choices … … but how do we classify trans-sexual people, or those who don’t want to reveal their sex at all? Note: We haven’t even talked about the concept of gender yet.
- 9. The Scourge of Bad Data (2) An IT professional wanted to mess with California’s Automatic License Plate Reader system … so he registered his vanity plate as the word NULL After he paid the ticket, the 3rd party administrator of the ticket fines collection system apparently connected his personal details to all plates which LEOs had registered as missing or invalid The next year, he got a $35 ticket when he tried to renew his registration … because NULL was no longer acceptable $12,000 in fines later, he realized the joke was on him
- 10. The Scourge of Bad Data (3) Any decent clustering ML algorithm would likely produce findings like this when looking for unseen patterns within features The reason for same dates? Values entered for birth date (1/1/00) and registration date (1/1/18) from some municipalities’ voting records during conversion to a centralized voter registration system in 2002 And the duplicate phone numbers? They turned out to match a City of Racine office telephone number that had been entered by default because Racine’s voting registration system required a non-NULL value
- 11. So What Does a DE Do, Exactly? Giuliano Liguori, a well-known digital transformation leader, recently described the differences between Data Scientists, Data Engineers, and Data Analysts
- 12. So What Does a DE Do, Exactly? SQL Excel Tableau Python Cloud R Distributed Computing Guess what? If you’re a DBA or Developer, you’re already doing most of the work of a Data Engineer! Notice what’s at dead center of these skillsets? That’s right. The sharpest tool you already know.
- 13. What Current DE Skills Do I Need? Note: These are only my impressions of what skills are typically needed across a wide spectrum. So what skills do your Data Science team really need? Ask them. Understand statistics & probability Know how your DS team extracts & processes data (PANDA, etc.) Learn to clean & transform data before your DS team needs to Grasp key metrics of model success Remember all that high school math you asked your teacher if you’d ever really use in real life? Yeah. It’s this stuff. Yep, this means learning at least one other new language: Python
- 14. How Do You Get To Carnegie Hall? Practice, Practice, Practice. If you’re still a “core” DBA, don’t fret! You can start practicing all the skills you’ll need to become a Data Engineer It’s easy to leverage the extremely powerful Machine Learning (ML) algorithms and Analytic functions already within the Oracle database … … because sometimes the only way to acquire the skills for a new career vector is to read > learn > do > teach Check out the newest and latest features of Autonomous Database, including AutoML, OML4Py, OML4SQL, Property Graph support, and Graph Studio UI
- 15. Configuring Your OML Environment (1) Request new ML User creation 1
- 16. Configuring Your OML Environment (1) Request new ML User creation 1 Specify username, password, and details 2
- 17. Leveraging DBMS_DATA_MINING (1) Access Machine Learning tools … 1
- 18. Leveraging DBMS_DATA_MINING (1) Access Machine Learning tools … 1 … and choose from a number of available data mining examples and templates 2
- 19. AutoML: Let the Database Decide! Check out the summary of all the latest AutoML enhancements! This makes it easier for “citizen data scientists” to apply the power of ML & Analytics … … the new AutoML interface makes selection of the proper algorithms a snap … … and many more new features, including Graph Studio
- 20. Building a Data Source for AutoML to Devour CREATE TABLE t_smartmeter_business_profiles AS SELECT sm_id ,CD.cd_minority_owned ,CD.cd_family_generations ,CD.cd_years_in_business ,CD.cd_locale_ownership ,CF.pct_profit_margin ,CF.avg_credit_score ,SM.avg_kwh_used ,SM.avg_solar_kwh ,SM.pct_solar ,SM.solar_superuser FROM t_customer_creditscoring CF . . . We’re drawing on data summarized from a Hybrid Partitioned table containing financial statistics …
- 21. Building a Data Source for AutoML to Devour CREATE TABLE t_smartmeter_business_profiles AS SELECT sm_id ,CD.cd_minority_owned ,CD.cd_family_generations ,CD.cd_years_in_business ,CD.cd_locale_ownership ,CF.pct_profit_margin ,CF.avg_credit_score ,SM.avg_kwh_used ,SM.avg_solar_kwh ,SM.pct_solar ,SM.solar_superuser FROM t_customer_creditscoring CF . . . We’re drawing on data summarized from a Hybrid Partitioned table containing financial statistics … . . . ,t_customer_demographics CD ,(SELECT sm_id ,ROUND(AVG(smr_kwh_used),2) AS avg_kwh_used ,ROUND(AVG(smr_solar_kwh),2) AS avg_solar_kwh ,ROUND(AVG(smr_solar_kwh) / AVG(smr_kwh_used) ,2) AS pct_solar ,CASE WHEN ROUND(AVG(smr_solar_kwh) / AVG(smr_kwh_used) ,2) >= 0.15 THEN 1 ELSE 0 END AS solar_superuser FROM t_smartmeters ,t_meter_readings WHERE smr_id = sm_id GROUP BY sm_id ORDER BY sm_id) SM WHERE SM.sm_id = CF.cf_id AND SM.sm_id = CD.cd_id ORDER BY sm_id; … as well as customer demographics and solar energy usage data
- 22. Regression Experiments with AutoML (1) First, select an appropriate data source 1
- 23. Regression Experiments with AutoML (1) First, select an appropriate data source 1 AutoML automatically builds a list of potential features and their key metrics 2
- 24. Regression Experiments with AutoML (2) Review settings for prediction type, run time, model metric, and ML algorithms to apply 3 Start the experiment, choosing either speed or accuracy 4
- 25. Regression Experiments with AutoML (3) AutoML now finishes any sampling needed and moves on to feature selection 5
- 26. Regression Experiments with AutoML (3) AutoML now finishes any sampling needed and moves on to feature selection 5 Next, AutoML begins building the selected models 6
- 27. Regression Experiments with AutoML (4) Model generation is complete! On to Feature Prediction Impact assessment … 7
- 28. Regression Experiments with AutoML (5) Regression(s) complete! Now let’s transform the Neural Network model into a Zeppelin notebook, with just a few mouse clicks 8
- 29. Transform an AutoML Experiment into a NoteBook (1) From the Leader Board, select one of the algorithms and click on Create Notebook 1
- 30. Transform an AutoML Experiment into a NoteBook (1) From the Leader Board, select one of the algorithms and click on Create Notebook 1 Name the new notebook 2
- 31. Transform an AutoML Experiment into a NoteBook (2) The new notebook is ready. Click the link to start building paragraphs and retrieving data 3
- 32. Transform an AutoML Experiment into a NoteBook (2) The new notebook is ready. Click the link to start building paragraphs and retrieving data 3 Don’t know Python? No worries! The new notebook uses OML4Py to construct paragraphs for data retrieval and modeling 4
- 33. Transform an AutoML Experiment into a NoteBook (3) Et voila! Here’s your first results from a notebook completely generated via AutoML! 5
- 34. How Do I Keep My DE Career Relevant? How did you keep your Developer / DBA career relevant? How is this any different? ✓ Associate with other DEs, and help uplift others to DE status ✓ Attend conferences and training sessions on latest industry trends ✓ Consider certifying your hard-won, newly-acquired skills They call it life-long learning for a reason - it never, ever stops!
- 35. Are There Any DE Professional Organizations? Maybe. American Statistical Association (ASA) Offers a wide array of meetings, publications, and training as well as the vaunted PStat and GStat accreditations Data Science Council of America (DASCA) Offers several different certifications in Big Data, Analytics, and Data Science Institute for Operations Research and the Management Sciences (INFORMS) Offers various trainings, events, publications, and certifications The Association of Data Scientists (ADaSci) Based in India, they offer a Chartered Data Scientist (CDS) certification exam and training
- 36. Further Reading In the Real World of Data Science • AI Projects Fail All Too Often. Successful Ones Share a Common Secret https://gestaltit.com/tech-talks/intel/intel-2021/jimthewhyguy/ai-projects-fail-all-too-often-successful-ones-share-a-common-secret/ • Machine Learning in Production: Why Is It So Hard and So Many Fail? https://towardsdatascience.com/machine-learning-in-production-why-is-it-so-difficult-28ce74bfc732 • Fact Check-Claims about 23,000 Wisconsin voters with the same phone number and 4,000 voters registered on 1/1/1918 https://www.reuters.com/article/factcheck-wisconsin-numbers/fact-check-claims-about-23000-wisconsin-voters-with-the-same-phone- number-and-4000-voters-registered-on-1-1-1918-missing-context-idUSL1N2RU1WC • How a 'NULL' License Plate Landed One Hacker in Ticket Hell https://www.wired.com/story/null-license-plate-landed-one-hacker-ticket-hell/
- 37. Useful Oracle Documentation • What is Data Science? https://www.oracle.com/data-science/what-is-data-science/ • Machine Learning Solutions with Oracle’s Services and Tools https://www.oracle.com/a/ocom/docs/build-machine-learning-solutions-cloud-essentials.pdf • Oracle Cloud Infrastructure Data Catalog https://www.oracle.com/a/ocom/docs/ebook-cloud-infrastructure-data-catalog.pdf • OML Algorithms “Cheat Sheet” https://www.oracle.com/a/tech/docs/oml4sql-algorithm-cheat-sheet.pdf • Oracle 21c Machine Learning Basics (including AutoML) https://docs.oracle.com/en/database/oracle/machine-learning/oml4sql/21/dmcon/machine-learning-basics.html