Data - Science and Engineering slide at Bandungpy Sharing Session
0 likes64 views
This document discusses data science and engineering roles. It defines data scientist and data engineer roles. Data scientists analyze large amounts of data to answer questions and drive organizational strategy, while data engineers build systems to collect, manage and transform raw data for analysis. The document also discusses the role of AI engineers, who develop complex algorithms and infrastructure for AI systems. It provides examples of responsibilities for each role and the data science experiment process.
Data - Science and Engineering slide at Bandungpy Sharing Session
- 1. - Data - Science and Engineering
- 2. HELLO! I am Hendri Karisma Principal R&D and Data Lead at Akar Inti Data ✘ Research and Development for Advanced Data Analytics Platform. ✘ Data marketplace at nusadata.ai . You can find me at : ✘ telegram @siganteng ✘ Instagram @karism4_ ✘ X @infoHendri 2
- 3. Data What is data ?
- 5. 5
- 6. Data marketplace 6 Open Finance Telco eKYC Market Place Single platform Consumer Data Provider
- 7. 7
- 9. Science The goal of a scientist is to answer questions and discover information about their chosen field of study. What is the difference? Engineering An engineer might produce physical item or blueprint for a new process. 9
- 10. Data roles ✘ Data Scientist ✘ Data Analyst ✘ Business Intelligence ✘ Data Engineer ✘ AI/ML Engineer There is a Scientist role and 2 engineer roles. 10
- 11. Data Scientist The term “data scientist” was coined as recently as 2008 when companies realized the need for data professionals who are skilled in organizing and analyzing massive amounts of data.1 In a 2009 McKinsey&Company article, Hal Varian, Google’s chief economist and UC Berkeley professor of information sciences, business, and economics, predicted the importance of adapting to technology’s influence and reconfiguration of different industries. Data-driven individuals with high-level technical skills who are capable of building complex quantitative algorithms to organize and synthesize large amounts of information used to answer questions and drive strategy in their organization. 11
- 12. Data Engineer Data engineers work in a variety of settings to build systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret. Their ultimate goal is to make data accessible so that organizations can use it to evaluate and optimize their performance. These are some common tasks you might perform when working with data: ✘ Acquire datasets that align with business needs ✘ Develop algorithms to transform data into useful, actionable information ✘ Build, test, and maintain database pipeline architectures ✘ Collaborate with management to understand company objectives ✘ Create new data validation methods and data analysis tools ✘ Ensure compliance with data governance and security policies 12
- 13. AI Engineer Artificial intelligence (AI) engineers are responsible for developing, programming and training the complex networks of algorithms that make up AI so that they can function like a human brain. This role requires combined expertise in software development, programming, data science and data engineering. Though this roleis related to data engineering, AI engineers are rarely required to write the code that develops scalable data sharing. 13
- 14. AI Engineer Responsibilities ✘ Build AI models from the ground up and explain results to product managers and stakeholders ✘ Develop, test, and deploy AI models ✘ Convert machine learning models into APIs so other applications can utilize it ✘ Build data ingestion and data transformation infrastructure ✘ Work alongside data and business analysts ✘ Execute statistical analysis and tune results to extract better insights ✘ Automate infrastructure used by the data science team ✘ Create and manage AI development and production infrastructure 14
- 15. Data Science Experiments ✘ Do the data understanding ✘ Data exploration ✘ Feature Engineering ✘ Method/Model Experiments 15
- 16. Data flow 16
- 17. Zoom in 17
- 18. 18
- 19. ML Ops 19
- 20. Tech Stack 20
- 21. 21
- 22. 22
- 23. AI Engineering Not Only About The Tech Stack ✘ SOLID, Design Pattern, Clean Architecture, etcs ✘ Software Engineering ✘ Tech Tools Characteristics, Behaviour ✘ Basic statistics, probilistics and random variable, basic machine learning. ✘ High performance computing ✘ Security and Regulation ✘ ML Ops 23
- 24. The Benefit of Python ✘ Could run AI Algorithm as service ✘ Experiments for research phase ✘ Having a lot of Math (statistics, probabilistics) and machine learning libraries ✘ Data pipeline ✘ Easy to use and easy to learn ✘ Could implement OOP or FP 24
- 25. Example of The Engine ✘ We need to run ML service ✗ Sometimes for specific cases ✗ The classification process is about 500ms ✘ We run the process (preprocess to main analytics process) as a batch process or need to process data with Large Volume ✘ The model is lightweight 25
- 28. Could handling several cases 28
- 29. Model ✘ We often use pickle or serializable file as model output ✘ The size bigger than it should be ✘ Impacted to the memory or the space complexity 29
- 30. Sample Model 30
- 31. Sample Model 31
- 32. THANKS! Any questions? You can find me at ✘ X @infoHendri ✘ Telegram @siganteng ✘ Email: situkangsayur@gmail.com hendri.karisma@akarintidata.ai 32