Lessons learned working for (almost) 4 years in Data Science projects

In this article, I will share some insights into the experiences I have had since I transitioned from academia in computational/theoretical Physics into a Data Scientist, and recently Data Scientist/Developer, role.

The beginning of my journey started about on April 2021 (see this medium article) when I got hired by a healthcare startup through a technical training scholarship (see this article) in order to work with medical data analysis and related supervised and unsupervised machine learning modeling. There have been almost two years of a lot of data analysis and insights obtained from relatively small datasets, since the startup when trying to scale and get more clients adhering to its pain releasing solution.

Two scientific articles in respected journals were published concerning data analysis and modeling, as you can see here and here, which gave robust evidence for the validation of the proposed scientific methodology to relieve the patient's pain throguh customized photobiomodulation therapy. The work routine in this startup was not tough, as there was not a lot of data to analyze, however, I spent my free time always increasing my technical skills related to AI/Data Science and programming, and searching for new opportunities in the job market.

After the technical training period finished, I was the day after hired by PUC-Rio to work in a Research, Development, and Innovation (R,D&I) project named Ensights in partnership with (currently) Eletrobras/Furnas, the largest enterprise in the brazilian electric sector. The main goal of this project was to use IoT, chromotographic and sensor data in order to forecast the likelihood of a power transformer failure/breakdown with some days in advance, potentially avoiding disasters in these power transformers, as well reducing penalties from ANEEL (brazilian regulatory entity in the brazilian electric sector) because of these "unexpected" failures. The project started in an accelerated pace, since we were at that time, on April 2023, rushing to deliver high-impact results because the project was about intended to finish at the end of that year. Fortunately, the whole project evolved considerably well in that period, and the project was further extended for more 6 months, aiming at putting models in the production environment and fine-tune the existing models by bringing new data and methodologies/heuristics. The main achieved results of this project were published in international periodicals, which you can find here and here, and the deployed models are delivering high-value to Eletrobras nowadays, despite the team responsible for the continuation of the project changed.

After the finishing of the extension of the Ensights project, I was transfered to another PUC-RIO university's department, the heretofore department named Laboratory of Software Engineering, now named ECOA/PUC-Rio, in order to work in a hybrid role of Data Scientist and Web Developer in a R,D&I project in collaboration with Petrobras, the largest and most valuable enterprise in Brazil, South America.

The project consists of developing web applications to be used by Petrobras's employees aiming at describing and predicting conditions related to pipe's maintenance and risk of failure. So far, many top-edge technologies/frameworks have been used, such as Django, React, JavaScript, HTML, CSS, Python, bash scripts, docker/containerization, and other AI-related, visualizations and statistical methods are under development.

Four years after the career transitioning, I was brought in each project into more complex and challenging projects, which also made me develop new skills and change the way I worked. From my first job in data science to my latest job, at ECOA, I noticed that when you are working for an enterprise:

1. fast pace: projects for/with enterprises evolve at a much higher pace than in academia, since the proposed methodology is aimed at delivering valuable contributions to enterprises and have to be tested in real-world scenarios as soon as possible: be it increasing profits or reducing operational costs;
2. recommendation for a job is crucial: for the current project I am doing, I did not have most of the stacks in my portfolio, however, I got a good recommendation from the past project and my current manager regarded it as very important, since I was able to make a good team work, as well as learning quickly;
3. good team-work: of course, working well in group, manage emotions and stress in hard situations is a big plus in programming, since this profession is very stressful and there are several reported cases of burnout among programmers. I learned how to master this skill throughout my academic life, and, yes, it was tough to get it! Moreover, from my own job experiences, I learned that if there is just one problematic person in the work environment, the project's performance will be highly negatively impacted, unfortunately. Therefore, managing emotions is very important and surely is not easy to develop it - it comes along you age -;
4. hard work/commitment: of course that managing emotions is important, but technical skills are also important. In a fast-evolving world, mainly in technology-related jobs, the technical skills evolve at a very rapid pace and it is very hard to accompany these changes, as you have to learn the whole concept/paradigm, and, as well, know how to apply it in a real-world situation, doing high-quality code followed by comprehensive unitary tests. It is noteworthy that five years ago, I had much more time to learn what I wanted by "passion" and much more free time;
5. good amount of money comes with a cost: although I have more than tripled my netto income comparatively to when I was a post-doc in Physics, I confess that on friday night I am very tired and I really want to rest. On Saturday, most times I spend my free time by watching sit-coms and related entertainment. On Sunday I study a little bit of data science and work on personal/side projects. But comparatively to the academia, 4 years ago or more, I really feel that now it is a real job and I get exhausted by the end of the week. Therefore, the cost of a comfortable life and getting a good amount of money at the end of the month is loneliness, stress, and few free time!! Yes, everything comes at a cost.

That's all I wanted to share with you by now.

Hope you enjoyed it.

Cheers.