How do GitHub, Google, Dropbox, Monzo, Atlassian, and 13 other companies know how well AI tools work for devs? A deepdive sharing exclusive details, with CTO Laura Tacho
One thing I find challenging is: how do you know that the positive impact in the engineering organization comes because of the usage of AI or just because the teams become better (maturity) at their job?
In order to know if the positive impact comes from the usage of AI tools, do you use some kind of rule? like, "if the usage of the AI tools is >X% and we see a positive impact, then we can conclude that it is thanks to the AI usage".
There are two things that help here: we can look at AI users and non-AI users on the same teams and see the difference (controlling for different team processes, etc). We can also do a before-and-after comparison of the same cohort and see the rate of change, talk to the team about what else changed (a new process? Another new tool?) at the same time as the AI rollout, and then draw some conclusions, and test those conclusions/hypotheses again as time goes on. We still need leaders to interpret what is happening and understand the context of the org.
I cannot stress more: I dream of the day that AI takes the Product Manager's specs and deals with Jira management, scheduling the sprints based on current capacity of the teams, and deal again with Jira tickets... 😅
One thing I find challenging is: how do you know that the positive impact in the engineering organization comes because of the usage of AI or just because the teams become better (maturity) at their job?
In order to know if the positive impact comes from the usage of AI tools, do you use some kind of rule? like, "if the usage of the AI tools is >X% and we see a positive impact, then we can conclude that it is thanks to the AI usage".
There are two things that help here: we can look at AI users and non-AI users on the same teams and see the difference (controlling for different team processes, etc). We can also do a before-and-after comparison of the same cohort and see the rate of change, talk to the team about what else changed (a new process? Another new tool?) at the same time as the AI rollout, and then draw some conclusions, and test those conclusions/hypotheses again as time goes on. We still need leaders to interpret what is happening and understand the context of the org.
Super useful article, thanks a lot, Laura!
I cannot stress more: I dream of the day that AI takes the Product Manager's specs and deals with Jira management, scheduling the sprints based on current capacity of the teams, and deal again with Jira tickets... 😅
I would love to see a label of which companies are dx customers. I guess they all have pretty similar metrics.
Google, Microsoft, and Atlassian aren't DX customers.