One thing I find challenging is: how do you know that the positive impact in the engineering organization comes because of the usage of AI or just because the teams become better (maturity) at their job?
In order to know if the positive impact comes from the usage of AI tools, do you use some kind of rule? like, "if the usage of the AI tools is >X% and we see a positive impact, then we can conclude that it is thanks to the AI usage".
There are two things that help here: we can look at AI users and non-AI users on the same teams and see the difference (controlling for different team processes, etc). We can also do a before-and-after comparison of the same cohort and see the rate of change, talk to the team about what else changed (a new process? Another new tool?) at the same time as the AI rollout, and then draw some conclusions, and test those conclusions/hypotheses again as time goes on. We still need leaders to interpret what is happening and understand the context of the org.
One thing I find challenging is: how do you know that the positive impact in the engineering organization comes because of the usage of AI or just because the teams become better (maturity) at their job?
In order to know if the positive impact comes from the usage of AI tools, do you use some kind of rule? like, "if the usage of the AI tools is >X% and we see a positive impact, then we can conclude that it is thanks to the AI usage".
There are two things that help here: we can look at AI users and non-AI users on the same teams and see the difference (controlling for different team processes, etc). We can also do a before-and-after comparison of the same cohort and see the rate of change, talk to the team about what else changed (a new process? Another new tool?) at the same time as the AI rollout, and then draw some conclusions, and test those conclusions/hypotheses again as time goes on. We still need leaders to interpret what is happening and understand the context of the org.