GitHub has conducted research into GitHub Copilot’s Technical Preview and uncovered the impact it has had on developer productivity and happiness to be improved satisfaction and conserved mental energy.
GitHub researcher Dr. Eirini Kalliamvakou says that Copilot “supports faster completion times, conserves developers’ mental energy, helps them focus on more satisfying work, and ultimately find more fun in the coding that they do.”
After early observations and interviews with users, GitHub surveyed more than 2,000 developers to learn at scale about their experience using GitHub Copilot. The research was designed with the following three goals in mind:
- To look at productivity holistically.
- To include developers’ first-hand perspectives.
- To assess GitHub Copilot’s effects in everyday development scenarios..
GitHub says the questions were modelled off of the SPACE framework which measures developer productivity using the following metrics: satisfaction and wellbeing, performance, activity, communication and collaboration, and efficiency and flow.
Results were as follows:
- Improved developer satisfaction: 60-75 per cent of users reporting feeling more fulfilled with their job, less frustrated when coding and were able to focus on more satisfying work when using GitHub Copilot.
- Conserved mental energy: 73 per cent of developers reported that GitHub Copilot helped them stay in the flow, while 87 per cent reported that their mental effort was preserved during repetitive tasks.
When speaking about their overall experience with GitHub Copilot, a senior software engineer said, “(With Copilot) I have to think less, and when I have to think it’s the fun stuff. It sets off a little spark that makes coding more fun and more efficient.”
GitHub’s takeaway from the research was that letting GitHub Copilot shoulder the boring and repetitive work of development reduces cognitive load, making room for developers to enjoy the more meaningful work that requires complex, critical thinking and problem solving.
Survey results revealed that developers completed tasks faster when using GitHub Copilot, especially for repetitive ones.
To test the result, GitHub conducted a controlled experiment, which found that developers who used GitHub Copilot completed the chosen task 55% faster than the developers who didn’t. Specifically, the developers using GitHub Copilot took, on average, 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took, on average, 2 hours and 41 minutes.
In addition to developers, GitHub says that the benefits of GitHub Copilot are being realised by engineering leaders. When considering how to keep their engineers healthy and productive, they are looking through the same lens of holistic developer wellbeing and promoting the use of tools that bring delight.
GitHub states that it will continue to conduct experiments with GitHub Copilot and share its findings.
