GitHub did an experiment to see if people who use AI to help them code are faster than people who don’t use AI. They found that the debate over who is faster is still an “open question.”
Key Points:
- On a mission to measure AI-assisted developer productivity, researchers at GitHub recently ran an experiment comparing coding speeds of a group using its Copilot code completion tool versus a group relying on human ability alone.
- Copilot is used as an extension to code editors, such as Microsoft’s VS Code. It generates code suggestions in multiple programming languages that users can accept, reject or edit.
- The suggestions are provided by OpenAI’s Codex, a system that translates natural language to code and is based on OpenAI’s GPT-3 language model.
- Google Research and the Google Brain Team concluded in July, after studying the impact of AI code suggestions on over 10,000 of its own developers’ productivity, that the debate over relative performance speed remains an “open question”.
- Other researchers earlier this year, using a small sample of 24 developers found that Copilot didn’t necessarily improve task completion time or success rate but it did save developers the effort of searching online for snippets of code to solve particular problems. This reduces context switches which is important for productivity.
Source Article:
Article Highlights:
- GitHub Copilot is an AI pair-programming service that launched publicly earlier this year for $10 per user per month or a $100 per user per year
- Researchers found that between 60% to 75% of users feel more fulfilled with their job when using Copilot, feel less frustrated when coding, and are able to focus on more satisfying work
- Developers working with Copilot were 55% faster than the group without it.
Personal Insights:
- When I first saw these large language models applied to what I would call copywriting text, like advertising and marketing, I was very impressed. I didn’t know it had ingested all the world’s understanding of code…
- Then I saw how quickly it could create a SQL statement to retrieve complex data from a system.
- You can also create structured python code and complex and documented functions.
- I was really struck by how quickly a tool like this allows you to iterate. It just lets you think about the big picture and helps you work on the details much quicker.
