Empirical Analysis of AI-Assisted Code Generation Tools Impact on Code Quality, Security and Developer Productivity

AI-assisted code generation tools have been the main cause of the increase in practices like code completion, bug fixing, and documentation among developers. However, the main concern regarding their effects on code quality, security vulnerabilities, and developer productivity still lacks empirical evidence.

Objective: This study conducts an empirical assessment of the AI-assisted code generation tools' effectiveness in terms of software quality metrics, security vulnerability introduction, and developer productivity, depending on the programming languages and project complexities. Methodology: A controlled experiment was performed with 120 professional developers where they were divided into experimental and control groups and 480 code modules were analyzed among Python, Java, JavaScript, and C++ projects. Cyclomatic complexity, maintainability index, and code smell density were the three parameters for measuring code quality. Static analysis tools were employed in the evaluation of security vulnerabilities, while productivity was gauged through measuring task completion time and conducting cognitive load surveys.

Results: The use of AI-assistive tools lead to a 31.4% increase in average developer productivity; however, 23.7% more security vulnerabilities were introduced in the codes generated. Code maintainability went up 18.2%, while cyclomatic complexity decreased by 14.6%. The variations in programming languages were significant, with Python being the one that realized the highest quality improvement (26.3%) and C++ the one that faced the most security risk increase (34.8%).