A Comparative Security Evaluation of AI-Generated Code Across Large Language Models and Programming Languages

Large Language Models (LLMs) have been widely used in software development, yet the security of AI-generated code remains a critical concern. This research examines security vulnerabilities in code generated by seven LLMs, which are GPT-5, Claude Sonnet 4.5, DeepSeek R1 70B, Gemini 3 Flash, Llama 3.1 405B, Mistral Large 2, and Nova Pro, across four programming languages, which are JavaScript, Python, C++, and Java. We propose a new approach using 100 complex prompts per language identical across the LLMs.

We analyzed the most frequent Common Weakness Enumeration (CWE) categories across different LLMs and programming languages. Each snippet was analyzed via a hybrid pipeline that combines three Static Application Security Testing (SAST) tools, which are Semgrep, Snyk and SonarCloud, with manual review. Findings were classified by Common Vulnerability Scoring System (CVSS) severity.

The results show that 936 of the 2,800 snippets contain vulnerabilities, and we identified 1,703 vulnerabilities from 36 CWEs, unevenly distributed across languages (JavaScript: 695, Python: 398, Java: 348, C++: 262). Although all LLMs introduce vulnerabilities, their CWE and severity distributions vary significantly across both LLMs and programming languages, with the same model concentrating on different CWE subsets depending on the language. Because raw vulnerabilities ignore differences in code size and vulnerability severity, we introduce two new metrics for a fair comparison, namely Vulnerabilities per Line of Code (V/LoC) and Weighted Security Risk per Line of Code (WSR/LoC).

Our research highlights the need for language aware and model aware security measures to mitigate risks in AI-generated code.