Human oversight remains crucial in AI-assisted security workflows, as one firm’s researchers have concluded

AI-powered security reviews, such as those in Anthropic’s Claude Code platform launched in August 2025, may inadvertently introduce new risks rather than reliably improving software safety, according to researchers.

While AI-driven automated code assessments can detect common vulnerabilities such as like cross-site scripting (XSS) and some authorization bypass flaws, security researchers from Checkmarx had on 9 September 2025  disclosed that they have found significant shortcomings when the Claude Code AI is tasked with more-complex or deceptive security issues.

The basic idea of using AI to test code before it reaches production is to run it through a basic security check, targeting issues such as insecure data handling, vulnerable dependencies, SQL injection, and authentication flaws. However, in their testing of the Claude Code automated review system, the testing process had missed a remote code execution vulnerability involving Python’s pandas library, mistakenly classifying a genuine threat as a false positive.

Furthermore, the researchers had intentionally devised code with misleading commentary and functions designed to appear safe, which the AI subsequently mistakenly marked as having “no security impact” despite the presence of unsafe operations.

Another concern highlighted is the platform’s practice of generating and executing its own test cases during reviews. The researchers are warning that executing code — especially in developer environments — can open new avenues for risk if, for example, testing inadvertently triggers malicious code hidden in dependencies. Anthropic itself cautions users to run security reviews only on trusted code, with explicit warnings about possible errors and by prompt-injection risks.

Organizations need to maintain strict separation between development and production systems, prevent use of production credentials in development environments, mandate human approval for risky AI actions, and strengthen endpoint security practices.

The general conclusion is that such AI features can be susceptible to suggestion, and human oversight remains crucial in AI-assisted security workflows. Other AI code-security review systems have distinct approaches that may result in different vulnerabilities and strengths.