AUTOMATED DETECTION OF SQL INJECTION VULNERABILITIES IN CHATBOTS USING REINFORCEMENT LEARNING

Abstract

This paper investigates the use of reinforcement learning algorithms for automated detection of SQL injection vulnerabilities in conversational AI agents that use APIs and databases. It was developed a Gymnasium-compatible environment called SQLiChatbotEnv and implemented a system based on Proximal Policy Optimisation (PPO), Advantage Actor-Critic (A2C) and REINFORCE methods to train an intelligent agent to detect and exploit various types of SQL injections in an automated manner. The created environment simulates realistic scenarios of interaction with vulnerable chatbots, including a multi-component action space, a reward system, and mechanisms for tracking the progress of vulnerability detection. A specialised reinforcement learning environment, SQLiChatbotEnv, simulates real-world scenarios of interaction with vulnerable chatbots and allows you to configure a chatbot environment with one of 5 major database management systems to choose from (MySQL, PostgreSQL, Microsoft SQL Server, Oracle, and SQLite). SQLiChatbotEnv also supports several key types of SQL injections, such as union-based attacks and error-based exploitation, and allows you to explore the database schema – to identify table and column names. The system allows you to use SQL query obfuscation in a message to a chatbot, which allows you to bypass basic security checks that may be present in a real system. Contextual framing allows you to use the natural integration of SQL injection into the conversation, for example, masking with phrases such as ‘I'm trying to understand...’, imitating the behaviour of a regular user. To encourage the agent to search for vulnerabilities efficiently, the system allows you to configure rewards and penalties for typical actions, such as discovering new information, data leakage, or using a SQL injection template that is incompatible with the actual database type. A comparative analysis of the performance of the three reinforcement learning algorithms over 2500 training episodes is conducted. The experimental results show that A2C demonstrates the best combination of convergence speed and learning stability, reaching a reward of 100 points in 30 episodes and a final performance of 232.82 ± 16.44 with the lowest coefficient of variation of 16.5%. PPO is characterised by the slowest convergence to high thresholds (221 episodes to a score of 150) and the highest variability of results (35.6%), but demonstrates the best ability to fully detect all types of vulnerabilities (87.4% of episodes). REINFORCE shows balanced intermediate results with a moderate convergence rate (145 episodes to a reward of 100), stability (coefficient of variation 21.4%) and high efficiency of vulnerability research (78.0% of episodes with all types of attacks). The practical significance of the work is to create an automated tool for testing the security of conversational AI agents. The results of the study demonstrate the prospects of using reinforcement learning for cybersecurity tasks and automating penetration testing processes.