COMPARATIVE ANALYSIS OF POLICY-BASED AND VALUE-BASED RL METHODS FOR LOCAL MOTION PLANNING OF UAVS

Abstract

The paper presents a comparative study on the effectiveness of two fundamental Deep Reinforcement Learning architectures—Policy-based (using PPO) and Value-based (using SAC)—for the task of UAV local navigation under stochastic uncertainty. Experimental modeling, conducted in the PyFlyt environment considering turbulent wind, variable drone mass, and target motion, revealed critical discrepancies in algorithm performance. It was established that despite high training speed and sample efficiency, the SAC algorithm forms vulnerable policies with a low mission success rate (63.2%) due to value function estimation errors in dynamic states. In contrast, the PPO method ensured the formation of a robust control strategy with a 97.1% success rate and the generation of optimal trajectories without oscillations. The results confirm that for continuous control tasks in unpredictable environments, direct policy optimization methods are more effective as they avoid the "Reward Hacking" effect and better generalize the physical laws of flight.