TINYBERT WITH CROSS-ATTENTION + LORA + BI-GRU: A COMPACT NEURAL NETWORK ARCHITECTURE FOR FAKE NEWS DETECTION

Abstract

This paper proposes a compact neural network architecture based on TinyBERT with a Cross-Attention mechanism, Low-Rank Adaptation (LoRA), and a bidirectional recurrent Bi-GRU layer for automatic fake news detection. The rapid development of the digital information space has led to a sharp increase in the amount of disinformation, while traditional fact-checking methods can no longer effectively counter this trend. Existing neural network approaches based on large transformer models are often unsuitable for practical use due to their high computational cost and inability to explicitly analyze semantic inconsistencies between a news headline and its body text. The proposed model addresses these limitations by employing a distilled TinyBERT transformer, which significantly reduces computational requirements, and a specialized Cross-Attention module that provides sensitivity to headline–body discrepancies. The use of LoRA adapters minimizes the number of trainable parameters, speeding up the fine-tuning process, while the integration of Bi-GRU enables the preservation of contextual information in sequences. Experimental evaluation on the Fake News Classification dataset demonstrated that the model outperforms classical machine learning algorithms, achieving an accuracy of 99%, an F1-score of 0.985, and a ROC AUC of 0.998. Due to its low resource consumption, rapid adaptability to new topics, and high interpretability of decisions, the model shows strong potential for integration into real-time fact-checking services.