© Wuhan University 2026

0 Introduction

With the rapid advancement of communication and sensor technologies, the Internet of Things (IoT) is increasingly integrated into daily life and industrial applications. However, this growth brings significant security challenges. IoT devices, constrained by limited computational resources and weak defenses, are often deployed in physically accessible environments, making them prime targets for cyberattacks. Network Intrusion Detection Systems (NIDS) are critical in IoT security. Machine learning and deep learning-based anomaly detection methods can learn normal traffic patterns and identify deviations, enhancing detection of unknown threats^[1].

A prevalent challenge for machine learning-based NIDS is data imbalance, where normal traffic dominates while attack traffic is scarce and unevenly distributed. Traditional solutions include data-level techniques like oversampling^[2], under-sampling^[3], SMOTE (Synthetic Minority Over-sampling Technique)^[4], and its variants (e.g., SMOTETomek), as well as algorithm-level approaches like cost-sensitive learning^[5] and ensemble methods. Recently, deep learning techniques, particularly Generative Adversarial Networks (GANs)^[6] such as CTGAN^[7] and TAC-GAN^[8], have shown promise in generating high-quality synthetic minority class samples to address imbalance. However, current NIDS still face limitations in handling imbalanced traffic distributions, particularly in detecting low-frequency attacks.

This study proposes a multi-generator GAN based on TMG-GAN^[9] (Tabular Multi-Generator GAN) and R3GAN^[10] (Re-GAN), utilizing cosine similarity of high-dimensional generator features to decouple different class generators, reducing inter-class overlap. Relative pairing loss and R₁/R₂ gradient penalties stabilize training, preventing mode collapse. Additionally, a classifier is introduced to enhance class discrimination, avoiding inter-class confusion in generated samples. Experiments on the TON-IoT and Edge-IIoTset datasets, using F₁-score and Recall as metrics, compare the proposed method against oversampling, SNGAN (Spectral Normalization GAN), and TMG-GAN, demonstrating superior performance.

1 Overview of Imbalanced Learning

1.1 Sample-Based Imbalanced Learning

Sample-based methods address data imbalance by adjusting the proportion of different class samples, primarily through oversampling, which increases minority class samples, and undersampling, which reduces majority class samples. Traditional random oversampling duplicates minority class samples to achieve balance but risks overfitting due to mechanical replication, failing to capture intrinsic data relationships. To address this, Chawla et al^[4] proposed the SMOTE algorithm, which generates synthetic minority samples by using k-nearest neighbors to define local structures in the feature space and applying linear interpolation, effectively mitigating the scarcity of minority samples, as shown in Fig. 1.

Fig.1 Diagram of the SMOTE algorithm[4]

Undersampling achieves balance by selectively removing majority class samples, avoiding the noise introduced by synthetic samples but risking information loss. Random undersampling, while simple, may discard critical samples, disrupting the original data distribution. Density-based methods like Edited Nearest Neighbors (ENN) optimize majority class structure by retaining boundary samples and removing redundant ones, preserving discriminative information. However, both oversampling and undersampling artificially alter the original data distribution, potentially leading to models learning incorrect statistical patterns. This is particularly problematic in scenarios like industrial fault diagnosis, where synthetic or removed samples may introduce diagnostic biases.

1.2 Imbalanced Learning Based on Generative Adversarial Networks

GANs offer a novel approach to handling imbalanced data through adversarial learning. A GAN consists of a generator, which learns the latent feature distribution of minority class samples to produce realistic synthetic data, and a discriminator, which distinguishes real samples from generated ones. Through adversarial training, both components improve, enabling the generator to produce high-quality minority samples to balance the dataset. The adversarial process can be described by equation (1):

$\begin{array}{l}\underset{G}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{D}{\mathrm{m}\mathrm{a}\mathrm{x}}V\left(D,G\right)=E_{\mathit{x}\sim p_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\left(\mathit{x}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}D\left(\mathit{x}\right)\right]\\ +E_{\mathit{z}\sim p_{\mathit{z}}\left(\mathit{z}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(\mathrm{1}-D\left(G\left(\mathit{z}\right)\right)\right)\right],\left(\mathrm{1}\right)\end{array}$

where $p_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ denotes the real data distribution, $D\left(x\right)$ represents the discriminator's output for real samples (approaching 1 for strong discrimination), and $z\sim p_z\left(z\right)$ is noise sampled from a prior distribution, with $G\left(z\right)$ as the generated sample. $V\left(D,G\right)$ is the value function in GANs, describing the minimax game between generator G and discriminator D. The discriminator aims for $D\left(G\left(z\right)\right)\approx \mathrm{0}$, identifying fake samples, while the generator strives for $D\left(G\left(z\right)\right)\approx \mathrm{1}$, making generated samples indistinguishable from real ones.

Compared with traditional oversampling, GANs excel at capturing complex nonlinear feature relationships, producing semantically meaningful samples, particularly in high-dimensional data like images and text. For example, in drone aerial imaging, cloud occlusion often causes data loss. Wei et al^[11] proposed a two-stage DCGAN-based method for thick cloud region content generation, incorporating Bag of Words (BoW) algorithms and an affine network to improve DCGAN's structure and loss function, outperforming baseline methods in semantic accuracy and visual quality.

In imbalanced learning, GAN variants enhance performance. Conditional GANs (CGANs)^[12] incorporate class labels to generate targeted samples, addressing multi-class imbalance. Wasserstein GANs (WGANs)^[13] use Wasserstein distance to stabilize training and improve sample quality. Auxiliary Classifier GANs (ACGANs)^[14] extend the discriminator to classify categories, incorporating labels into both generator and discriminator to ensure generated samples align with specific classes and are realistic. This enhances class-specific sample generation, improving model learning in imbalanced scenarios.

However, these technical innovations do not fully mitigate GANs' inherent limitations. Mode collapse remains a critical issue, where the generator may overfit to specific minority class features, producing samples that lack diversity and fail to cover the full feature space of real samples. In financial fraud detection, for instance, if the generator only captures a few typical fraud patterns, the synthetic samples may not reflect the complexity of evolving fraud tactics, limiting the model's ability to detect novel fraud types.

Additionally, training instability is a key constraint for GAN applications^[15]. An overly strong discriminator can cause vanishing gradients in the generator, while a weak discriminator leads to poor-quality samples. This delicate balance is highly sensitive to training parameters and data preprocessing. In practice, GAN training for imbalanced data often requires regularization techniques like gradient penalties or spectral normalization, increasing model tuning complexity and computational demands. Inter-class overlap further exacerbates the aforementioned issues^[16]. When data from different classes show significant overlap in the feature space, the discriminator struggles to clearly distinguish the boundary between real samples and generated ones, causing the generator to fall into local optima during training.

1.3 Imbalanced Learning Based on Diffusion Models

Diffusion models^[17] have emerged as a powerful approach for data generation, offering an alternative to GANs for addressing class imbalance in tabular datasets. TabDDPM^[18] is a diffusion-based generative framework specifically designed for tabular data which can handle mixed data types consisting of numerical and categorical features. By iteratively refining noisy data through a series of denoising steps, TabDDPM generates realistic and diverse samples that enhance the representation of underrepresented classes. This approach improves the robustness of intrusion detection systems and other classifiers in imbalanced settings, such as IoT environments.

2 Proposed Approach

2.1 Research Motivation

Compared with traditional oversampling, conventional GANs achieve better results in imbalanced learning but still face challenges like mode collapse and inter-class overlap. This study proposes a novel model integrating R3GAN's adversarial loss mechanism with TMG-GAN's multi-generator architecture, as shown in Fig. 2. The approach employs a dual optimization strategy: a multi-generator structure with classification loss captures fine-grained feature distributions of different attack types, enhancing class specificity; and R3-loss with gradient penalties significantly improves training stability. This model reduces inter-class overlap while enhancing intrusion detection performance. Experiments were conducted on the TON_IoT ^[19] and Edge-IIoTset ^[20] datasets, comparing the proposed method against other data augmentation techniques.

Fig.2 Multi-generator generative adversarial network based on R3-Loss

2.2 Proposed Methodology

2.2.1 TMG-GAN

TMG-GAN, proposed by Ding et al^[9], enhances GAN performance for imbalanced data through multiple generators, a classifier, and a feature extractor to separate class-specific generators. Its implementation is as follows:

1) Multi-Generator Structure: Each class uses a dedicated generator to handle diverse data types simultaneously.

2) Classifier Integration: A classifier is added to output class labels alongside authenticity judgments. The feature extractor, shared by the discriminator and classifier, splits into two branches: one for the discriminator (judging sample authenticity) and one for the classifier (determining sample class).

3) Feature Extractor: High-dimensional feature vectors are extracted from the discriminator's final layer. During generator training, cosine similarity between high-dimensional features of real $x_k$ and generated $\widetilde{x_k}$ samples for class $k$ is computed and maximized to align generated samples with real ones, as shown in equation (2):

$\uparrow O_k\left(\mathit{F}\left(\widetilde{x_k}\right),\mathit{F}\left(x_k\right)\right)=\left|\frac{\mathit{F}\left(\widetilde{x_k}\right)\mathit{F}\left(x_k\right)}{\Vert \mathit{F}\left(\widetilde{x_k}\right)\Vert \Vert \mathit{F}\left(x_k\right)\Vert }\right|,k\in \left\{\mathrm{1},\cdots ,N\right\},(\mathrm{2})$

where $\mathit{F}\left(x\right)$ and $\mathit{F}\left(\widetilde{x_k}\right)$ are feature vectors, and the goal is to increase cosine similarity for each class $k\in \{\mathrm{1},\cdots ,N\}$.

4) Generator Decoupling: To reduce inter-class overlap, cosine similarity between high-dimensional features of samples from different generators $(k\ne j)$ is minimized, as shown in equation (3):

$\downarrow O_k\left(\mathit{F}\left(\widetilde{x_k}\right),\mathit{F}\left(\widetilde{x_j}\right)\right)=\frac{\mathrm{1}}{N-\mathrm{1}}\sum _j\left|\frac{\mathit{F}\left(\widetilde{x_k}\right)\mathit{F}\left(\widetilde{x_j}\right)}{\Vert \mathit{F}\left(\widetilde{x_k}\right)\Vert \Vert \mathit{F}\left(\widetilde{x_j}\right)\Vert }\right|.\left(\mathrm{3}\right)$

This encouraging distinct feature learning and separating generated samples in the feature space.

The loss functions of TMG-GAN are summarized as:

discriminator loss = WGAN loss + classification loss;

generator loss = WGAN loss + classification loss + feature similarity + decoupling loss.

This approach generates high-quality samples with minimal inter-class overlap.

2.2.2 R3GAN

Proposed by Huang et al^[10], R3GAN integrates the relativistic pairing GAN (RpGAN) loss function with specialized gradient penalties to address mode collapse and non-convergence issues previously mitigated by ad-hoc techniques. RpGAN, introduced by Jolicoeur-Martineau et al^[21] in 2019, contributes a novel loss function. The general GAN loss is:

$L_{\theta ,\psi }=E_{z\sim p_z}\left[f\left(D_{\psi }\left(G_{\theta }\left(z\right)\right)\right)\right]+E_{x\sim p_D}\left[f\left(-D_{\psi }\left(x\right)\right)\right].\left(\mathrm{4}\right)$

RpGAN uses the softplus function: $f\left(t\right)=-\mathrm{l}\mathrm{o}\mathrm{g}\left(\mathrm{1}+{\mathrm{e}}^{-t}\right)$, and computes the loss based on the difference between discriminator outputs for real and generated samples:

$L\left(\theta ,\psi \right)=E_{z\sim p_z,x\sim p_D}\left[f\left(D_{\psi }\left(G_{\theta }\left(z\right)\right)-D_{\psi }\left(x\right)\right)\right].\left(\mathrm{5}\right)$

Unlike traditional GANs, which can only distinguish real from fake samples, RpGAN amplifies the gap between them by feeding their output difference into the loss function. R3GAN further incorporates zero-centered gradient penalties for real data $R_{\mathrm{1}}$ and generated data $R_{\mathrm{2}}$, with ablation studies confirming their optimality, naming the model R3GAN and its loss R3-loss.

2.2.3 Model fusion design

Building on TMG-GAN and R3GAN, this study proposes a fused model, illustrated in Fig. 3, which systematically optimizes architecture and training strategies. The model retains TMG-GAN's multi-generator structure, assigning a generator per class, and uses a shared discriminator-classifier with dual branches. It incorporates R3GAN's training mechanisms as follows:

Fig.3 Overall process of data augmentation

1) Loss Function Replacement: TMG-GAN's WGAN loss is replaced with RpGAN's loss to better amplify the real-fake sample gap.

2) Gradient Penalties: $R_{\mathrm{1}}$ and $R_{\mathrm{2}}$ penalties stabilize training, suppress mode collapse, and enhance decision boundary clarity.

The discriminator loss is:

$L_{\mathrm{D}}=L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{D}}+{\lambda }_{\mathrm{c}}{L}_{\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}}+{\lambda }_{\mathrm{1}}{R}_{\mathrm{1}}+{\lambda }_{\mathrm{2}}{R}_{\mathrm{2}},\left(\mathrm{6}\right)$

where $L_{{}_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{D}}}$ is the RpGAN loss, $L_{\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}}$ is the classification loss, and ${\lambda }_{\mathrm{c}},{\lambda }_{\mathrm{1}},{\lambda }_{\mathrm{2}}$ are weight hyperparameters. The calculation formula for $L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐D}$ is as follows.

$L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{D}}=E_{z\sim p_z,x\sim p_{\mathrm{D}}}\left[f\left(D\left(x\right)-D\left(G\left(z\right)\right)\right)\right],(\mathrm{7})$

where $p_z$ is the normal distribution, $p_{\mathrm{D}}$ is the real data distribution, $D\left(x\right)$ is the output of discriminator, $G\left(z\right)$ is the generated example of generator.

The generator loss is:

$L_{\mathrm{G}}=L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{G}}+{\lambda }_{\mathrm{c}}{L}_{\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}‐\mathrm{G}}+{\lambda }_{\mathrm{f}}{L}_{\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}+{\lambda }_{\mathrm{d}}{L}_{\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{p}\mathrm{l}\mathrm{e}},\left(\mathrm{8}\right)$

where $L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{G}}$, $L_{\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}‐\mathrm{G}}$, $L_{\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}$, and $L_{\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{p}\mathrm{l}\mathrm{e}}$ represent RpGAN generator loss, classification loss, feature similarity loss, and decoupling loss, respectively, with corresponding weights. The calculation formula for $L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{G}}$ is as follows:

$L_{\mathrm{R}\mathrm{p}\mathrm{G}\mathrm{A}\mathrm{N}‐\mathrm{G}}=E_{z\sim p_z,x\sim p_{\mathrm{D}}}\left[f\left(D\left(G\left(z\right)\right)-D\left(x\right)\right)\right].(\mathrm{9})$

The differences between the proposed model, TMG-GAN, and R3GAN are shown in Table 1.

The overall process, shown in Fig. 3, involves training the generative model, augmenting minority attack data to 50% of the dataset using class-specific generators, and evaluating the augmented data quality with an intrusion detection model.

3 Experimental Setup

3.1 Datasets

This study uses the TON-IoT and Edge-IIoTset datasets, both recent IoT intrusion detection datasets reflecting the long-tail distribution typical of network traffic: abundant normal traffic and scarce attack traffic.

TON-IoT^[19], developed by the University of New South Wales, Australia, supports AI-based security applications in Industry 4.0. It integrates heterogeneous data sources, including IoT sensor telemetry, multi-OS audit logs, and network traffic. Table 2 shows its class distribution.

Edge-IIoTset^[20], released in 2022, is a novel dataset for IoT and Industrial IoT (IIoT) security, collected using platforms like ThingsBoard, OPNFV, specifically incorporating industrial protocols like Modbus TCP/IP and Mosquitto MQTT. It includes 15 attack types, with this study focusing on the top 10 by frequency. From the original 1.8 million records, 10% were systematically sampled for efficiency. Table 3 details its class distribution.

Data preprocessing addressed continuous and categorical features. Categorical features were handled using one-hot encoding, transforming each category into a binary vector with a single 1 and remaining 0 s. To mitigate scale and unit differences, data standardization was applied, converting features to a standard normal distribution (mean 0, standard deviation 1), as shown in equation (10):

$x^{\text{'}}=\frac{x-\overline{x}}{\sigma },\left(\mathrm{10}\right)$

where $x$ is the original feature value, $\overline{x}$ is the feature mean, $\sigma$ is the standard deviation, and $x^{\text{'}}$ is the standardized value. This eliminates adverse effects of scale differences on model training.

3.2 Baseline Setup

Three baseline models were selected for comparison: SMOTETomek^[4], SNGAN^[22], and TMG-GAN^[9] (also serving as an ablation study).

1) SMOTETomek combines SMOTE's oversampling, generating synthetic minority samples via k-nearest neighbor interpolation, with undersampling to balance class distribution.

2) SNGAN employs spectral normalization on discriminator weights, constraining the Lipschitz constant to enhance training stability and generate high-quality, diverse minority samples that preserve data structure.

3) TMG-GAN, described in Section 2.2.1, is not

reiterated here.

3.3 Experimental Configuration

For fairness and reproducibility, all experiments were conducted under identical hyperparameter settings. The configuration of data augmentation model are as Table 4.

Besides, intrusion detection system (IDS) is implemented as a Deep Neural Network (DNN) because of the remarkable performance of DNNs^[23]. For this model, the hyperparameter are listed in Table 5 and the architecture settings are shown in Fig. 4.

Fig.4 Architecture of deep neural network

4 Experimental Results

Using the augmented datasets, this study conducted multi-class intrusion detection experiments with a deep neural network, comparing the performance of unaugmented data, SMOTETomek, SNGAN, TMG-GAN, and the proposed model across Precision, Recall, and F₁-score metrics.

Bar charts illustrate the performance of the five models on the TON-IoT and Edge-IIoTset datasets, with overall Precision, Recall, and F₁-score comparisons shown in Fig. 5 and Fig. 6 (best values in bold). On the TON-IoT dataset, unaugmented data yielded lower scores: Precision (74.01%), Recall (83.93%), and F₁-score (78.34%), indicating severe performance degradation due to data imbalance. While SMOTETomek, SNGAN, and TMG-GAN improved performance, the proposed model achieved the highest scores: Precision (98.53%), Recall (98.51%), and F₁-score (98.51%), demonstrating its effectiveness in data augmentation and model training.

Fig.5 TON-IoT dataset performance comparison of various models

Fig.6 Edge-IIoTset dataset performance comparison of various models

On the Edge-IIoTset dataset, unaugmented data recorded Precision (87.10%), Recall (89.74%), and F₁-score (87.97%). Although SMOTETomek, SNGAN, and TMG-GAN showed improvements, the proposed model outperformed them with Precision (98.41%), Recall (98.38%), and F₁-score (98.39%), highlighting its strong generalization and ability to address data imbalance.

Table 6 compares TMG-GAN and the proposed model's Precision, Recall, and F₁-score for each attack class. The proposed model achieved overall scores of 98.53%, 98.51%, and 98.51%, surpassing TMG-GAN's 97.80%, 97.65%, and 97.67%, respectively. It excelled in detecting complex attacks, with Precision improving by nearly 5% to 99.96% for password attacks and Recall reaching 99.97% for SQL_injection attacks. For DDoS_UDP and DDoS_ICMP, the proposed model's F₁-score significantly outperformed TMG-GAN. The model proposed in this paper may not outperform TMG-GAN on certain metrics, but due to the use of R3-loss, it achieves higher overall scores compared to TMG-GAN. Its balanced performance across attack types, with less fluctuation than TMG-GAN, ensures higher detection consistency and reliability, reducing false negatives and positives in practical cybersecurity scenarios.

Figure 7 illustrates the loss curves of TMG-GAN and the proposed model on the TON-IoT dataset. TMG-GAN's loss, without R3-loss, exhibits significant fluctuations, while the proposed model's loss is notably smoother, indicating improved training stability.

Fig.7 Classifier loss variation of TMG-GAN and the proposed model

5 Conclusion

This study addresses performance degradation in intrusion detection systems due to class imbalance in IoT datasets by introducing a novel data augmentation framework based on a multi-generator adversarial network enhanced with R3-loss. The framework integrates TMG-GAN's class-specific generators for targeted sample generation and inter-class feature decoupling with R3GAN's relativistic adversarial training, incorporating RpGAN loss and dual gradient penalties to ensure stability and prevent mode collapse. It features a dual-branch discriminator-classifier and cosine-similarity-driven feature optimization to produce high-quality, diverse, and class-consistent synthetic samples, enhancing minority class representation and sharpening class boundaries. Evaluations on TON-IoT and Edge-IIoTset datasets show superior performance over baselines like SMOTETomek, SNGAN, and TMG-GAN, particularly in detecting complex attacks.

Future work could explore lightweight model architectures to reduce computational demands, dynamic training strategies for adaptive learning in varying IoT scenarios, and cross-domain applicability to extend the framework beyond industrial IoT, such as to consumer or edge computing environments, thereby improving real-world deployment and efficiency.

References

All Tables

All Figures

	Fig.1 Diagram of the SMOTE algorithm[4]
In the text

	Fig.2 Multi-generator generative adversarial network based on R3-Loss
In the text

	Fig.3 Overall process of data augmentation
In the text

	Fig.4 Architecture of deep neural network
In the text

	Fig.5 TON-IoT dataset performance comparison of various models
In the text

	Fig.6 Edge-IIoTset dataset performance comparison of various models
In the text

	Fig.7 Classifier loss variation of TMG-GAN and the proposed model
In the text