Reinforcement learning for channel selection

Integrating Artificial Intelligence (AI) into a hybrid PLC–RF communication enables adaptive channel selection based on real-time network conditions. Such intelligent switching is vital for improving data integrity (minimizing packet loss) and timeliness (reducing latency) in smart metering and industrial automation environments. This work focuses on developing a reinforcement learning-based channel selection mechanism for hybrid PLC–RF communication under dynamic conditions.

Threshold-based methods in G3-Hybrid face three limitations that threshold optimization alone cannot resolve: (i) they cannot jointly optimize over correlated metrics (e.g., high SNR but elevated packet loss simultaneously), (ii) they degrade under non-stationary channel statistics that shift outside the calibrated operating range, and (iii) they do not generalize across heterogeneous PLC and RF channel characteristics without per-deployment re-tuning. A PPO-based policy, by contrast, learns a non-linear decision boundary across the full joint metric space from experience and updates through gradient-based optimization, enabling adaptation to channel dynamics that invalidate fixed thresholds. Furthermore, the proposed interface-selection layer operates entirely above the MAC sublayer, acting on link-quality observations exposed by the existing G3-PLC Hybrid and TI 15.4 stacks without modifying their internal frame scheduling, retransmission, or deduplication mechanisms. This means the architecture is fully coexistent with standard-compliant G3-Hybrid or PRIME Hybrid MAC implementations and does not require a replacement protocol stack.

Methodological framework

A key challenge in developing such a mechanism is the lack of publicly available, labeled datasets that capture comprehensive metrics (e.g., latency, packet loss, SNR) across diverse PLC and RF environments. This data scarcity makes traditional supervised learning approaches impractical. To overcome this, a Reinforcement Learning (RL) approach was adopted. RL provides a model-free, data-efficient alternative by letting an agent learn a policy through direct interaction with the environment. To identify the most effective strategy for this dynamic channel selection problem, four distinct RL algorithms were implemented and compared: Q-Learning, as a foundational, tabular-based baseline to assess performance in a discretized state space; Double Deep Q-Network (DDQN), chosen for its ability to handle continuous state spaces using function approximation and for its known stability improvements over standard DQN; Deep Deterministic Policy Gradient (DDPG), selected as a state-of-the-art actor-critic method for continuous action spaces, adapted here for a binary choice; and Proximal Policy Optimization (PPO), implemented due to its strong reputation as a policy-gradient method that balances exploration, convergence speed, and stable performance, making it a suitable candidate for dynamic network environments. This comparative approach enables a robust evaluation of trade-offs between model complexity and decision performance in hybrid communication.

The selection of Q-Learning, DDQN, DDPG, and PPO as the four evaluated baselines was guided by the specific structural properties of the channel selection problem rather than an attempt to survey the full space of modern reinforcement learning algorithms. The task addressed in this work is a single-agent, binary discrete-action decision made independently at each node from a low-dimensional, memoryless six-dimensional state, as established in “Simulated training environment” section. The four chosen algorithms span the principal methodological axes relevant to this setting: tabular value-based learning through Q-Learning, deep value-based learning with function approximation through DDQN, continuous-action actor-critic learning adapted to a binary decision through DDPG, and clipped-objective policy-gradient learning through PPO. This spread allows the comparison to isolate whether representation capacity, learning paradigm, and update stability mechanism drive performance differences on this task, which is the central methodological question addressed by the testing and multi-seed evaluations in “Testing and evaluation” and ’Statistical significance and multi-seed validation” sections.

More recent algorithms noted by the reviewer target problem characteristics that are not present in the current formulation. SAC and TD3 extend the continuous-control paradigm already represented here by DDPG through additional entropy regularization or twin critics, improvements that primarily address continuous action spaces rather than the binary PLC or RF decision studied here. DDPG was retained specifically to test whether a continuous-action actor-critic method transfers to this discrete setting, and the results in “Statistical significance and multi-seed validation” and “Reward function sensitivity and policy robustness analysis” sections show pronounced instability, with a mean F1 score as low as 13.03 percent in multi-seed evaluation and policy collapse under reward coefficient perturbations, providing empirical rather than assumed justification for this choice. Rainbow DQN combines several discrete value-based refinements, most of which target large discrete action spaces such as those in Atari-scale benchmarks; the two-action space evaluated here already isolates the double Q-value correction through DDQN, so the incremental contribution of the remaining Rainbow components is expected to be small. A3C and IMPALA are primarily distributed training architectures developed to accelerate learning through many parallel environment instances; the present environment converges within 200 to 300 episodes, as reported in “Training and hyperparameters” section , so distributed training infrastructure offers no meaningful benefit at this scale. Transformer-based reinforcement learning is designed to model long-range temporal dependencies in sequential or partially observable tasks, whereas the PLCRFNetworkEnv simulation environment is explicitly memoryless, with channel metrics re-sampled independently at every step as detailed in “Simulated training environment” section , removing the sequential structure that transformer-based architectures are built to exploit. Graph reinforcement learning and multi-agent reinforcement learning are suited to problems requiring coordination across an explicit network topology; the present architecture instead performs independent, per-node inference with no inter-node coordination, a deliberate design choice discussed in “Scalability and deployment considerations” section that keeps the per-node computational and communication overhead constant as the network scales. Extending the comparison to a coordinated multi-agent or graph-based formulation is acknowledged as a promising direction for future work. A concrete motivating scenario for such coordination is the DCU-level correlated impulsive noise event modeled in “Simulation-based network scalability and stress testing” section: when all nodes within an affected DCU cluster experience degraded PLC quality simultaneously, independent per-node policies may each individually switch to RF at the same time, concentrating traffic and increasing the risk of the RF congestion already observed in the high-traffic stress scenarios in “Simulation-based network scalability and stress testing” section. A multi-agent framework in which neighboring meters share local channel-quality information, or in which a DCU-level coordinator arbitrates channel assignment across its cluster, could in principle balance channel load during such correlated events rather than allowing every node to react identically and independently, and represents a promising extension of the per-node decision framework studied in this work.

Simulated training environment

In RL, the environment defines the context for learning. A custom environment, ‘PLCRFNetworkEnv‘, was developed to simulate the two-channel (PLC and RF) communication system. The agent’s objective is to select the optimal channel on a per-packet basis. Each state is a 6-dimensional vector containing three features for each channel: latency, packet loss, and Signal-to-Noise Ratio (SNR). The agent has a discrete action space with two choices: select the PLC channel (Action 0) or the RF channel (Action 1).

The three-metric state representation was designed to capture the essential and orthogonal dimensions of link quality for the binary PLC/RF interface selection task. Latency characterizes access delay, packet loss characterizes link reliability, and SNR characterizes channel quality; together they provide a complete description of whether a communication path is suitable for AMI data delivery at the packet level. The strong classification performance achieved by PPO (93.78% accuracy, 87.74% F1-score) and the reward sensitivity analysis, which shows policy stability under \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm 50%$$\end{document} perturbations of each individual metric weight, both confirm that this representation is sufficient to learn a robust and well-generalizing channel selection policy. Extending the state with additional indicators such as RSSI, LQI, BER, queue occupancy, or residual energy is a natural and promising direction for future work, as a richer state space could enable finer-grained decisions in more complex multi-hop or heterogeneous deployments. Directionally, indicators such as BER, RSSI, and LQI are expected to refine rather than qualitatively change the current decision boundary, since they largely provide lower-level confirmation of the same channel-quality dimension already captured jointly by SNR and packet loss, with the largest expected benefit concentrated in marginal-SNR conditions where the coarser three-metric state may be ambiguous. Queue occupancy, by contrast, would expose congestion state before it manifests as packet loss, and could be expected to reduce reactive switching frequency under the traffic-overload conditions evaluated in “Simulation-based network scalability and stress testing” section by allowing the agent to anticipate congestion rather than respond to it after the fact. Residual node energy would allow the policy to weigh the fixed RF-preference bonus in the reward function defined in “Reward function design” section against the actual battery state of each node, which would be expected to shift channel selection toward PLC for energy-depleted nodes even when RF offers a marginal link-quality advantage. For the present binary selection problem, however, the three chosen metrics provide a compact, hardware-native, and proven-sufficient basis that also keeps the policy network within the 64.7 KB Flash budget required for embedded inference on the Cortex-M4 microcontroller.

A primary challenge in simulated training is the gap between simulation and real data. To ground the simulation in a realistic environment and ensure its foundation is based on real-world data, the simulation parameters were calibrated using empirical data collected from the hardware testbed. While this logged data provided the statistical basis for the simulation, a static dataset is insufficient for training an RL agent, which must be able to actively explore the environment to learn an effective policy. The calibrated simulation provides a safe and scalable environment for this exploration, allowing the agent to learn from a wide variety of channel conditions, including rare events not captured in the limited hardware logs.

Fig. 7: The hardware testbed used for validation, showing the system running on a Raspberry Pi 4.

Five distinct test scenarios were conducted across multiple environments, including a personal residence and a research lab, with varying electrical loads on the PLC network. During these tests, the performance of the LX200V50 PLC modules and TI CC1352R RF modules was logged. The hardware testbed used for validation is shown in Fig. 7, with the PPO agent running on a Raspberry Pi 4. The aggregated, observed ranges from these hardware logs are summarized in Table 3.

Metric | PLC (LX200V50) | RF (TI CC1352R)
Latency (ms) | 0.5–1.5 | 10.0–25.0
Packet Loss (%) | 0.0–10.0 | 0.0–8.0
SNR (dB) | 3.0–12.55 | 50.0–90.0

The simulation parameters in the ‘PLCRFNetworkEnv‘ were then set to match these observed characteristics. For instance, ‘self.packet_loss_plc_max‘ was set to 0.10 (10%) and ‘self.latency_rf_mean‘ was set to 16.0, directly reflecting the empirical data from Table 3. This calibration ensures the agent is trained in an environment statistically representative of the target deployment and grounded in real hardware data.

To support reproducibility, Table 4 summarises the per-metric sampling distributions used in PLCRFNetworkEnv, with all parameters calibrated directly from the hardware-observed ranges in Table 3. The following qualitative assumptions also apply. The environment is memoryless: channel metrics are re-sampled independently at every time step, so no temporal correlation or fading memory is modeled. No explicit multipath fading, cyclostationary PLC noise, or co-channel RF interference model is applied; the empirical bounds on packet loss and SNR implicitly capture the aggregate effect of these impairments as measured across five hardware test scenarios at varying electrical loads. All nodes are static, consistent with the fixed-infrastructure AMI deployment scenario. The environment does not model packet queuing or a stochastic arrival process; each discrete time step represents one channel-selection decision, abstracting the packet-level transport layer. The environment is fully synchronous; at each step the agent observes the current 6-dimensional state vector, selects an action, and receives an instantaneous scalar reward with no propagation delay modeled. The state vector is min-max normalized to [0, 1] before being passed to the agent. Each episode consists of 10 sequential time steps; new channel conditions are independently re-sampled at every step and at each episode reset, ensuring statistically independent episodes. The global random seed is fixed to 42 for single-run evaluations and varied across ten distinct seeds for the multi-seed analysis, enabling exact reproducibility of all reported results. The minimum and maximum bounds of each sampling distribution in Table 4 directly correspond to the best-case and worst-case values recorded across the five hardware test scenarios described above, so the range of channel severity observed in practice, including comparatively poor conditions encountered during testing, is represented within the calibrated bounds even though the base environment does not model the temporal clustering of such conditions. Burst-type degradation is evaluated separately at network scale through the correlated impulsive noise mechanism described in “Simulation-based network scalability and stress testing” section , where the PPO policy Packet Delivery Ratio drops by less than one percentage point under bursty PLC noise relative to clean conditions, as reported in the stress-test results. Correlated RF fading specifically, however, has not been modeled at either the base environment or network-scale level, since RF channel samples remain independently generated at every step in both cases.

Metric | PLC | RF
Latency (ms) | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {N}(1.0,\,0.5^2)$$\end{document}, clipped [0.5, 1.5] | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {N}(16.0,\,3.0^2)$$\end{document}, clipped [10.0, 25.0]
Packet Loss | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {U}(0.0,\,0.10)$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {U}(0.0,\,0.08)$$\end{document}
SNR (dB) | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {U}(3.0,\,12.55)$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {N}(70.0,\,10.0^2)$$\end{document}, clipped [50.0, 90.0]

Algorithm implementation

To learn the optimal channel selection policy, four distinct reinforcement learning algorithms were implemented and compared. Each algorithm explores the simulated environment and attempts to learn a policy that maximizes a cumulative reward signal. The specific definition of this reward, which forms the core of the agent’s decision-making logic, is detailed in the following section. The implemented algorithms are as follows:

Q-Learning. The Q-Learning model follows a standard tabular approach, maintaining a Q-table to learn the action-value function Q(s,a). This function estimates the expected future reward for taking an action (e.g., choose PLC) from a given state (the current channel metrics). To use a tabular function with continuous inputs, the 6-dimensional state space is discretized into fixed bins, forming a unique, compact state representation. Action selection is governed by an ε-greedy strategy, a standard method to balance exploration (random actions) and exploitation (choosing the action with the highest Q-value)36. The learning rate α and discount factor γ control the influence of new information and the valuation of future rewards, respectively. This lightweight implementation serves as an interpretable baseline.

Double Deep Q-Network (DDQN). To mitigate the overestimation bias of standard Q-learning and handle the continuous state space directly, the Double Deep Q-Network (DDQN) algorithm was implemented. DDQN uses two neural networks: an online network for action selection and a target network for evaluating Q-values. The model takes the 6-dimensional state vector as input and outputs Q-values for each action. Its architecture consists of three fully connected hidden layers (256, 256, 128 neurons) with ReLU activation. Training utilizes a replay buffer and the Adam optimizer. The target Q-value is computed using the DDQN update rule:In (1), the target Q-value (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_{\text {target}}$$\end{document}) is calculated by adding the immediate reward (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_t$$\end{document}) to the discounted (γ) value of the best possible next action. Crucially, the online network (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_{\text {online}}$$\end{document}) selects the best next action (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arg \max a$$\end{document}), but the target network (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q{\text {target}}$$\end{document}) evaluates its value. This decoupling of selection and evaluation reduces overestimation bias.

1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q_{\text {target}}(s_t, a_t) = r_t + \gamma \cdot Q_{\text {target}}\left( s_{t+1}, \arg \max _a Q_{\text {online}}(s_{t+1}, a)\right) \end{aligned}$$\end{document}

Proximal Policy Optimization (PPO). PPO is a policy gradient method that uses an actor–critic framework, where the actor selects actions (the policy) and the critic estimates the value of those actions. It enhances stability by restricting policy updates using a clipped surrogate objective. Distinct feedforward networks were used for the actor and critic, both optimized with Adam. The actor outputs a categorical probability distribution over the discrete actions, and the critic estimates the state-value function to compute advantages using Generalized Advantage Estimation (GAE). The core training objective is the clipped surrogate loss:In (2), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_t(\theta )$$\end{document} is the probability ratio between the new and old policies, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{A}_t$$\end{document} is the estimated advantage (how much better an action was than the average). The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {clip}$$\end{document} function constrains this ratio within a small range (1 - ε to 1 + ε). By taking the \min of the normal ratio and the clipped ratio, PPO ensures that policy updates are not excessively large, which prevents unstable learning and improves convergence.

2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}_{\text {CLIP}}(\theta ) = \mathbb {E}_t \left[ \min \left( r_t(\theta ) \hat{A}_t, \; \text {clip}\left( r_t(\theta ), 1 - \epsilon , 1 + \epsilon \right) \hat{A}_t \right) \right] \end{aligned}$$\end{document}

Deep Deterministic Policy Gradient (DDPG). DDPG is an off-policy actor–critic method designed for continuous action spaces. It was adapted for this system’s binary action space by thresholding the actor’s continuous output. The actor network, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu (s \mid \theta ^{\mu })$$\end{document}, outputs a scalar value via a sigmoid function, which is thresholded at 0.5 to select action 0 (PLC) or 1 (RF). The critic network, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q(s, a \mid \theta ^{Q})$$\end{document}, evaluates the “goodness” of that state-action pair. The critic is optimized by minimizing the loss:In (3), the loss L is the mean squared error between the critic’s predicted Q-value (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q(s, a \mid \theta ^{Q})$$\end{document}) and a target Q-value (y). The target Q-value is calculated as:In (4), Q' and μ ' represent the stable target critic and actor networks, respectively, which are slow-moving copies of the main networks. This prevents the target value from changing too rapidly, thereby stabilizing learning. The actor is then updated using the deterministic policy gradient:This update in (5) guides the actor (μ) in a direction that maximizes the critic’s (Q) estimated Q-value. Ornstein–Uhlenbeck (OU) noise is added to the actor’s output during training to ensure sufficient exploration.

3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L = \mathbb {E}_{s, a, r, s'} \left[ \left( Q(s, a \mid \theta ^{Q}) - y \right) ^2 \right] \end{aligned}$$\end{document}

4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} y = r + \gamma Q'(s', \mu '(s' \mid \theta ^{\mu '}) \mid \theta ^{Q'}) \end{aligned}$$\end{document}

5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla _{\theta ^{\mu }} J \approx \mathbb {E}_{s} \left[ \nabla _{a} Q(s, a \mid \theta ^{Q}) \big |_{a = \mu (s)} \nabla _{\theta ^{\mu }} \mu (s \mid \theta ^{\mu }) \right] \end{aligned}$$\end{document}

In terms of computational complexity, Q-Learning is the lightest of the four algorithms: action selection and updates are O(1) table lookups, and its memory footprint scales only with the number of discretization bins per state dimension, reported as 4 to 6 bins per dimension in Table 5. DDQN, DDPG, and PPO all require a forward pass through feedforward networks of comparable size at inference time, dominated by the largest hidden layer reported in Table 5, so their per-decision inference cost is of the same order of magnitude, as confirmed for the PPO actor by the 162,523-clock-cycle, 1.93 ms measurement in Section Edge deployment feasibility and hardware profiling. At training time, PPO and DDPG each maintain two networks, an actor and a critic, roughly doubling the parameter count and memory footprint relative to DDQN’s single Q-network, while DDQN and DDPG additionally require experience replay buffers of 50,000 and 10,000 transitions respectively, reported in Table 5, which PPO’s on-policy training does not require.

Reward function design

The core of the agent’s decision-making logic is defined by the reward function, which quantitatively evaluates the quality of choosing either the PLC or RF channel at each step. This function is engineered to reflect the primary communication objectives of an AMI system. For each time step, a potential reward is computed for both the PLC and RF channels, and the agent’s goal is to learn a policy that selects the action maximizing this reward over time.The PLC reward is defined in (6) where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{PL}{\text {plc}}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L{\text {plc}}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{SNR}{\text {plc}}$$\end{document} represent the packet loss, latency, and signal-to-noise ratio on the PLC channel, respectively.While for the RF channel, the reward is calculated in (7) where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{PL}{\text {rf}}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\text {rf}}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{SNR}_{\text {rf}}$$\end{document} represent the same parameters for the RF channel.

6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {PLC}_{\text {Reward}} = 100 \cdot (1 - \textrm{PL}_{\text {plc}}) - 4.0 \cdot L_{\text {plc}} + 2.8 \cdot \textrm{SNR}_{\text {plc}} \end{aligned}$$\end{document}

7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {RF}_{\text {Reward}} = 100 \cdot (1 - \textrm{PL}_{\text {rf}}) - 0.1 \cdot L_{\text {rf}} + 0.1 \cdot \textrm{SNR}_{\text {rf}} \end{aligned}$$\end{document}

The weights in (6) and (7) were empirically tuned during simulation development. The design rationale is as follows. Data integrity is the primary objective, so packet loss (PL) is weighted most heavily, with a factor of 100 assigned to the success rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(1 - \textrm{PL})$$\end{document}. The asymmetric latency penalties reflect the differing numerical scales of the two channels rather than a judgment about which medium is inherently worse. As shown in Table 3, PLC latency falls in the range 0.5–1.5 ms, whereas RF latency spans 10–25 ms. Applying a weight of -4.0 to PLC latency yields absolute penalties of -2.0 to -6.0, while applying -0.1 to RF latency yields -1.0 to -2.5. This scaling ensures that both channels contribute comparable penalty magnitudes to the total reward, preventing the numerically larger RF latency values from dominating the reward signal. Similarly, the SNR weights (2.8 for PLC and 0.1 for RF) compensate for the large difference in absolute SNR ranges (3–12.55 dB for PLC versus 50–90 dB for RF), so that SNR improvements on either channel are reflected proportionally. All coefficients were determined through iterative tuning to produce a reward landscape in which the agent learns adaptive switching rather than a static preference, and the resulting policy is validated by the balanced channel selection behavior observed during testing. A formal sensitivity analysis examining the stability of the policy under variations of these coefficients is presented in “Reward function sensitivity and policy robustness analysis” section. The coefficients therefore follow a consistent mathematical principle: each weight is chosen such that the contribution of its associated metric to the total reward falls within the same order of magnitude as the dominant packet-loss term, preventing any single metric from being numerically suppressed. Concretely, the target contribution magnitude for each term is approximately 0–15 reward units; dividing this target range by the empirically observed metric range from Table 3 yields the coefficient values reported in (6) and (7). Regarding the use of automated search methods such as Bayesian optimization or grid search: these were considered, but since the coefficient magnitudes are directly derived from the hardware-observed metric ranges rather than from abstract hyperparameter space, the principal design freedom lies in selecting relative priorities among the three communication objectives, which was done intentionally based on AMI operational requirements. The sensitivity analysis in “Reward function sensitivity and policy robustness analysis” section serves as a systematic post-hoc validation, showing that the policy remains stable under \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm 25%$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm 50%$$\end{document} perturbations of all coefficients, confirming that the chosen values are not a fragile local optimum. Inverse reinforcement learning, which infers a reward function from expert demonstration trajectories, was also considered but is not directly applicable to this problem: as noted in “Methodological framework” section , the same absence of labelled, expert-annotated channel selection data that motivated the choice of reinforcement learning over supervised learning also precludes constructing the expert demonstration set that inverse reinforcement learning requires. Automated reward shaping techniques that adjust coefficients based on observed training performance, rather than inferring them from demonstrations, remain a promising alternative.

To further influence behavior, a switching penalty of -5 is applied whenever the agent changes the selected channel, discouraging unstable “flapping” between channels. A bonus of +15 is added if the selected channel has the higher computed reward (reinforcing the correct choice), and an additional +10 is granted if that chosen channel is RF. This RF bonus reflects the operational consideration that RF transmission, which utilizes low-power sub-GHz radios, generally incurs lower energy consumption per packet than PLC modulation over power lines6. The bonus does not override the primary reward components; rather, it acts as a tie-breaking preference when both channels offer comparable link quality, encouraging the agent to favor the more energy-efficient medium in such cases.

Training and hyperparameters

The main hyperparameters for each reinforcement learning model, listed in Table 6, were determined by adopting standard configurations validated in foundational RL literature36. For PPO specifically, an entropy coefficient of 0.1 is used to encourage exploration, and the clip ratio (ε) is set to 0.2 to safely bound policy updates, following established PPO recommendations. Q-Learning uses a higher learning rate (0.1) suited to tabular updates, while gradient-based methods, including PPO, DDQN, and DDPG, employ smaller rates for stable convergence. The discount factor γ is set near 0.98–0.99 across all models to emphasize long-term rewards. Exploration differs by algorithm: PPO relies on policy entropy and stochastic sampling, Q-Learning and DDQN use ε-greedy (with ε decaying from 0.5 to 0.01), and DDPG applies Ornstein–Uhlenbeck (OU) noise (θ =0.15, σ =0.3). Neural networks are used in PPO, DDQN, and DDPG for actor–critic or Q-value estimation, whereas Q-Learning employs a Q-table. Replay buffers (capacity 50,000 for DDQN and 10,000 for DDPG) and target networks are incorporated in DDQN and DDPG to further stabilize learning. Table 5 provides the neural network architectures and parameter specifications for the deep RL models. All models were implemented in TensorFlow 2.x using Python 3.10. Each model was trained with a fixed random seed of 42 to ensure reproducibility. Episodes terminate after 10 time steps (i.e., 10 channel-selection decisions per episode). Training was conducted for 700 episodes (PPO, DDPG, Q-Learning) and 1,000 episodes (DDQN), with convergence observed within the first 200–300 episodes for all models based on reward stabilization.

Parameter | PPO | DDQN | DDPG | Q-Learning
Architecture | Actor–Critic | Single Q-Network | Actor–Critic | Q-Table
Hidden layers | 256–128 (each net) | 256–256–128 | 256–128 / 256–256 | –
Activation | Swish + LayerNorm | ReLU | Swish + LayerNorm | –
Dropout | 0.2 | – | 0.2 | –
Batch size | 64 | 64 | 64 | –
Replay buffer | – | 50,000 | 10,000 | –
Entropy Coeff. | 0.1 | – | 0.3 | –
Clip ratio | 0.2 | – | – | –
Target update | — | Every 500 steps | Soft (τ=0.005) | –
State Bins | – | – | – | 4–6 per dim.

Hyperparameter | PPO | DDQN | Q-Learning | DDPG
Learning Rate | 0.0001 | 0.0005 | 0.1 | 0.0001/0.002
Discount Factor ( γ ) | 0.98 | 0.98 | 0.98 | 0.99
Exploration strategy | Entropy sampling | ε-greedy | ε-greedy | OU noise
Training episodes | 700 | 1000 | 700 | 700
Target update | – | Every 500 steps | – | Soft update
Optimizer | Adam | Adam | – | Adam

The reinforcement learning system operates in two phases: training and testing. During training, algorithms interact with the simulated environment to learn channel selection policies from latency, SNR, and packet loss feedback. In testing, the learned policies are evaluated on unseen real-life experimentation data from the hardware system to assess generalization. The best-performing model was then validated for real-time inference on a Raspberry Pi 4 testbed.

Testing and evaluation

It is important to note that the classification metrics reported in this work (accuracy, F1-score, and AUC-ROC) are based on a reward-derived oracle: at each decision step, the channel yielding the higher instantaneous reward is designated as the correct label. These metrics therefore measure agreement with the reward formulation rather than an independently labelled ground truth. Network metrics (latency, packet loss, and SNR of the selected channel) are reported separately in Table 9 to corroborate the policy’s real-world communication performance which is independent from the agent’s reward based-evaluation. This evaluation methodology is consistent with standard RL benchmarking practice, where the learned policy is assessed against the optimal oracle policy that has full knowledge of instantaneous rewards for both actions. The critical distinction is that the 5,000 test decisions are drawn from channel conditions generated independently of the training episodes, so the agent is evaluated on genuinely unseen states rather than on memorized training transitions. Beyond the oracle-based metrics, the reward-independent network results in Table 9 provide a complementary and independently grounded assessment: PPO achieves the lowest average packet loss (4.07%) and competitive latency among all agents, confirming that the learned switching behaviour translates to real communication improvements that do not depend on the reward formulation. The hardware-logged statistics in Table 3 further anchor the simulation distributions to empirically observed PLC and RF characteristics, providing an additional layer of real-world grounding. Reinforcement learning is increasingly important for smart grid communication, though scalability, stability, and efficiency remain challenges. Performance was assessed using accuracy, F1 score, AUC-ROC, and cumulative reward, alongside reward-independent network-level metrics (average latency, packet loss, and SNR of the chosen channel) reported in Table 9. The ground truth label for each of the 5,000 test decisions is defined as follows: at each time step, the reward for both the PLC and RF channels is computed independently using Equations (6) and (7). The channel yielding the higher reward is designated as the “correct” (optimal) action. This labeling mechanism is applied identically during both training and testing; however, the testing environment generates channel conditions from the same calibrated distributions independently of any training data, ensuring that the test set is unseen. This evaluation approach is consistent with standard RL benchmarking practice, where the learned policy is assessed against an oracle that has access to the instantaneous reward of both actions. To further validate performance beyond the reward function, Table 9 reports reward-independent network metrics (average latency, packet loss, and SNR) of the channels actually selected by each agent, confirming that the PPO policy achieves the lowest average packet loss (4.07%) while maintaining competitive latency.

To contextualize the RL results against non-learning baselines, Table 7 compares PPO with three trivial deterministic policies evaluated on the same 5,000-decision test set. The “Always-PLC” policy achieves a relatively high reward (1240.8) because PLC is frequently the optimal channel under the simulated conditions, but it incurs the highest average packet loss (4.96%). The “Always-RF” policy yields lower packet loss (3.89%) owing to the RF channel’s lower loss rates, but its total reward is the lowest (1127.2) because it forfeits the correct-choice bonus whenever PLC is optimal. The “Random” policy falls between the two. PPO achieves the highest reward (1282.9) by adaptively selecting the optimal channel, confirming that the learned switching strategy materially outperforms all static alternatives.

Policy | Avg. reward | Avg. loss (%)
Always-PLC | 1240.8 | 4.96
Always-RF | 1127.2 | 3.89
Random (50/50) | 1161.5 | 4.49
PPO (This Work) | 1282.9 | 4.07

Among the evaluated models, PPO consistently achieved the best overall performance, maintaining accuracy above 93% and an F1 score above 87%. In contrast, Q-Learning reached only about 70% accuracy with a very low F1 score around 25%, indicating limited effectiveness despite its stability. DDQN and DDPG achieved higher accuracy than Q-Learning but suffered from instability, with fluctuating accuracy and erratic F1 scores. All models demonstrated rapid initial learning within the first 50 episodes, quickly increasing their cumulative rewards before stabilizing at high values. While PPO maintained the highest reward levels (around 1250–1300), the other three models converged between 1220 and 1280, showing that each was capable of achieving strong cumulative returns, though with varying stability and reliability.

Fig. 8: Testing accuracy across different models.

Fig. 9: Testing F1 score across different models.

Fig. 10: Testing rewards across different models.

Fig. 11: Testing AUC-ROC curve across different models.

The testing phase assessed each model’s generalization by evaluating the trained agents over 500 test episodes. Since each episode in the environment consists of 10 time steps (i.e., 10 packet-level decisions), this evaluation comprised a total of 5,000 channel selections. To measure pure performance, exploratory mechanisms the random behaviors agents use during training to discover new strategies were disabled. Instead, agents selected actions deterministically to purely exploit their learned policies. This means models like DDQN and Q-Learning, which use an ε-greedy strategy for exploration (taking random actions with probability ε), were set to ε =0, forcing them to always choose the action with the highest predicted Q-value. Similarly, the PPO agent, which normally samples from an action probability distribution, was set to deterministically select the action with the highest probability (the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arg \max$$\end{document}). As illustrated in Fig. 8, PPO achieved the highest accuracy, consistently above 90% with minimal variation. DDQN reached high peaks but suffered sharp drops, suggesting instability or overfitting. DDPG maintained stable accuracy in the 75–85% range, while Q-Learning performed the weakest, fluctuating between 60–75%. These results highlight clear differences in adaptability across the models.

F1 scores in Fig. 9 followed similar patterns to the accuracy results. PPO and DDQN achieved the strongest performance, maintaining stable values between 70% and 90%, indicating balanced precision and recall under varying conditions. In contrast, DDPG rarely exceeded 50%, and Q-Learning remained below 30%, showing limited ability to generalize effectively.

Figure 10 compares the models’ testing rewards during this evaluation. All models quickly adapted within the first 20 episodes, after which rewards stabilized at high levels. PPO and DDQN achieved the strongest performance, consistently maintaining rewards between 1250 and 1300. DDPG was slightly lower but still effective, while Q-Learning showed the weakest and most erratic behavior, with rewards occasionally dropping below 1200. These results highlight how policy design influences robustness under unfamiliar conditions.

Figure 11 shows the testing AUC-ROC curves of the four models. PPO achieved the best performance with an AUC of 0.98, followed by DDQN at 0.91. DDPG reached 0.65, while Q-Learning performed poorly at 0.52, close to random guessing. These results highlight the superior generalization of PPO and DDQN in distinguishing between classes under unseen conditions. Overall, these results provide a comparative view of the four reinforcement learning models, highlighting their learning dynamics, reward optimization, and generalization behavior.

Model | Precision (%) | Recall (%) | F1 score (%) | Accuracy (%) | AUC-ROC | Average reward
PPO | 86.68 | 88.83 | 87.74 | 93.78 | 0.98 | 1287.19
DDQN | 73.54 | 92.50 | 82.53 | 89.76 | 0.91 | 1278.39
Q-Learning | 32.88 | 19.39 | 24.40 | 69.88 | 0.52 | 1220.68
DDPG | 92.12 | 30.23 | 45.34 | 81.48 | 0.65 | 1257.66

Model | Latency (ms) | Loss (%) | SNR (dB)
PPO | 4.94 | 4.07 | 25.07
DDQN | 6.23 | 4.09 | 31.55
Q-Learning | 3.07 | 4.87 | 16.46
DDPG | 2.20 | 4.66 | 13.42

Table 8 summarizes model performance during testing. PPO achieved the highest single-run accuracy, F1 score, and reward, confirming its reliable generalization and stable convergence; note that this advantage over DDQN is not statistically significant in multi-seed evaluation (Table 10), where PPO’s primary benefit is its substantially lower variance rather than a higher mean. DDQN also performed well but showed a weaker balance between precision and recall. DDPG delivered moderate results overall, yet demonstrated adaptability with improved precision and reward under new conditions. Q-Learning consistently lagged, with low F1 and AUC scores indicating poor policy learning and weak generalization. The ROC-AUC results reinforce these trends, with PPO approaching perfect class separability, DDQN and DDPG showing competitive scores, while Q-Learning close to random. Overall, PPO proved most data-efficient and stable, while DDPG showed potential as an alternative in novel environments.

Table 9 links the models’ policies to the resulting network performance, addressing the need to connect learning outcomes with actual network behavior. These metrics represent the average conditions of the channels chosen by each agent during the 500-episode test. The results reveal a clear trade-off heavily relevant to real-time AMI requirements. DDPG and Q-Learning, for example, achieved the lowest average latencies (2.20 ms and 3.07 ms, respectively) by developing policies that overwhelmingly favored the lower-latency channel, but this was achieved at the severe cost of high packet loss, yielding the poor F1 scores observed in Table 8. In contrast, PPO’s policy achieved the lowest average packet loss (4.07%) but experienced a slightly higher average latency (4.94 ms) than DDPG. Given that smart metering applications prioritize high reliability (often requiring >99% delivery rates) over saving a few milliseconds in transmission, PPO represents the optimal trade-off compared to the unreliable edge chosen by DDPG, as packet retransmissions required by higher loss would drastically inflate overall end-to-end latency. Furthermore, DDQN’s policy selected links with the highest average SNR (31.55 dB) but lacked the balanced consistency of PPO, demonstrating that PPO’s overall policy which prioritizes data integrity (low loss) while managing other metrics results in the most effective channel selection. The three reported metrics serve as proxies for the broader set of AMI communication performance indicators. SNR is a direct predictor of BER via the link-specific modulation-and-coding sensitivity curve; the reported SNR values indicate that both PLC and RF links operate well within their reliable communication range. Packet loss is a first-order surrogate for retransmission overhead, effective goodput, and outage probability: a 4.07% packet loss rate implies a corresponding retransmission load and a goodput reduction relative to raw throughput, both of which scale linearly with the loss fraction. Per-step latency variance across episodes implicitly captures jitter, and the simulation-based network-scale evaluation in “Simulation-based network scalability and stress testing” section reports PDR across 500–5,000 nodes, which is directly proportional to channel utilization and effective throughput at the network level. Energy consumption is partially addressed through the RF-preference bonus in the reward function, which systematically favors the lower-power RF medium under comparable link quality, and through the Cortex-M4 edge inference power profile. A comprehensive measurement of throughput, goodput, BER, jitter, outage probability, and channel utilization from physical hardware would require packet-level instrumentation beyond what the current testbed exposes.

Regarding failure cases specifically, the 6.22 percent of test decisions in which the deployed PPO policy selects a channel other than the reward-optimal one, implied by the 93.78 percent accuracy reported in Table 8, represent single-step misclassifications rather than systemic misbehaviour, since each channel-selection decision is generated independently, as described in “Simulated training environment” section. In the worst case, such a misclassification incurs a communication overhead bounded by the calibrated PLC and RF ranges in Table 3: selecting RF when PLC was optimal can add up to approximately 24.5 ms of latency relative to the fastest PLC link, while selecting PLC when RF was optimal can add up to approximately 10 percentage points of packet loss relative to the most reliable RF link. Because misclassifications are not correlated across consecutive packets in the present memoryless environment, their aggregate effect on end-to-end reliability remains bounded by these per-decision ranges rather than compounding across a session, which is consistent with the low average packet loss of 4.07 percent reported for the PPO policy in Table 9 despite the non-zero misclassification rate.

Statistical significance and multi-seed validation

To validate the robustness and consistency of the reinforcement learning models against random initialization and environment variations, a multi-seed evaluation was performed. Each model was trained and tested across ten distinct random seeds ranging from 43 to 52. The aggregated performance metrics, including the mean and standard deviation for each model, are summarized in Table 10. These standard deviations, together with the Welch’s t-test results in Table 11, already convey the variability and statistical confidence associated with each mean performance value; for a sample of ten seeds, an approximate 95 percent confidence interval for any reported mean can be obtained as the mean plus or minus 1.96 times the reported standard deviation divided by the square root of ten. The training convergence curves representing the mean training accuracy and the standard deviation (indicated by the shaded area) across the ten seeds are shown in Fig. 12. While the initial learning rates are comparable across the algorithms, the PPO model converges to a highly stable policy with minimal standard deviation. In contrast, the DDQN model exhibits substantial performance fluctuations during training, reflecting instability in the offline target update mechanism. The statistical distribution of the testing accuracy, F1 score, and average testing reward is presented in Fig. 13. Although the mean testing accuracy of the PPO model (75.13%) is close to that of the DDQN model (76.49%), the standard deviation of PPO is significantly lower (8.74% compared to 18.26% for DDQN). Similarly, PPO achieves a much lower standard deviation in F1 score (7.41% compared to 22.15% for DDQN) and testing reward (20.87 compared to 41.98 for DDQN). This comparison demonstrates that the PPO policy is substantially more stable and robust to seed-induced variance than the DDQN policy. DDPG demonstrates a relatively narrow accuracy variance but fails to learn a balanced policy, as evidenced by a low mean F1 score (13.03%). Q-Learning exhibits low variance but converges to a suboptimal policy, achieving only 67.14% accuracy.

Figure 14 displays the mean Receiver Operating Characteristic (ROC) curves overlaid with their respective mean AUC values and standard deviations. The PPO model achieves the highest mean AUC of 0.904 with a low standard deviation of 0.048, notably higher than DDQN (0.788 ± 0.136), DDPG (0.548 ± 0.115), and Q-Learning (0.521 ± 0.013). This confirms the superior generalization capability and classification robustness of PPO under unseen network conditions.

To evaluate whether the performance differences between PPO and the other models are statistically significant, Welch’s t-test (which assumes unequal variances) was conducted. The computed p-values for testing accuracy, F1 score, and average reward are reported in Table 11. The results indicate that the performance improvements of PPO over Q-Learning are statistically highly significant across all metrics (p < 0.05). The difference in F1 score between PPO and DDPG is also highly significant (p < 0.001). While the mean performance metrics of PPO and DDQN are similar (p > 0.05), the primary advantage of PPO is its significantly lower variance and training stability, which is essential for deployment in operational smart metering environments. It is standard practice in reinforcement learning deployments to select the best-performing initialization (Seed 42 in this study) for actual edge inference. Consequently, while the deployed Seed 42 model achieves a peak accuracy of 93.78%, the multi-seed evaluation confirms that PPO provides the safest, lowest-variance training trajectory among the tested algorithms, eliminating the severe policy collapse risks associated with DDQN’s ±18.26% accuracy swing.

From a communication systems perspective, rather than a purely optimization based one, PPO’s advantage stems from the specific structure of the channel selection reward landscape rather than from generic optimizer properties alone. As established in “Simulated training environment” section, channel conditions are re-sampled independently at every packet level decision, so the training signal is inherently high variance from one step to the next, and the switching penalty and correct choice bonus in the reward function, defined in “Reward function design” section, create a sharp decision boundary near the point where the PLC and RF reward values cross. PPO’s clipped surrogate objective explicitly bounds the magnitude of each policy update, which limits the influence that any single noisy or anomalous channel sample, such as an isolated impulsive noise event or an atypically weak RF reading, can have on the learned policy, directly addressing this channel level volatility rather than only stabilising an abstract optimisation trajectory. This same volatility offers a communication systems explanation for the weaknesses of the other three baselines. Q-Learning discretises the six dimensional state into fixed bins, which coarsens exactly the narrow crossover region near the PLC to RF reward boundary where correct channel selection matters most, consistent with its low F1 score reported in Table 8. DDPG commits to a deterministic, thresholded actor output, which is sensitive to the differing numerical scales of the PLC and RF metrics discussed in “Reward function design” section, and the sensitivity analysis in “Reward function sensitivity and policy robustness analysis” section shows that this determinism causes the DDPG policy to collapse onto a single medium under coefficient perturbation rather than adapting its decision boundary. PPO instead retains a stochastic, entropy regularised action distribution during training, which naturally represents the genuine ambiguity present when PLC and RF channel quality are simultaneously comparable, rather than forcing a premature deterministic commitment. Taken together, these properties indicate that PPO’s benefit arises from a good match between its update mechanism and the volatile, boundary sensitive nature of per packet hybrid channel selection, rather than solely from faster or more stable numerical convergence in the abstract.

Model | Precision (%) | Recall (%) | F1 Score (%) | Accuracy (%) | AUC-ROC | Average Reward
PPO | 55.74 ± 13.01 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {84.25} \pm 18.61$$\end{document} | 63.94 ± 7.41 | 75.13 ± 8.74 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.904} \pm 0.048$$\end{document} | 1241.10 ± 20.87
DDQN | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {66.94} \pm 24.18$$\end{document} | 83.72 ± 27.23 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {65.43} \pm 22.15$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {76.49} \pm 18.26$$\end{document} | 0.788 ± 0.136 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1247.19} \pm 41.98$$\end{document}
Q-Learning | 30.80 ± 2.60 | 20.44 ± 1.89 | 24.54 ± 2.01 | 67.14 ± 1.54 | 0.521 ± 0.013 | 1210.69 ± 4.42
DDPG | 37.45 ± 27.53 | 10.62 ± 23.64 | 13.03 ± 25.82 | 76.20 ± 5.74 | 0.548 ± 0.115 | 1243.30 ± 13.83

Model comparison | Accuracy | F1 score | Average reward
PPO vs DDQN | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$8.43 \times 10^{-1}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$8.52 \times 10^{-1}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.03 \times 10^{-1}$$\end{document}
PPO vs DDPG | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.64 \times 10^{-1}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.70 \times 10^{-4}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.95 \times 10^{-1}$$\end{document}
PPO vs Q-Learning | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {2.30 \times 10^{-2}}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1.89 \times 10^{-8}}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1.70 \times 10^{-3}}$$\end{document}

Fig. 12: Multi-seed training accuracy comparison with shaded standard deviation shading.

Fig. 13: Multi-seed testing performance with Welch’s t-test significance bracket.

Fig. 14: Multi-seed testing mean receiver operating characteristic (ROC) curves.

Edge deployment feasibility and hardware profiling

To evaluate the feasibility of deploying the proposed PPO actor model directly on resource-constrained edge hardware, the network’s execution characteristics are profiled against the architecture of the target STM32F401RCT6 microcontroller (84 MHz Arm Cortex-M4, 256 KB Flash, 64 KB SRAM) embedded within the smart meter nodes. The pre-trained PPO actor network features a fully-connected topology with approximately 35,000 parameters. To satisfy strict embedded limits, the baseline 32-bit floating-point (FP32) model was compressed using post-training full integer quantization (INT8) via the TensorFlow Lite (TFLite) Micro runtime framework37. Evaluation of 16-bit floating-point (FP16) quantization was also attempted. However, the target STM32F4 series features a standard Arm Cortex-M4F core whose hardware Floating-Point Unit (FPU) natively accelerates only 32-bit single-precision floating-point operations. Because it lacks native hardware support for half-precision floating-point arithmetic, the compilation process in the ST Edge AI environment triggers an explicit internal support fault due to unmappable hardware data types. Consequently, while functional software accuracy can be evaluated for the FP16 format, it remains unsupported for target hardware deployment.

To rigorously substantiate edge compatibility without physical hardware dependency, profiling for the supported variants was executed using the ST Edge AI Cloud38 infrastructure on a NUCLEO-F401RE39 development platform utilizing Arm Cortex-M4 architectural simulation engines. Table 12 details the resulting structural footprint, operational performance, and target-specific resource utilization. The optimization process significantly compresses the network’s footprint. The baseline FP32 model occupies 153.0 KB of Flash memory. Under full INT8 quantization, the serialized model footprint drops to 64.7 KB, achieving a 2.36× compression ratio for the total binary size while preserving over 99.9% of the baseline operational accuracy (93.74% vs. 93.78%) and F1 score (87.71% vs. 87.74%). Crucially, the 64.7 KB optimized binary resides entirely in the 256 KB non-volatile Flash, consuming only 25.2% of the available allocation. Runtime memory characterization via the Arm Cortex-M4 profile confirms that the peak SRAM consumption, including the intermediate activation buffers (1, 288 bytes) and the runtime engine overhead, is strictly restricted to 6.8 KB for the INT8 model, leaving over 89% of the 64 KB SRAM completely unencumbered for core meter routines. Furthermore, leveraging the Cortex-M4 hardware DSP extensions (such as the SMLAD instruction for parallelized multiply-accumulate operations), the actual on-device execution requires only 162,523 clock cycles, yielding a deterministic inference latency of 1.93 ms.

To clarify the scope of this figure, the reported 1.93 ms latency corresponds specifically to the forward pass of the compiled neural network operator graph, as measured by the ST Edge AI Cloud profiling tool described above, and does not include the time required to acquire the six raw channel metrics from the PLC and RF modules, the min-max normalization preprocessing step described in “Simulated training environment” section, or any communication-layer overhead associated with subsequently transmitting data over the selected channel. The normalization step consists of six scalar subtract-and-divide operations and is expected to contribute a negligible amount of additional latency relative to the 162,523 clock cycles required by the approximately 35,000-parameter network itself. Feature acquisition and post-decision transmission timing are governed by the underlying PLC and RF communication stacks rather than by the AI inference pipeline profiled in this section, and are not included in the 1.93 ms figure.

Model Variant | Precision | Size (KB) | Compression | Peak SRAM (KB) | Clock Cycles | F1 Score (%) | Accuracy (%) | AUC-ROC | Latency (ms)
FP32 (Baseline) | 32-bit Float | 153.0 | 1.0× | 5.8 | 421,083 | 87.74 | 93.78 | 0.98 | 5.01
INT8 (Optimized) | 8-bit Integer | 64.7 | 2.36× | 6.8 | 162,523 | 87.71 | 93.74 | 0.98 | 1.93
FP16 | 16-bit Float | 77.9 | 1.96× | N/A | N/A | 87.74 | 93.78 | 0.98 | N/A

To quantitatively address practical energy considerations, the worst-case peak power consumption was calculated using the component datasheets. The STM32 microcontroller draws a maximum of 453.6 mW (126 mA at 3.6 V) when running at its peak 84 MHz frequency under typical high-load operation. The CC1352R RF module, when transmitting at 915 MHz with a 10 dBm TX power setting, consumes 13.5 mA at 3.6 V, totaling 48.6 mW. Conversely, while the LX200V50 evaluation board documentation does not explicitly specify maximum power consumption, it utilizes the Qualcomm QCA7550 chipset. A commercial device employing this identical chipset (the MikroTik PWR-LINE PRO) reports a maximum power consumption of 5 W (5,000 mW) without attachments40. Consequently, the combined worst-case peak power for an RF transmission is approximately 502.2 mW, whereas a PLC transmission peaks significantly higher at roughly 5,453.6 mW. The minimal energy consumed by the microcontroller to execute the 1.93 ms INT8 inference is vastly outweighed by the substantial power savings achieved whenever the intelligent agent selects the energy-efficient RF interface over the PLC interface.

Beyond the model compression and latency results reported above, several additional considerations govern the long-term embedded viability of the proposed PPO agent. Regarding non-volatile storage, the 64.7 KB INT8 model resides in the same internal Flash memory bank as the STM32F401RCT6 application firmware and is therefore only rewritten when the deployed policy itself is updated, not during routine inference, which only reads the stored weights. Because a smart meter is expected to receive at most a small number of firmware or model revisions over its multi-year operational lifetime, the resulting Flash write count remains far below the endurance limits typical of embedded Flash technology in this microcontroller family, leaving substantial margin for periodic policy updates. Regarding memory management, the TensorFlow Lite Micro runtime allocates a single fixed-size static tensor arena at initialization rather than performing dynamic heap allocation during inference, so the reported peak SRAM occupancy of 6.8 KB in Table 12 is a deterministic, non-fragmenting allocation that persists unchanged across the device operational lifetime, avoiding the heap fragmentation risks associated with dynamic memory management on constrained microcontrollers. Regarding energy, combining the measured 1.93 ms INT8 inference latency with the STM32 datasheet peak power draw of 453.6 mW reported above yields an estimated worst case energy cost of approximately 0.88 mJ per channel selection decision, a negligible addition compared to the energy expended by a single PLC or RF packet transmission. Regarding thermal behaviour, the inference computation is a brief, event triggered operation invoked once per channel selection decision rather than a continuously running workload, so its duty cycle and consequent self heating contribution are expected to be small relative to the RF and PLC transceiver circuitry, though full thermal characterization under sustained field operating conditions has not been performed. Regarding inference scheduling, the PPO agent forward pass is invoked synchronously within the existing communication control loop immediately before each transmission decision, rather than on a fixed high frequency timer, so it does not contend with the communication stack interrupt driven radio and PLC modem handling for Cortex-M4 core time. Finally, regarding model update mechanisms, the current prototype uses a fixed, offline trained policy flashed at deployment time. A full over the air update pipeline, including versioned model delivery over the existing PLC or RF backhaul, on device integrity verification, and rollback on failed validation, has not been implemented in the present system.

Reward function sensitivity and policy robustness analysis

To evaluate the stability and robustness of the learned policies under variations of the reward function’s heuristic parameters, a one-factor-at-a-time (OAT) sensitivity analysis was performed. The manual weights for PLC latency (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {PLC}}$$\end{document}), RF latency (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {RF}}$$\end{document}), RF selection bonus (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {RF}}$$\end{document}), and switching penalty (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {switch}}$$\end{document}) were systematically perturbed by -50%, -25%, +25%, and +50% from their calibrated baseline values. These baseline configurations correspond to the primary coefficient values established in the reward function design in Section Reward function design. For each parameter configuration, the four reinforcement learning agents (PPO, DDQN, DDPG, and Q-Learning) were retrained from scratch and evaluated over 500 test episodes using the calibrated network simulator. Table 13 reports the average testing accuracy, F1 score, and cumulative reward for all four models under each perturbation.

Factor | Variation | PPO | DDQN | DDPG | Q-Learning
 |  | Acc (%) | F1 (%) | Reward | Acc (%) | F1 (%) | Reward | Acc (%) | F1 (%) | Reward | Acc (%) | F1 (%) | Reward
Baseline | 0% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {93.78}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {87.74}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1287.19}$$\end{document} | 89.76 | 82.53 | 1278.39 | 81.48 | 45.34 | 1257.66 | 69.88 | 24.40 | 1220.68
PLC Latency Wt (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {PLC}}$$\end{document}) | -50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {88.30}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {75.22}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1286.24}$$\end{document} | 87.50 | 55.00 | 1285.31 | 80.66 | 2.03 | 1268.39 | 74.42 | 22.06 | 1243.66
-25% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {90.00}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {79.59}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1284.05}$$\end{document} | 83.52 | 72.29 | 1270.51 | 77.32 | 1.90 | 1253.49 | 72.22 | 23.56 | 1232.24
+25% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {92.68}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {87.53}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1279.59}$$\end{document} | 86.48 | 80.73 | 1267.94 | 70.60 | 2.00 | 1223.66 | 67.68 | 25.60 | 1209.52
+50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {88.68}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {83.17}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1266.60}$$\end{document} | 80.32 | 76.69 | 1252.23 | 67.32 | 1.80 | 1208.83 | 64.94 | 26.56 | 1197.30
RF Latency Wt (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {RF}}$$\end{document}) | -50% | 84.96 | 78.99 | 1274.98 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {94.26}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {90.21}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1294.31}$$\end{document} | 69.50 | 1.93 | 1232.05 | 66.76 | 26.07 | 1218.86
-25% | 90.20 | 83.42 | 1282.30 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {94.02}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {89.18}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1290.82}$$\end{document} | 71.66 | 1.94 | 1235.21 | 68.22 | 25.22 | 1219.59
+25% | 86.82 | 75.82 | 1268.28 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {92.16}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {84.52}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1280.65}$$\end{document} | 76.28 | 1.82 | 1241.95 | 71.46 | 23.97 | 1221.80
+50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {90.20}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {78.24}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1272.96}$$\end{document} | 88.76 | 76.64 | 1269.34 | 78.50 | 2.01 | 1245.22 | 73.00 | 23.38 | 1222.72
RF Bonus (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {RF}}$$\end{document}) | -50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {92.88}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {86.37}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1273.88}$$\end{document} | 91.84 | 80.97 | 1273.73 | 74.04 | 1.82 | 1238.55 | 69.88 | 24.32 | 1218.22
-25% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {91.42}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {84.53}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1276.47}$$\end{document} | 90.78 | 83.59 | 1274.92 | 74.04 | 1.82 | 1238.61 | 69.88 | 24.40 | 1219.46
+25% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {92.04}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {84.24}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1288.46}$$\end{document} | 87.08 | 79.49 | 1280.04 | 74.04 | 1.82 | 1238.73 | 70.04 | 25.25 | 1222.26
+50% | 88.14 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {80.59}$$\end{document} | 1287.71 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {90.80}$$\end{document} | 78.32 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1288.12}$$\end{document} | 74.04 | 1.82 | 1238.79 | 70.04 | 25.25 | 1223.52
Switch Penalty (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {switch}}$$\end{document}) | -50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {88.70}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {81.14}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1286.40}$$\end{document} | 83.08 | 74.32 | 1274.54 | 74.04 | 1.82 | 1239.09 | 69.88 | 24.47 | 1226.34
-25% | 92.26 | 85.53 | 1288.47 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {92.80}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {87.12}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1290.15}$$\end{document} | 74.04 | 1.82 | 1238.88 | 69.90 | 24.49 | 1223.55
+25% | 92.60 | 84.91 | 1280.35 | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {94.80}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {89.05}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1285.10}$$\end{document} | 74.04 | 1.82 | 1238.46 | 69.90 | 24.41 | 1217.93
+50% | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {91.66}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {84.57}$$\end{document} | \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {1273.20}$$\end{document} | 82.16 | 73.75 | 1252.46 | 74.04 | 1.82 | 1238.25 | 69.92 | 24.42 | 1215.15

The sensitivity analysis results demonstrate that the proposed PPO policy exhibits high stability and robustness against reward coefficient changes. Across most configurations, PPO maintains an accuracy above 88% and an F1 score above 75%, outperforming other models under almost all perturbations. Several key behavioral trends are observed. Under negative perturbations of the RF latency weight (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {RF}}$$\end{document}) by -50% and -25%, the DDQN policy achieves higher accuracy (94.26% and 94.02%, respectively) and F1 score (90.21% and 89.18%) compared to PPO. This indicates that DDQN benefits from a reduced penalty on RF latency, which allows the offline Q-network updates to optimize selection ratios under high-frequency switching. However, under positive perturbations of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{l,\text {RF}}$$\end{document}, DDQN’s performance degrades (dropping to 88.76% accuracy at +50%), whereas PPO maintains stable performance (90.20% accuracy).

In contrast, the DDPG policy shows extreme vulnerability to parameter variations. Under almost all non-baseline settings, the DDPG policy collapses, yielding an F1 score of approximately 1.8% to 2.0%. This collapse is attributed to the priority replay buffer implementation and deterministic continuous action clipping: variations in the reward scale shift the gradient steps outside stable bounds, causing the actor to converge to a static, single-channel policy. Similarly, Q-Learning exhibits low variability but converges to suboptimal policies, with the F1 score constrained within 22.06% to 26.56% across all configurations. These findings confirm that the PPO formulation is highly reliable for smart grid deployment, as the policy maintains near-optimal performance even under significant heuristic parameter changes, reducing the need for precise reward function calibration in variable field environments.