Abstract

Deploying deep neural networks on edge devices demands architectures that balance accuracy with stringent latency, memory, and energy constraints. We introduce EdgeNAS, a latency-aware NAS framework that incorporates hardware-specific cost models directly into the search objective. EdgeNAS discovers architectures achieving 89.3% accuracy at 12ms inference latency on Cortex-M7, outperforming MobileNetV3-Small and MCUNet while reducing NAS compute cost by 83% compared to hardware-in-the-loop approaches.

1. Introduction

The proliferation of intelligent edge devices—from IoT sensors and wearables to autonomous micro-robots—has created urgent demand for neural network architectures that operate within severe resource budgets. A typical ARM Cortex-M7 microcontroller offers 512 KB flash memory, 256 KB SRAM, and operates at 480 MHz without floating-point acceleration beyond single-precision. Deploying even modest neural networks in this regime requires careful co-optimization of accuracy and hardware cost.

Neural Architecture Search (NAS) has revolutionized architecture design by automating the discovery of high-performing network topologies (Zoph & Le, 2017). However, standard NAS formulations optimize a single objective—typically validation accuracy—on powerful GPU hardware:

$$\alpha^* = \arg\max_{\alpha \in \mathcal{A}} \text{Acc}_{\text{val}}(w^*(\alpha), \alpha)$$

where $\alpha$ denotes the architecture parameters, $\mathcal{A}$ the search space, and $w^*(\alpha)$ the optimal weights. This formulation ignores deployment constraints entirely, often yielding architectures that exceed the memory or latency budget of target edge devices by orders of magnitude.

Recent work has incorporated hardware awareness into NAS (Cai et al., 2019; Wu et al., 2019), but these approaches typically target mobile phones or GPUs rather than microcontrollers (MCUs), and many require expensive hardware-in-the-loop evaluation during search. In this paper, we present EdgeNAS, a framework that addresses these limitations through three contributions:

1. A microcontroller-optimized search space incorporating depthwise separable convolutions, squeeze-and-excitation blocks, and quantization-friendly activation functions.
2. Device-specific latency lookup tables (LUTs) constructed via one-time profiling, enabling accurate latency estimation during search without physical hardware.
3. A differentiable multi-objective search strategy that navigates the accuracy-latency Pareto frontier efficiently.

2. Related Work

Neural Architecture Search. Early NAS methods relied on reinforcement learning (Zoph & Le, 2017) or evolutionary algorithms (Real et al., 2019), requiring thousands of GPU-hours. Differentiable approaches like DARTS (Liu et al., 2019) reduced search cost dramatically by relaxing the discrete search space to a continuous one, enabling gradient-based optimization. One-shot methods (Guo et al., 2020) further improved efficiency by training a supernet and extracting sub-networks.

Hardware-Aware NAS. MnasNet (Tan et al., 2019) pioneered the integration of latency into the NAS objective for mobile phones, using a weighted product $\text{Acc}(\alpha) \times [\text{LAT}(\alpha)/T]^w$ as the reward. ProxylessNAS (Cai et al., 2019) made the latency differentiable through latency prediction models. FBNet (Wu et al., 2019) used lookup tables for latency estimation. However, these works target mobile-class hardware (Snapdragon, Pixel) with orders of magnitude more compute than MCUs.

MCU-Targeted NAS. MCUNet (Lin et al., 2020) jointly optimized the neural architecture and inference library for microcontrollers, achieving state-of-the-art accuracy on MCU deployment. MicroNets (Banbury et al., 2021) established benchmarks for TinyML. Our work differs by providing a general framework applicable across MCU families with minimal per-device profiling cost.

3. Methodology

3.1 Search Space Design

Our search space $\mathcal{A}$ is structured as a sequential chain of $N = 7$ searchable blocks, each selecting from the following operations:

• Depthwise separable convolution with kernel sizes $k \in {3, 5, 7}$
• Inverted residual block (MobileNetV2-style) with expansion ratios $e \in {2, 4, 6}$
• Squeeze-and-excitation with reduction ratio $r \in {4, 8}$
• Skip connection (identity)
• Channel counts $c \in {16, 24, 32, 48, 64}$

All operations use ReLU6 activation (quantization-friendly) and support INT8 quantization. The total search space contains approximately $|\mathcal{A}| \approx 3.2 \times 10^{11}$ unique architectures.

3.2 Latency Lookup Tables

For each target device $d$, we construct a latency lookup table $\mathcal{T}_d$ by profiling each primitive operation $o_i$ across all feasible input dimensions:

$$\text{LAT}$$d(\alpha) = \sum{i=1}^{N} \mathcal{T}d(o_i, h_i, w_i, c{\text{in},i}, c_{\text{out},i})

where $h_i, w_i$ are spatial dimensions and $c_{\text{in},i}, c_{\text{out},i}$ are channel counts at block $i$. Profiling is performed once per device using the TFLite Micro interpreter, measuring wall-clock latency averaged over 1000 inferences. The complete LUT construction takes approximately 4 hours per device—a one-time cost amortized across all subsequent searches.

We validate LUT accuracy by comparing predicted versus measured end-to-end latency for 200 randomly sampled architectures:

3.3 Multi-Objective Differentiable Search

We formulate the search as a constrained optimization problem:

$$\max_{\alpha} \quad \text{Acc}$${\text{val}}(w^*(\alpha), \alpha)αmax​Accval​(w∗(α),α) $$\text{s.t.} \quad \text{LAT}$$d(\alpha) \leq T{\text{max}}, \quad \text{MEM}(\alpha) \leq M{\text{max}}

Following DARTS, we relax the discrete choices to a continuous mixture via softmax-weighted combinations. The latency constraint is incorporated as a differentiable penalty:

$$\mathcal{L}$${\text{total}} = \mathcal{L}{\text{CE}} + \mu \cdot \max\left(0, \frac{\text{LAT}d(\alpha)}{T{\text{max}}} - 1\right)^2 + \nu \cdot \max\left(0, \frac{\text{MEM}(\alpha)}{M_{\text{max}}} - 1\right)^2

where $\mu$ and $\nu$ are Lagrange multiplier-like penalty coefficients that we adaptively increase during search using an augmented Lagrangian schedule:

$$\mu_{t+1} = \mu_t \cdot \begin{cases} 1.5 & \text{if constraint violated at step } t \ 1.0 & \text{otherwise} \end{cases}$$

The architecture parameters $\alpha$ and network weights $w$ are optimized in alternating steps, with $w$ updated on the training set and $\alpha$ on a held-out validation set.

# Differentiable latency estimation during search
def estimate_latency(arch_params, lut, temperature=1.0):
    total_latency = 0.0
    for block_idx, params in enumerate(arch_params):
        probs = F.softmax(params / temperature, dim=-1)
        block_lat = sum(
            p * lut[block_idx][op_idx]
            for op_idx, p in enumerate(probs)
        )
        total_latency += block_lat
    return total_latency

4. Results

4.1 Visual Wake Words Benchmark

We evaluate on Visual Wake Words (VWW), a standard TinyML benchmark for person detection with binary classification. All models are quantized to INT8 and deployed using TFLite Micro.

EdgeNAS achieves the highest accuracy while simultaneously being the fastest and most memory-efficient. The discovered architecture features aggressive channel reduction in early layers (16→24→24 channels) with expansion only in later layers (→48→64), a pattern that standard NAS methods do not discover because they lack awareness of the MCU memory hierarchy.

4.2 Cross-Platform Generalization

Using device-specific LUTs, we run EdgeNAS targeting four different platforms and compare against the best manually-designed baselines for each:

4.3 Search Cost Analysis

EdgeNAS completes its search in 8 GPU-hours on a single V100, compared to 48 GPU-hours for MCUNet's two-stage search and 150+ GPU-hours for RL-based approaches with hardware-in-the-loop evaluation. The LUT-based latency estimation eliminates the need for physical hardware during search, enabling architecture exploration on cloud infrastructure.

The search cost breakdown is: supernet training (5 GPU-hours), architecture parameter optimization (2.5 GPU-hours), and final retraining of the discovered architecture (0.5 GPU-hours).

4.4 Ablation Studies

We ablate key design choices on Cortex-M7:

• Without latency penalty: Accuracy reaches 90.1% but latency balloons to 34ms (constraint violated)
• Without adaptive penalty scheduling: Architecture often gets stuck in infeasible regions, final accuracy 87.2% at 14ms
• With FLOP proxy instead of LUT: Accuracy 88.1% at 16ms; FLOP count is a poor proxy for MCU latency due to memory-bound operations
• With $K=3$ searchable blocks instead of 7: Accuracy drops to 85.8% due to reduced expressiveness

5. Discussion

Our results demonstrate that hardware-aware NAS with accurate device-specific cost models substantially outperforms both manual design and hardware-unaware NAS for edge deployment. The key insight is that FLOP count is a poor proxy for actual MCU latency—operations that are compute-efficient may be memory-bandwidth-bound on MCUs with limited cache. Our LUT approach captures these device-specific characteristics.

The discovered architectures reveal interesting patterns: EdgeNAS consistently prefers narrow early layers and wider later layers, contrary to the common wisdom of pyramidal widening. This reflects the MCU memory constraint—early layers process high spatial resolution and must be narrow to fit in SRAM, while later layers at reduced resolution can afford more channels.

A limitation is that our LUT approach assumes sequential execution; devices with hardware accelerators (NPUs) that support operation fusion would require graph-level profiling rather than operator-level LUTs.

6. Conclusion

We presented EdgeNAS, a latency-aware neural architecture search framework for edge deployment on microcontrollers. By combining device-specific latency lookup tables with differentiable multi-objective search, EdgeNAS discovers architectures that achieve state-of-the-art accuracy-latency tradeoffs across four MCU platforms while requiring only 8 GPU-hours of search compute. The framework is extensible to new devices with approximately 4 hours of one-time profiling. Our work demonstrates that principled hardware-cost integration into the NAS objective is essential for practical edge AI deployment and that proxy metrics like FLOPs are insufficient for MCU-class devices.

Future directions include extending EdgeNAS to support dynamic inference (early exit), multi-task architectures, and neural processing unit (NPU) accelerators with operation fusion capabilities.

References

1. Zoph, B. & Le, Q. V. (2017). Neural architecture search with reinforcement learning. ICLR.
2. Liu, H., Simonyan, K., & Yang, Y. (2019). DARTS: Differentiable architecture search. ICLR.
3. Cai, H., Zhu, L., & Han, S. (2019). ProxylessNAS: Direct neural architecture search on target task and hardware. ICLR.
4. Wu, B., et al. (2019). FBNet: Hardware-aware efficient ConvNet design via differentiable neural architecture search. CVPR.
5. Tan, M., et al. (2019). MnasNet: Platform-aware neural architecture search for mobile. CVPR.
6. Lin, J., et al. (2020). MCUNet: Tiny deep learning on IoT devices. NeurIPS.
7. Banbury, C., et al. (2021). MLPerf Tiny benchmark. NeurIPS Datasets and Benchmarks.
8. Real, E., et al. (2019). Regularized evolution for image classifier architecture search. AAAI.
9. Guo, Z., et al. (2020). Single path one-shot neural architecture search with uniform sampling. ECCV.
10. Howard, A., et al. (2019). Searching for MobileNetV3. ICCV.