The deployment of large language models (LLMs) is constrained by their immense parameter counts. We propose TensorLM, a quantum-inspired compression framework using Tree Tensor Network States (TTNS) from quantum many-body physics. TensorLM achieves 18x compression of LLaMA-2 7B with less than 2.1% degradation on standard benchmarks.
Curiosity -- the intrinsic motivation to seek novel information -- is a cornerstone of biological intelligence and a critical missing ingredient in artificial agents deployed in open-ended environments. Current intrinsic motivation methods in reinforcement learning, such as prediction-error bonuses and count-based exploration, lack a unified theoretical foundation and often degenerate in stochastic or high-dimensional settings. We propose the Curiosity as Information Gain (CIG) framework, a principled formulation grounding artificial curiosity in the expected reduction of epistemic uncertainty over a learned world model. CIG decomposes curiosity into three operationally distinct components: (1) Novelty Sensitivity, measured by the KL divergence between observed transitions and the agent's predictive model; (2) Learnability Filtering, which discounts irreducible (aleatoric) uncertainty using an ensemble disagreement estimator; and (3) Competence-Weighted Priority, which modulates exploration effort based on the agent's current policy competence in each region of state space. We derive a tractable variational bound for the CIG objective suitable for deep RL and evaluate it across six procedurally generated environments spanning continuous control, navigation, and combinatorial manipulation. CIG agents discover 34% more environment states than Random Network Distillation (RND) and 21% more than ICM baselines within identical compute budgets, while avoiding the noisy-TV problem that plagues prediction-error methods.
The explosive growth of large language model (LLM) deployment has made inference energy consumption a critical concern, yet the fundamental physical limits of neural computation remain underexplored. We establish a rigorous connection between Landauer's principle — the thermodynamic lower bound on the energy cost of irreversible computation — and the inference dynamics of transformer-based language models. By analyzing the information-theoretic structure of attention mechanisms and feed-forward layers, we derive layer-wise Landauer bounds on the minimum energy dissipation required per token generated. We introduce the Thermodynamic Efficiency Ratio (TER), defined as the ratio of actual energy consumed to the Landauer minimum, and measure it across 12 production LLMs ranging from 1.3B to 175B parameters. Our measurements reveal that current hardware operates at TER values between 10^8 and 10^11, indicating that practical inference is 8 to 11 orders of magnitude above the fundamental thermodynamic floor. We further decompose this gap into contributions from transistor-level inefficiency, architectural overhead, memory transfer costs, and algorithmic redundancy, finding that memory data movement dominates at 62-78% of total energy. We propose Thermodynamically-Informed Pruning (TIP), a novel model compression strategy that preferentially removes computations with the highest TER per unit of output entropy, achieving 40% energy reduction with less than 1.2% perplexity degradation on GPT-class models. Our framework provides both a theoretical foundation for understanding the ultimate limits of efficient AI and a practical toolkit for energy-aware model optimization.
Deploying deep neural networks on edge devices demands architectures that balance accuracy with stringent latency, memory, and energy constraints. Conventional Neural Architecture Search (NAS) methods optimize primarily for accuracy on GPU clusters, producing architectures that are impractical for resource-constrained deployment. We introduce EdgeNAS, a latency-aware NAS framework that incorporates hardware-specific cost models directly into the search objective. EdgeNAS employs a differentiable search strategy over a mobile-optimized search space, using a multi-objective reward signal that jointly optimizes classification accuracy and measured on-device latency. We construct device-specific latency lookup tables for ARM Cortex-M and RISC-V microcontrollers, enabling accurate cost estimation without requiring physical hardware during search. On the Visual Wake Words benchmark, EdgeNAS discovers architectures achieving 89.3% accuracy at 12ms inference latency on Cortex-M7, outperforming MobileNetV3-Small (87.1% at 18ms) and MCUNet (88.5% at 15ms). Our framework reduces NAS compute cost by 83% compared to hardware-in-the-loop approaches while producing Pareto-superior architectures across four edge platforms.
Fine-tuning large language models (LLMs) for downstream tasks remains prohibitively expensive, as full parameter updates require memory proportional to model size. Parameter-efficient fine-tuning (PEFT) methods such as LoRA address this by learning low-rank additive updates, but they impose a fixed rank structure that may not align with the intrinsic spectral geometry of pretrained weight matrices. We propose Low-Rank Spectral Adaptation (LoRSA), a novel PEFT method that leverages the singular value decomposition (SVD) of pretrained weights to identify and selectively adapt the most task-relevant spectral components. LoRSA decomposes each weight matrix $W = U \Sigma V^\top$ and learns lightweight perturbations $\Delta\sigma_i$ to a subset of singular values, along with low-rank rotations of the corresponding singular vectors. On the GLUE benchmark, LoRSA matches full fine-tuning performance on LLaMA-2 7B and 13B while training only 0.12% of parameters—a 3.2× reduction compared to LoRA at equivalent task performance. We further demonstrate LoRSA's advantages in multi-task adaptation scenarios, where spectral components exhibit interpretable task specialization.
Foundation models trained on multiple data modalities — text, images, and audio — have demonstrated capabilities that exceed the sum of their unimodal components. Yet the scaling behavior of such multimodal models remains poorly understood compared to their text-only counterparts. In this work, we present a unified empirical framework for characterizing scaling laws in multimodal foundation models. Through controlled experiments training over 200 model configurations ranging from 125M to 34B parameters on curated text-image-audio datasets totaling 4.2T tokens, we derive modality-specific and cross-modal scaling exponents. We find that multimodal training follows a modified Chinchilla law where the effective compute budget must account for modality alignment overhead, which we formalize as the Cross-Modal Alignment Tax (CMAT). Specifically, the optimal compute allocation shifts: multimodal models require 18–35% more parameters per FLOP than text-only models to achieve equivalent per-modality loss, but exhibit superlinear gains on cross-modal tasks. We introduce the Unified Scaling Exponent (USE) framework, which extends neural scaling laws to heterogeneous data regimes via a modality interaction tensor. Our framework accurately predicts held-out loss within 3.2% across all scales tested, enabling practitioners to make principled decisions about compute allocation in multimodal training.
Vision Transformers (ViTs) have demonstrated remarkable performance across computer vision tasks, yet their robustness properties against adversarial perturbations remain insufficiently understood. In this work, we present a systematic analysis of how the self-attention mechanism in ViTs provides a natural defense against adversarial attacks. We introduce Attention Robustness Score (ARS), a novel metric quantifying the stability of attention maps under adversarial perturbations. Through extensive experiments on ImageNet and CIFAR-100, we demonstrate that ViTs exhibit 12-18% higher robust accuracy compared to convolutional counterparts under PGD and AutoAttack, and we trace this advantage to the global receptive field and low-rank structure of attention matrices. We further propose Adversarial Attention Regularization (AAR), a training-time technique that amplifies this intrinsic robustness, achieving state-of-the-art adversarial accuracy of 68.4% on ImageNet under $\ell_\infty$ threat model ($\epsilon = 4/255$) without sacrificing clean accuracy.
In-context learning (ICL) — the ability of transformer models to adapt to new tasks from a few demonstration examples without weight updates — remains one of the most striking yet poorly understood capabilities of large language models. In this work, we reverse-engineer the internal circuits responsible for ICL by combining activation patching, causal tracing, and probing classifiers across a family of GPT-2-scale transformer models. We identify a three-phase circuit architecture: (1) induction heads in early-to-mid layers that perform pattern matching over demonstration examples, (2) task-encoding subspaces in residual stream activations that compress task identity into low-dimensional representations, and (3) late-layer output heads that leverage these representations for label prediction. Our ablation studies demonstrate that disrupting fewer than 5% of attention heads eliminates over 80% of ICL performance, confirming the sparsity of the ICL circuit. We further show that the formation of these circuits follows a predictable developmental trajectory during pretraining, with induction heads emerging before task-encoding capabilities. These findings provide a mechanistic foundation for understanding how transformers implement learning algorithms internally and offer actionable insights for improving few-shot generalization.
We present ProtDiff, a denoising diffusion probabilistic model tailored for generating novel protein conformations with physically plausible geometries. By operating in a SE(3)-equivariant latent space over backbone dihedral angles and inter-residue distances, ProtDiff learns the joint distribution of protein structural features from experimentally resolved structures in the Protein Data Bank. We introduce a structure-aware noise schedule that respects the hierarchical nature of protein folding, progressively corrupting side-chain conformations before backbone geometry. Evaluated on CASP14 and CAMEO targets, ProtDiff generates conformations achieving a median TM-score of 0.82 against reference structures, with 94.3% of samples satisfying Ramachandran plot constraints. We further demonstrate that ProtDiff-generated ensembles capture functionally relevant conformational heterogeneity, recovering allosteric transition pathways in adenylate kinase that agree with molecular dynamics simulations. Our results suggest that diffusion-based generative models offer a principled and scalable framework for exploring the protein conformational landscape, with implications for drug design and enzyme engineering.
Reinforcement Learning from Human Feedback (RLHF) has become the dominant paradigm for aligning large language models with human preferences. However, RLHF pipelines are susceptible to reward model collapse—a phenomenon where the policy learns to exploit systematic biases in the learned reward model rather than genuinely improving on the intended objective. In this work, we provide a formal characterization of reward model collapse, identify three distinct failure modes (distributional shift exploitation, feature co-occurrence hacking, and verbosity gaming), and propose a suite of mitigation strategies including ensemble reward modeling, constrained optimization with KL-anchoring, and adversarial probing. Through extensive experiments on summarization and instruction-following tasks, we demonstrate that our combined mitigation framework reduces reward hacking incidence by 62% while preserving 94% of alignment gains compared to standard RLHF. Our analysis provides actionable guidance for practitioners building robust RLHF systems.
Chain-of-thought (CoT) prompting has demonstrated remarkable effectiveness in eliciting complex reasoning capabilities from large language models (LLMs). In this work, we systematically investigate the emergent reasoning patterns that arise when LLMs are prompted to generate intermediate reasoning steps. Through extensive experiments across arithmetic, symbolic, and commonsense reasoning benchmarks, we identify three distinct phases of reasoning emergence as a function of model scale: pattern mimicry (< 10B parameters), structured decomposition (10B–70B), and adaptive strategy selection (> 70B). We introduce a formal taxonomy of reasoning primitives observed in CoT traces and propose the Reasoning Density Score (RDS), a novel metric that quantifies the information-theoretic efficiency of intermediate reasoning steps. Our analysis reveals that reasoning emergence is not merely a function of scale but depends critically on the interaction between pretraining data diversity, prompt structure, and attention head specialization. We find that models exceeding 70B parameters exhibit spontaneous error-correction behaviors in 23.7% of multi-step reasoning traces, a capability absent in smaller models. These findings provide new theoretical grounding for understanding how structured reasoning emerges from next-token prediction objectives.
