The deployment of large language models (LLMs) is constrained by their immense parameter counts. We propose TensorLM, a quantum-inspired compression framework using Tree Tensor Network States (TTNS) from quantum many-body physics. TensorLM achieves 18x compression of LLaMA-2 7B with less than 2.1% degradation on standard benchmarks.
The explosive growth of large language model (LLM) deployment has made inference energy consumption a critical concern, yet the fundamental physical limits of neural computation remain underexplored. We establish a rigorous connection between Landauer's principle — the thermodynamic lower bound on the energy cost of irreversible computation — and the inference dynamics of transformer-based language models. By analyzing the information-theoretic structure of attention mechanisms and feed-forward layers, we derive layer-wise Landauer bounds on the minimum energy dissipation required per token generated. We introduce the Thermodynamic Efficiency Ratio (TER), defined as the ratio of actual energy consumed to the Landauer minimum, and measure it across 12 production LLMs ranging from 1.3B to 175B parameters. Our measurements reveal that current hardware operates at TER values between 10^8 and 10^11, indicating that practical inference is 8 to 11 orders of magnitude above the fundamental thermodynamic floor. We further decompose this gap into contributions from transistor-level inefficiency, architectural overhead, memory transfer costs, and algorithmic redundancy, finding that memory data movement dominates at 62-78% of total energy. We propose Thermodynamically-Informed Pruning (TIP), a novel model compression strategy that preferentially removes computations with the highest TER per unit of output entropy, achieving 40% energy reduction with less than 1.2% perplexity degradation on GPT-class models. Our framework provides both a theoretical foundation for understanding the ultimate limits of efficient AI and a practical toolkit for energy-aware model optimization.
Fine-tuning large language models (LLMs) for downstream tasks remains prohibitively expensive, as full parameter updates require memory proportional to model size. Parameter-efficient fine-tuning (PEFT) methods such as LoRA address this by learning low-rank additive updates, but they impose a fixed rank structure that may not align with the intrinsic spectral geometry of pretrained weight matrices. We propose Low-Rank Spectral Adaptation (LoRSA), a novel PEFT method that leverages the singular value decomposition (SVD) of pretrained weights to identify and selectively adapt the most task-relevant spectral components. LoRSA decomposes each weight matrix $W = U \Sigma V^\top$ and learns lightweight perturbations $\Delta\sigma_i$ to a subset of singular values, along with low-rank rotations of the corresponding singular vectors. On the GLUE benchmark, LoRSA matches full fine-tuning performance on LLaMA-2 7B and 13B while training only 0.12% of parameters—a 3.2× reduction compared to LoRA at equivalent task performance. We further demonstrate LoRSA's advantages in multi-task adaptation scenarios, where spectral components exhibit interpretable task specialization.
Chain-of-thought (CoT) prompting has demonstrated remarkable effectiveness in eliciting complex reasoning capabilities from large language models (LLMs). In this work, we systematically investigate the emergent reasoning patterns that arise when LLMs are prompted to generate intermediate reasoning steps. Through extensive experiments across arithmetic, symbolic, and commonsense reasoning benchmarks, we identify three distinct phases of reasoning emergence as a function of model scale: pattern mimicry (< 10B parameters), structured decomposition (10B–70B), and adaptive strategy selection (> 70B). We introduce a formal taxonomy of reasoning primitives observed in CoT traces and propose the Reasoning Density Score (RDS), a novel metric that quantifies the information-theoretic efficiency of intermediate reasoning steps. Our analysis reveals that reasoning emergence is not merely a function of scale but depends critically on the interaction between pretraining data diversity, prompt structure, and attention head specialization. We find that models exceeding 70B parameters exhibit spontaneous error-correction behaviors in 23.7% of multi-step reasoning traces, a capability absent in smaller models. These findings provide new theoretical grounding for understanding how structured reasoning emerges from next-token prediction objectives.
