Abstract

Curiosity -- the intrinsic motivation to seek novel information -- is a cornerstone of biological intelligence and a critical missing ingredient in artificial agents deployed in open-ended environments. Current intrinsic motivation methods in reinforcement learning, such as prediction-error bonuses and count-based exploration, lack a unified theoretical foundation and often degenerate in stochastic or high-dimensional settings. We propose the Curiosity as Information Gain (CIG) framework, a principled formulation grounding artificial curiosity in the expected reduction of epistemic uncertainty over a learned world model. CIG decomposes curiosity into three operationally distinct components: Novelty Sensitivity, Learnability Filtering, and Competence-Weighted Priority. We derive a tractable variational bound and evaluate across six procedurally generated environments, demonstrating substantial improvements over existing intrinsic motivation baselines while avoiding the well-known noisy-TV problem.

1. Introduction

One of the most remarkable features of biological intelligence is the capacity for curiosity-driven exploration. Human infants, long before receiving explicit reward signals, systematically explore their environments in ways that maximize learning about the world's causal structure (Gopnik, 1996; Kidd & Hayden, 2015). This intrinsic motivation to reduce uncertainty is not merely a developmental curiosity -- it is computationally essential. In environments with sparse or deceptive extrinsic rewards, agents without curiosity-like mechanisms face an exponential exploration problem that renders learning intractable.

In reinforcement learning (RL), the exploration problem has traditionally been addressed through simple heuristics: epsilon-greedy action selection, Boltzmann exploration, or optimistic initialization. While sufficient for tabular settings, these approaches scale poorly to the complex, high-dimensional environments that characterize modern RL benchmarks and real-world robotics. This gap has motivated a growing body of work on intrinsic motivation -- reward signals generated internally by the agent to encourage exploration of novel or informative states.

Despite significant progress, current intrinsic motivation methods suffer from three interrelated limitations:

1. Lack of theoretical grounding. Methods like Random Network Distillation (RND; Burda et al., 2019) and the Intrinsic Curiosity Module (ICM; Pathak et al., 2017) are motivated by intuition rather than derived from a coherent theory of what the agent should find interesting.

2. Vulnerability to stochastic distractors. Prediction-error-based methods assign high curiosity to inherently unpredictable transitions (the "noisy-TV problem"), wasting exploration budget on states that yield no learnable information.

3. No competence modulation. Existing methods do not account for the agent's ability to act on the information it discovers. An agent may identify an interesting region of state space but lack the policy competence to exploit it productively.

In this work, we address all three limitations through the Curiosity as Information Gain (CIG) framework. Drawing on information-theoretic formulations from Bayesian experimental design (Lindley, 1956) and computational models of curiosity in cognitive science (Gottlieb et al., 2013; Oudeyer & Kaplan, 2007), CIG defines curiosity as the expected reduction in epistemic uncertainty about the environment's dynamics, filtered for learnability and weighted by policy competence.

2. Related Work

2.1 Intrinsic Motivation in RL

The use of intrinsic motivation for exploration in RL has a rich history. Count-based methods (Bellemare et al., 2016; Ostrovski et al., 2017) assign bonuses inversely proportional to state visitation frequency, providing theoretical guarantees in tabular MDPs but requiring density estimation in continuous spaces. Prediction-error methods (Stadie et al., 2015; Pathak et al., 2017) use the error of a learned forward model as an exploration bonus, while RND (Burda et al., 2019) replaces the forward model with a fixed random network, using the prediction error of a trained network against the fixed target as a novelty signal. Information-gain methods (Houthooft et al., 2016; Still & Precup, 2012) are most closely related to our work, but prior formulations either rely on intractable posterior computations or do not address the learnability and competence issues we identify.

2.2 Curiosity in Cognitive Science

The information-gap theory (Loewenstein, 1994) posits that curiosity arises from the perception of a gap between what one knows and what one wants to know. More formal accounts (Gottlieb et al., 2013; Kidd & Hayden, 2015) model curiosity as a function of the expected information gain from an observation, modulated by the observer's confidence in their ability to resolve the uncertainty. Our CIG framework operationalizes this cognitive model in the RL setting.

3. The CIG Framework

3.1 Problem Setting

Consider an agent interacting with an environment modeled as a Markov Decision Process $(S, A, T, R, \gamma)$ where the transition function $T(s'|s,a)$ is unknown. The agent maintains a parametric world model $\hat{T}$\theta(s'|s,a)T^θ​(s′∣s,a) and a policy $\pi$\phi(a|s). We seek an intrinsic reward function $r^{\text{int}}(s, a, s')$ that drives the agent to explore efficiently.

3.2 Curiosity as Expected Information Gain

We define the curiosity signal at state-action pair $(s, a)$ as the expected information gain about the world model parameters $\theta$:

$$r^{\text{CIG}}(s, a) = \mathbb{E}$${s' \sim T(\cdot|s,a)} \left[ D{\text{KL}}\left( p(\theta | \mathcal{D} \cup {(s,a,s')}) | p(\theta | \mathcal{D}) \right) \right]

where $\mathcal{D}$ is the agent's experience buffer. This quantity measures how much the agent expects to learn from taking action $a$ in state $s$ -- transitions that are both surprising and informative yield high values, while transitions that are surprising but uninformative (stochastic noise) do not.

3.3 Decomposition into Three Components

Direct computation of the above is intractable for deep models. We derive a practical decomposition:

Component 1: Novelty Sensitivity ($\mathcal{N}$). We approximate the information gain using an ensemble of $K$ world models ${\hat{T}${\theta_k}}{k=1}^K. The novelty signal is the mutual information between the ensemble index and the predicted next state:

$$\mathcal{N}(s, a) = H\left[\frac{1}{K}\sum_k \hat{T}$${\theta_k}(\cdot|s,a)\right] - \frac{1}{K}\sum_k H\left[\hat{T}{\theta_k}(\cdot|s,a)\right]

This captures epistemic uncertainty (disagreement between models) while ignoring aleatoric uncertainty (entropy within each model).

Component 2: Learnability Filter ($\mathcal{L}$). Not all novel states are learnable. We estimate the aleatoric uncertainty using the average within-model entropy and discount the curiosity signal accordingly:

$$\mathcal{L}(s, a) = \sigma\left(\alpha - \frac{1}{K}\sum_k H\left[\hat{T}_{\theta_k}(\cdot|s,a)\right]\right)$$

where $\sigma$ is the sigmoid function and $\alpha$ is a learned threshold. This sigmoid gate suppresses curiosity for transitions dominated by irreducible stochasticity.

Component 3: Competence-Weighted Priority ($\mathcal{C}$). Following cognitive models of curiosity that emphasize the role of confidence in one's ability to learn (Gottlieb et al., 2013), we modulate exploration priority by a measure of policy competence:

$$\mathcal{C}(s) = \exp\left(-\beta \cdot \text{Var}$${a \sim \pi\phi(\cdot|s)}\left[Q_\psi(s, a)\right]\right)

High policy variance in Q-values indicates the agent is uncertain about how to act in state $s$ -- precisely the states where additional exploration is most valuable. The full CIG reward is:

$$r^{\text{CIG}}(s, a) = \mathcal{N}(s, a) \cdot \mathcal{L}(s, a) \cdot \mathcal{C}(s)$$

3.4 Variational Bound

We derive a tractable evidence lower bound (ELBO) for the CIG objective. Let $q_\lambda(\theta)$ be a variational approximation to the posterior $p(\theta|\mathcal{D})$. Then:

$$r^{\text{CIG}}(s,a) \geq \mathbb{E}$${s' \sim \hat{T}}\left[\log \hat{T}{\bar{\theta}}(s'|s,a) - \frac{1}{K}\sum_k \log \hat{T}_{\theta_k}(s'|s,a)\right] \cdot \mathcal{L}(s,a) \cdot \mathcal{C}(s)

where $\bar{\theta}$ denotes the mean ensemble parameters. This bound is tight when the ensemble provides a good approximation to the posterior and is computationally efficient, requiring only $K$ forward passes through the world model ensemble.

4. Experimental Setup

4.1 Environments

We evaluate CIG on six procedurally generated environments designed to test different facets of exploration:

1. MazeWorld-v3 -- A 2D navigation task with procedurally generated mazes of varying complexity. Tests spatial coverage and dead-end avoidance.
2. CausalChains-v1 -- A discrete environment where the agent must discover hidden cause-effect relationships between objects. Tests systematic hypothesis-driven exploration.
3. NoisyGridWorld -- A grid environment with stochastic teleportation tiles (the noisy-TV analog). Tests robustness to irreducible stochasticity.
4. RoboManip-v2 -- A continuous-control robotic manipulation task with 12-DOF action space. Tests exploration in high-dimensional continuous domains.
5. ComboPuzzle -- A combinatorial task requiring the agent to discover multi-step unlock sequences. Tests long-horizon structured exploration.
6. OpenField-Sparse -- A large continuous environment with sparse hidden rewards. Tests breadth of coverage.

4.2 Baselines

We compare against five baselines: (1) epsilon-greedy ($\epsilon=0.1$), (2) ICM (Pathak et al., 2017), (3) RND (Burda et al., 2019), (4) VIME (Houthooft et al., 2016), and (5) RE3 (Seo et al., 2021). All methods use the same PPO backbone with identical hyperparameters.

5. Results

5.1 Exploration Coverage

Across all six environments, CIG achieves the highest state coverage within a fixed compute budget of $10^6$ environment steps. The aggregate results (mean over 10 seeds):

CIG outperforms the best baseline (RND) by 13.6 percentage points on average. The largest gains appear on NoisyGridWorld (+35.5pp over RND), confirming CIG's robustness to stochastic distractors, and on CausalChains (+11.4pp), where the learnability filter helps the agent focus on causally informative transitions.

5.2 The Noisy-TV Experiment

To directly test robustness to the noisy-TV problem, we augmented MazeWorld with a "television" tile that emits random pixel patterns. ICM agents spent 67% of their time at the noisy tile; RND agents spent 41%. CIG agents spent only 3.2%, nearly identical to the oracle baseline with the tile removed (2.8%). The learnability filter correctly identifies the television's output as having high aleatoric uncertainty and suppresses curiosity accordingly.

5.3 Ablation Study

To understand the contribution of each CIG component, we ablated them individually on MazeWorld and CausalChains:

All three components contribute meaningfully. Novelty Sensitivity is the most critical (removing it collapses performance to near-random), but both the Learnability Filter and Competence-Weighted Priority provide substantial additive gains of 9.3pp and 3.8pp respectively on MazeWorld.

6. Discussion

The CIG framework provides a principled answer to the question: what should an artificial agent find interesting? By grounding curiosity in information gain, filtering for learnability, and weighting by competence, CIG avoids the pathologies that plague simpler intrinsic motivation schemes. The framework also establishes a formal connection between computational curiosity in RL and information-seeking models in cognitive science.

Several limitations and directions for future work deserve mention. First, the ensemble-based uncertainty estimation adds computational overhead linear in $K$; future work might explore more efficient uncertainty quantification methods such as spectral-normalized neural GPs. Second, the competence weighting currently uses Q-value variance as a proxy, which may not generalize to all policy parameterizations. Third, while CIG excels at exploration, integrating the intrinsic reward with extrinsic task rewards in a principled manner (beyond simple summation) remains an open problem.

7. Conclusion

We introduced the Curiosity as Information Gain (CIG) framework, a theoretically grounded approach to intrinsic motivation in reinforcement learning. By decomposing curiosity into novelty sensitivity, learnability filtering, and competence-weighted priority, CIG achieves state-of-the-art exploration performance across diverse environments while remaining robust to stochastic distractors. Our framework bridges computational models of curiosity from cognitive science with practical exploration algorithms, offering a foundation for building agents that can autonomously explore and learn in open-ended worlds.

References

[1] Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. NeurIPS.

[2] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2019). Exploration by random network distillation. ICLR.

[3] Gopnik, A. (1996). The scientist as child. Philosophy of Science, 63(4), 485-514.

[4] Gottlieb, J., Oudeyer, P. Y., Lopes, M., & Baranes, A. (2013). Information-seeking, curiosity, and attention. Trends in Cognitive Sciences, 17(11), 585-593.

[5] Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., & Abbeel, P. (2016). VIME: Variational information maximizing exploration. NeurIPS.

[6] Kidd, C., & Hayden, B. Y. (2015). The psychology and neuroscience of curiosity. Neuron, 88(3), 449-460.

[7] Lindley, D. V. (1956). On a measure of the information provided by an experiment. Annals of Mathematical Statistics, 27(4), 986-1005.

[8] Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116(1), 75-98.

[9] Oudeyer, P. Y., & Kaplan, F. (2007). What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 1, 6.

[10] Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. ICML.

[11] Seo, Y., Chen, L., Shin, J., Lee, H., Abbeel, P., & Lee, K. (2021). State entropy maximization with random encoders for efficient exploration. ICML.

[12] Stadie, B. C., Levine, S., & Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arXiv:1507.00814.

[13] Still, S., & Precup, D. (2012). An information-theoretic approach to curiosity-driven reinforcement learning. Theory in Biosciences, 131(3), 139-148.