Estimating the distance at which narwhal $(\textit{Monodon monoceros})$ respond to disturbance: a penalized threshold hidden Markov model

This paper introduces a novel lasso-penalized threshold hidden Markov model that effectively distinguishes meaningful behavioral shifts from spurious noise, revealing that narwhals react to vessel disturbances up to 4 kilometers away by altering their movement patterns and diving deeper.

The Gist

Imagine you are trying to figure out exactly how close a stranger needs to get to you before you start feeling uncomfortable and change your behavior. Maybe you stop walking in a straight line, you start looking around nervously, or you decide to dive into a deep pool to hide.

Now, imagine doing this for Narwhals—those Arctic whales with the long spiral tusk—while they are swimming in the freezing ocean. Scientists want to know: How close does a ship have to get before the Narwhal says, "Okay, I'm done with this, I'm changing my plan"?

This paper is about building a super-smart mathematical tool to answer that question, and then using it to actually find the answer for Narwhals.

Here is the breakdown in simple terms:

1. The Problem: The "Goldilocks" Zone of Noise

In the Arctic, sea ice is melting, which means more ships are sailing through areas that used to be quiet. Ships are loud. We know Narwhals don't like loud noises, but we don't know exactly how far away the noise has to be to bother them.

• The old way: Scientists used to watch one whale at a time and guess when it got scared. It was like trying to find a needle in a haystack by looking at one piece of hay at a time.
• The new way: The authors created a computer model that looks at all the whales at once and finds the exact "tipping point" distance.

2. The Tool: The "Smart Switch" (The Threshold Hidden Markov Model)

To understand the tool, imagine a light switch in your house.

• State A (Baseline): The light is off. You are walking normally, diving normally.
• State B (Disturbed): The light is on. You are acting weird because something is wrong.

In the past, scientists had to guess where the switch was. They would try a guess, see if it worked, then try another guess, and another. It was like trying to find the right combination on a safe by turning the dial one number at a time. It took forever and sometimes the computer got stuck.

The Innovation:
The authors invented a new kind of "Smart Switch" called a Penalized Threshold Hidden Markov Model (THMM).

• "Hidden Markov": This just means the model guesses what the whale is thinking (resting, hunting, running away) based on where it is swimming, even though we can't see its brain.
• "Threshold": This is the distance. The model tries to find the specific distance where the switch flips from "calm" to "scared."
• "Penalized" (The Lasso): This is the magic trick. Imagine you are trying to find a needle in a haystack, but you have a magnet that pulls away all the fake needles (false alarms). The "Lasso" is a mathematical magnet. If the model thinks a ship caused a scare, but the evidence is weak, the Lasso pulls that idea back to zero. It forces the model to only admit a disturbance exists if it is really sure.

Why is this cool?
It stops the computer from seeing ghosts. Sometimes, a whale might just decide to dive deep for fun, not because a ship is there. Old models might have said, "Oh, a ship must be nearby!" The new model says, "Nah, that's just the whale being a whale. No ship needed."

3. The Experiment: The Narwhal Test

The team took data from 18 Narwhals in the Canadian Arctic. They had GPS tags on the whales and satellite data on every ship in the area.

They fed this data into their "Smart Switch" model.

The Results:

• The Magic Number: The Narwhals start reacting when a ship gets within 4 kilometers (about 2.5 miles).
• The Reaction: When a ship gets that close, the Narwhals don't just swim away in a straight line. They:
  1. Stop being persistent: They stop swimming in a straight, purposeful path. They get jittery.
  2. Go deep: They dive deeper (average max depth of 356 meters). It's like they are hiding underwater to avoid the noise.
• The "Soundproof" Effect: The model also noticed something interesting. If there was an island between the ship and the whale, the whale didn't react as much. The land blocked the sound, acting like a soundproof wall.

4. Why This Matters

This isn't just about Narwhals; it's about saving them.

• Policy Making: Now, governments know exactly how far ships need to stay away to keep the whales happy. They can say, "Ships must slow down or stay 4km away."
• A Better Tool: The math they invented isn't just for whales. It can be used for anything where you need to find a "tipping point."
  • How close does an elephant have to get to a water hole before it knows it's there?
  • How hot does the ocean get before coral reefs start dying?
  • How much noise does a city need before birds stop singing?

The Bottom Line

The authors built a mathematical magnifying glass that filters out the noise (both literal and statistical) to find the exact moment an animal gets disturbed. They proved that Narwhals are sensitive to ships from 4km away and that they hide deep underwater to escape. Most importantly, they gave scientists a faster, smarter way to find these "tipping points" for any animal in the future, helping us protect wildlife in a noisy world.

Technical Summary

Here is a detailed technical summary of the paper "Estimating the Distance at Which Narwhal (Monodon Monoceros) Respond to Disturbance: A Penalized Threshold Hidden Markov Model."

1. Problem Statement

Understanding how wildlife responds to anthropogenic disturbances (e.g., vessel noise) is critical for conservation, particularly in the Arctic where shipping routes are expanding. While telemetry data offers fine-scale behavioral insights, existing methods face two major challenges:

• Data Limitations: Most disturbance studies rely on controlled exposure experiments. In the wild, disturbance is gradual and difficult to pinpoint, making it hard to define a specific "threshold" distance at which behavior changes.
• Methodological Gaps: Threshold Hidden Markov Models (THMMs) are powerful for detecting regime shifts (baseline vs. disturbed) based on covariates (e.g., distance to a vessel). However, standard THMMs suffer from:
  1. Computational Inefficiency: Estimating thresholds typically requires computationally prohibitive grid searches.
  2. Lack of Validity Assessment: There is no robust method to determine if an estimated threshold reflects a meaningful behavioral shift or is merely a spurious artifact of the model fitting a null hypothesis (no disturbance). Standard Likelihood Ratio Tests (LRTs) are invalid for mixture models, and Bootstrap LRTs (BLRTs) are too computationally expensive for large telemetry datasets.

2. Methodology

The authors propose a Lasso-Penalized Threshold Hidden Markov Model (THMM) combined with a Quasi-Restricted Maximum Likelihood (qREML) framework to address these issues.

A. Model Structure

• THMM Framework: The model assumes animal behavior is governed by a latent Markov chain with $N$ states. The transition probabilities between states depend on whether the system is in a Baseline Regime (no disturbance) or a Disturbed Regime.
• Threshold Mechanism: The probability of switching between regimes is controlled by a step function $\nu_{\beta_0}(u_t)$, which activates when a covariate $u_t$ (e.g., inverse distance to a vessel) exceeds a threshold $1/\beta_0$.
• Smooth Approximation: To enable gradient-based optimization, the discontinuous step function is approximated by a smooth, two-parameter logistic function.

B. Penalized Likelihood Estimation

• Lasso Penalty: To prevent the inclusion of a "disturbed" regime when none exists (i.e., to test if $\beta_0 = 0$), the authors apply an $\ell_1$-norm penalty (Lasso) to the threshold parameter $\beta_0$.
  • If the penalty is strong enough, $\beta_0$ is shrunk to zero, reducing the model to a standard HMM (single regime).
  • If $\beta_0$ remains non-zero, it indicates a statistically significant disturbance threshold.
• Multivariate Extension: The model can handle mutually exclusive covariates (e.g., vessel exposure with/without land blocking sound) to estimate distinct thresholds for different scenarios.

C. qREML for Penalty Selection

Selecting the tuning parameter ($\lambda$) for the Lasso penalty is traditionally done via cross-validation or grid search, which is slow for time-series data. The authors introduce a computationally efficient qREML approach:

• Random Effects Interpretation: The threshold parameter $\beta_0$ is treated as a random effect with an exponential prior (equivalent to the Lasso penalty).
• Laplace Approximation: The marginal likelihood is approximated using the Laplace method around the mode.
• Iterative Update: A closed-form update rule for $\lambda$ is derived based on the partial derivative of the marginal log-likelihood, allowing for rapid convergence without grid searches.

D. Numerical Implementation

• The optimization uses automatic differentiation (via R packages RTMB and LaMa).
• A progressive sharpness initialization strategy is employed: the model is first fitted with a smooth transition (low sharpness parameter $b$) and gradually increased to approximate a hard step function, avoiding numerical instability.

3. Key Contributions

1. Novel Estimator: Introduction of a penalized quasi-restricted maximum likelihood estimator for THMMs that simultaneously estimates thresholds and selects the model structure (number of regimes).
2. Computational Efficiency: The method eliminates the need for grid searches, offering a significant speedup over Bootstrap Likelihood Ratio Tests (BLRTs) while maintaining accuracy.
3. False Positive Control: The Lasso penalty effectively shrinks spurious threshold estimates to zero, providing a principled way to distinguish between true behavioral shifts and noise.
4. First Model-Based Threshold: This study provides the first model-based estimate of a disturbance threshold in movement ecology derived from non-controlled, observational data.

4. Results

Simulation Study

• Accuracy: The method accurately estimated $\beta_0$ and state-dependent parameters across various sample sizes ($T=1,000$ to $10,000$).
• False Positive Rate: For sample sizes $T \geq 3,000$, the false positive rate was below 0.02, significantly outperforming BLRTs in multivariate scenarios (where BLRTs struggled to identify the correct covariate).
• Speed: The Lasso-penalized THMM was substantially faster than BLRTs, making it feasible for large-scale telemetry datasets.

Case Study: Narwhal Response to Vessels

• Data: 18 narwhals tagged in Tremblay Sound, Nunavut (2017), with GPS and depth data, correlated with vessel AIS data.
• Key Findings:
  • Reaction Distance: Narwhals react to vessels up to 4 kilometers away.
  • Behavioral Changes: Upon detecting vessels within this range, narwhals:
    • Decrease movement persistence (become less directed).
    • Increase time spent in deep water (average max depth of 356m).
    • Show increased probability of transitioning from shallow, slow movement to deep diving (an "escape dive" behavior).
  • Land Effect: No behavioral change was detected when land (islands/peninsulas) lay between the whale and the vessel, suggesting land blocks or attenuates vessel noise.
  • Shore Proximity: Behavioral responses were stronger farther from shore; near-shore constraints (bathymetry) limited deep diving capabilities.

5. Significance and Implications

• Conservation Policy: The estimated 4km threshold provides a concrete metric for policymakers to design mitigation strategies, such as vessel slow-down zones or avoidance areas, to protect Arctic marine mammals.
• Methodological Advancement: The framework moves beyond animal movement, offering a general tool for any time-series analysis requiring threshold detection (e.g., finance, epidemiology, ecology).
• Future Applications: The authors suggest this method could be applied to estimate other difficult-to-characterize thresholds, such as the distance at which elephants detect water or the population density triggering predator-prey shifts.
• Robustness: By addressing the issue of spurious regime detection, the method ensures that conservation decisions are based on statistically rigorous evidence of behavioral change rather than model artifacts.

In summary, this paper presents a statistically rigorous and computationally efficient framework for quantifying wildlife responses to environmental stressors, successfully applying it to reveal specific, actionable thresholds for narwhal conservation in the face of increasing Arctic shipping.