Ratio-based Loss Functions

Technical Summary: Ratio-Based Loss Functions

Problem Statement
In supervised machine learning, the choice of loss function is a critical component alongside the hypothesis space and the set of probability measures. While distance-based loss functions (relying on $|y - f(x)|$) are standard for regression under additive error assumptions (signal plus noise), they fail to capture relative errors effectively. In many real-world applications—such as medical dosing, financial modeling, and quality control—output spaces are strictly positive ($Y \subset (0, \infty)$), and error structures are often multiplicative (signal times noise). Distance-based approaches are inherently scale-dependent; a constant absolute error may be negligible for large values but critical for small ones. Conversely, ratio-based approaches offer scale invariance. However, the literature lacks a systematic theoretical framework for ratio-based loss functions, particularly regarding their general mathematical properties (continuity, convexity, differentiability) and their relationship to the risk functional.

Methodology
The authors propose a formal definition for ratio-based (rb) loss functions and systematically analyze their properties.

• Definition: A supervised loss function $L: \mathcal{X} \times \mathcal{Y} \times \mathbb{R} \to [0, \infty)$ is defined as ratio-based if there exists a representing function $\ell: (0, \infty) \to [0, \infty)$, a constant $c \ge 0$, and a link function $u: \mathbb{R} \to \mathcal{Y}$ (measurable, surjective, monotonic) such that:
  $$L(x, y, t) = \ell\left(\frac{u(t) + c}{y + c}\right)$$
  where $\mathcal{Y} = (a, b)$ with $0 \le a < b \le \infty$. The case $c=0$ is termed "strictly ratio-based." The link function $u$ transforms real-valued predictions $t$ into the output space $\mathcal{Y}$ to ensure the ratio is well-defined.
• Theoretical Analysis: The paper investigates how properties of the representing function $\ell$ and the link function $u$ (e.g., continuity, Lipschitz continuity, convexity, differentiability) transfer to the loss function $L$ and the corresponding risk functional $R_{L,P}(f)$. The analysis relies on standard results from statistical learning theory (e.g., Csiszár, Steinwart, and Christmann) regarding the inheritance of properties from the loss to the risk.
• Comparison: The authors analyze the relationship between ratio-based and distance-based losses, demonstrating that while they can be mapped to one another via logarithmic transformations, they are not equivalent in their original argument spaces.

Key Contributions

1. General Framework: The paper establishes a unified theoretical framework for ratio-based loss functions, distinguishing them from existing margin-based (classification) and distance-based (regression) approaches.
2. Property Transfer: It provides rigorous lemmas and propositions detailing the conditions under which $L$ inherits continuity, differentiability, convexity, and Lipschitz continuity from $\ell$ and $u$. A key finding is the tension between convexity and Lipschitz continuity in ratio-based settings: convexity often requires $c=0$ (strict ratio-based), while global Lipschitz continuity often requires $c>0$ to bound the ratio near zero or infinity.
3. New and Surveyed Loss Functions: The authors catalog existing ratio-based losses from literature (e.g., Least Absolute Relative Error, Least Product Relative Error, Logarithmic RMSE) and propose several new functions, including:
   • Huber-type logarithmic and relative losses.
   • Log-cosh-log and Cosh-log relative losses.
   • Smoothed robust versions of Least Absolute Relative Error (LARE) and Least Product Relative Error (LPRE) inspired by Hampel's loss.
   • Weighted and insensitive variants of ratio-based losses.
4. Risk Analysis: The paper addresses the finiteness of the risk functional, showing that boundedness of $\ell$ or specific conditions on $u$ and $c$ are necessary to ensure finite risk for measurable functions, particularly when $\mathcal{Y} = (0, \infty)$.
5. Alternative Definitions: The authors briefly consider an alternative definition using the inverse ratio ($\frac{y+c}{u(t)+c}$) and demonstrate that while properties largely hold, the original definition (1) offers technical advantages in proving global Lipschitz continuity and risk boundedness.

Results

• Convexity: The paper proves that for $\mathcal{Y}=(0,1)$, non-constant convex ratio-based losses are generally impossible regardless of $c$. However, for $\mathcal{Y}=(0, \infty)$ with $c=0$ and $u(t)=e^t$, convex losses can be constructed if $\ell$ satisfies specific differential inequalities (e.g., $\ell(r) = e^{\tilde{\ell}(r)} + e^{\tilde{\ell}(r^{-1})} - 2e^{\tilde{\ell}(1)}$).
• Lipschitz Continuity: Global Lipschitz continuity is achievable if the output space is bounded or if $c>0$ is used with bounded domains. For unbounded domains with $c=0$, global Lipschitz continuity is often lost unless $\ell$ is specifically bounded.
• Differentiability: Differentiability of $L$ depends on the differentiability of both $\ell$ and $u$. The paper provides explicit formulas for the first and second derivatives.
• Nemitski Property: The authors establish conditions under which ratio-based losses satisfy the Nemitski property, which is crucial for the existence of minimizers and the continuity of the risk functional.
• Tables of Properties: Comprehensive tables (Table 2 and Table 3) summarize the properties (convexity, Lipschitz continuity, differentiability) of various proposed and existing loss functions under different link functions ($u(t) = e^t$ vs. logistic) and constants ($c=0$ vs. $c>0$).

Significance and Claims
The authors explicitly state that the paper is a survey and theoretical investigation rather than a derivation of specific learning rates, consistency proofs, or stability results for a particular algorithm. The significance lies in:

• Foundational Work: Providing the necessary mathematical groundwork (continuity, convexity, differentiability) to enable future research on ratio-based learning algorithms.
• Bridging the Gap: Highlighting that while relative errors are widely used in practice (e.g., in dose-response studies or inflation modeling), they lack the systematic theoretical treatment afforded to distance-based losses.
• Future Directions: The paper aims to stimulate research into finding additional convex and Lipschitz continuous ratio-based losses, deriving representer theorems, and establishing statistical robustness results (influence functions, maxbias) for kernel-based methods and CNNs using these losses. The authors note that the choice between distance-based and ratio-based approaches should be driven by the specific application's error structure (additive vs. multiplicative) rather than mathematical convenience alone.