DeLERP: Decomposed Linear Interpolation for Model Merging

We propose a merge technique that decomposes and decouples magnitude and direction for merger, an approach inspired by practical observations of the impact of norm preservation on post-abliteration performance.

The Problem with Linear Interpolation

When merging neural network weights, the simplest approach is linear interpolation (LERP):

$$\mathbf{v}(t)=(1-t){\mathbf{v}}_0+t{\mathbf{v}}_1\backslash mathbf\{v\}(t)\; =\; (1-t)\backslash mathbf\{v\}\_0\; +\; t\backslash mathbf\{v\}\_1$$

This works, but it has a subtle geometric flaw. Consider interpolating between two unit vectors—vectors of length 1. The midpoint $(0.5\mathbf{v}_0 + 0.5\mathbf{v}_1)$ doesn't lie on the unit sphere; it falls inside it, with a smaller norm. This "norm dip" means LERP gradually reduces the magnitude of weights during interpolation, potentially weakening the model's representational capacity at intermediate points.

The SLERP Solution (and Its Limitations)

Spherical Linear Interpolation (SLERP) fixes the norm dip by constraining interpolation to the surface of a sphere:

$$\text{SLERP}({\mathbf{v}}_0,{\mathbf{v}}_1,t)=\frac{\sin ((1-t)\theta )}{\sin (\theta )}{\mathbf{v}}_0+\frac{\sin (t\theta )}{\sin (\theta )}{\mathbf{v}}_1\backslash text\{SLERP\}(\backslash mathbf\{v\}\_0,\; \backslash mathbf\{v\}\_1,\; t)\; =\; \backslash frac\{\backslash sin((1-t)\backslash theta)\}\{\backslash sin(\backslash theta)\}\backslash mathbf\{v\}\_0\; +\; \backslash frac\{\backslash sin(t\backslash theta)\}\{\backslash sin(\backslash theta)\}\backslash mathbf\{v\}\_1$$

where $\theta = \arccos\left(\frac{\mathbf{v}_0 \cdot \mathbf{v}_1}{|\mathbf{v}_0| |\mathbf{v}_1|}\right)$.

SLERP maintains constant distance from the origin, but this reveals a conceptual issue: why should the zero vector be special? For neural network weights, there's no inherent reason that distance from zero is the "right" thing to preserve. SLERP privileges an arbitrary reference point. It's also mildly computationally expensive, requiring trigonometric operations for every weight.

Key Insight: Decompose Direction and Magnitude

The fundamental issue is that direction and magnitude are conflated in standard interpolation methods. What if we treat them separately? Zheng et al proposed exactly that, albeit in a different context.

• Direction encodes what the weights represent—the patterns, features, or behaviors they capture
• Magnitude encodes how strongly those patterns matter—their importance or confidence

For model merging, we want to:

1. Smoothly transition between the directional semantics of two models
2. Preserve representational capacity without arbitrary geometric constraints

The DeLERP Method

DeLERP (Decomposed Linear Interpolation) separates direction and magnitude, interpolating each independently:

Direction via NLERP (Normalized Linear Interpolation):

$$\widehat{\mathbf{d}}(t)=\frac{(1-t){\mathbf{v}}_0+t{\mathbf{v}}_1}{\mathrm{\parallel }(1-t){\mathbf{v}}_0+t{\mathbf{v}}_1\mathrm{\parallel }}\backslash hat\{\backslash mathbf\{d\}\}(t)\; =\; \backslash frac\{(1-t)\backslash mathbf\{v\}\_0\; +\; t\backslash mathbf\{v\}\_1\}\{\backslash |(1-t)\backslash mathbf\{v\}\_0\; +\; t\backslash mathbf\{v\}\_1\backslash |\}$$

This performs linear interpolation in the original space, then normalizes to get a unit direction vector. NLERP provides a cheap approximation to SLERP's spherical geodesic while remaining coordinate-free and geometric.

Magnitude via Max Norm:

$$m=\max (\mathrm{\parallel }{\mathbf{v}}_0\mathrm{\parallel },\mathrm{\parallel }{\mathbf{v}}_1\mathrm{\parallel })m\; =\; \backslash max(\backslash |\backslash mathbf\{v\}\_0\backslash |,\; \backslash |\backslash mathbf\{v\}\_1\backslash |)$$

Rather than averaging magnitudes (which would privilege the origin) or maintaining constant magnitude (which would privilege a specific sphere), we preserve the stronger importance signal from either model. This maintains representational capacity throughout the interpolation.

Final Interpolation:

$$\text{DeLERP}({\mathbf{v}}_0,{\mathbf{v}}_1,t)=m\cdot \widehat{\mathbf{d}}(t)\backslash text\{DeLERP\}(\backslash mathbf\{v\}\_0,\; \backslash mathbf\{v\}\_1,\; t)\; =\; m\; \backslash cdot\; \backslash hat\{\backslash mathbf\{d\}\}(t)$$

The result: smooth directional transitions without norm dips, no privileged origin, and minimal computational overhead. Each weight parameter is treated as a point in high-dimensional space, with its direction and magnitude handled separately.

Implementation

We used mergekit as a testbed for DeLERP. Below is the core algorithm in Python, should anyone want to immediately reproduce it.

def delerp(
    t: Union[float, np.ndarray],
    v0: Union[np.ndarray, torch.Tensor],
    v1: Union[np.ndarray, torch.Tensor],
    eps: float = 1e-8,
):
    """
    Decomposed Linear Interpolation (DeLERP)
    
    Separates direction and magnitude: direction via NLERP (Normalized Linear
    Interpolation), magnitude via maximum norm. Operates on each weight
    parameter, treating the entire tensor as a vector in high-dimensional space
    for norm calculations.
    
    Direction: NLERP on unit sphere (coordinate-free geometric interpolation)
    Magnitude: max(||v0||, ||v1||) to preserve stronger importance signal
    
    Args:
        t (float/np.ndarray): Float value between 0.0 and 1.0
        v0 (np.ndarray): Starting vector
        v1 (np.ndarray): Final vector
        eps (float): Small epsilon for numerical stability
    Returns:
        v2 (np.ndarray): Interpolation vector between v0 and v1
    """
    is_torch = False
    if not isinstance(v0, np.ndarray):
        is_torch = True
        v0 = v0.detach().cpu().float().numpy()
    if not isinstance(v1, np.ndarray):
        is_torch = True
        v1 = v1.detach().cpu().float().numpy()

    # Compute norms (treating entire tensor as single vector)
    norm_v0 = np.linalg.norm(v0)
    norm_v1 = np.linalg.norm(v1)
    
    # Handle zero vectors
    if norm_v0 < eps and norm_v1 < eps:
        return maybe_torch(np.zeros_like(v0), is_torch)
    
    # NLERP: Linear interpolation followed by normalization
    d_merged = (1 - t) * v0 + t * v1
    norm_d_merged = np.linalg.norm(d_merged)
    
    if norm_d_merged > eps:
        # Normalize to get unit direction vector
        d_merged = d_merged / norm_d_merged
    else:
        # Vectors cancel out - fallback to linear interpolation
        # This handles the edge case where v0 and v1 are nearly opposite
        return maybe_torch((1 - t) * v0 + t * v1, is_torch)
    
    # Max norm: Preserve the stronger magnitude signal
    m_merged = max(norm_v0, norm_v1)
    
    # Recombine: direction * magnitude
    result = d_merged * m_merged
    
    return maybe_torch(result, is_torch)

Key implementation details:

• Epsilon threshold (eps=1e-8) prevents numerical instability for near-zero weights
• Opposing vector handling: When v0 and v1 nearly cancel out (norm close to zero), we fall back to standard linear interpolation
• Framework flexibility: Automatically handles both NumPy arrays and PyTorch tensors

Results

For testing, we implemented DeLERP as a module for mergekit.

| Model | UGI | W/10 | Writing | NatInt |

| mistralai/Mistral-Nemo-Instruct-2407 | 30.26 | 5.5 | 33.07 | 20.8 |

| DeLERP Base+Instruct, 99.9% Instruct | 26.02 | 4.5 | 37.03 | 21.68 |

Where, according to the UGI Leaderboard:

UGI: Uncensored General Intelligence
W/10 (Willingness/10): A component of the UGI score that measures how far a model can be pushed before it refuses to answer or deviates from instructions.
Writing: A score of a model's writing ability, factoring in intelligence, writing style, amount of repetition, and adherence to requested output length. The score attempts to match the average person's preferences.
NatInt: Natural Intelligence

Discussion

For this experiment, we wanted to see what would happen if an Instruct model were "reinflated" with Base model magnitudes, with DeLERP as an alternative to the approach used Sharma et al. Notably, DeLERP demonstrates an unusual property: at t=0.999, the merge shows simultaneous improvements in capability metrics (Writing, Natural Intelligence) and alignment (lower Willingness score indicating more refusals on uncensored prompts). This suggests that preserving representational capacity through max norm can boost cognitive performance while strengthening—rather than compromising—safety properties.

Recent work on variance collapse in model merging (Sharma et al., 2024) suggests that naive interpolation can push weights into low-density regions of the training distribution. While DeLERP doesn't directly address the statistical aspects of this problem, its preservation of magnitude through max norm and geometric direction interpolation may help avoid some forms of distributional collapse. This remains an open question for empirical investigation.

References

• Sharma et al, "The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse", arXiv preprint, , 2024.
• Zheng et al, "Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging", arXiv preprint, 2025

External Links

• Merged 12B model weights on Huggingface
• mergekit
• UGI Leaderboard