Why Composited People Look Pasted: Lighting and Color Coherence

You can nail every facial feature and still produce a composite that looks obviously fake. I’ve watched people burn weeks on identity preservation, get it perfect, then ship an image where two people supposedly standing in the same room are lit by two different suns. The brain catches it instantly, without being able to say why. Lighting coherence is the part nobody writes about, so here’s what I’ve learned making it work.

Why “pasted” is a lighting judgment, not a geometry judgment

Human vision is startlingly bad at absolute measurements and startlingly good at consistency checks. We can’t tell you a room’s color temperature, but we detect within milliseconds that one face is 5500K daylight and the other 3000K tungsten. Same with shadow direction: nobody can estimate a light’s azimuth, but two faces with shadows falling opposite ways read as wrong before you’ve consciously looked.

Four signals dominate the “pasted” verdict:

1. Color temperature — the global chromatic cast on skin
2. Key light direction — where the dominant light comes from, which drives shadow side
3. Contrast ratio and softness — hard sun vs. overcast vs. bounced indoor light
4. Edge and atmospheric consistency — how the subject boundary and ambient haze behave

Identity isn’t on the list. You can preserve every landmark perfectly and fail all four.

Estimate the lighting before you generate anything

The mistake is treating harmonization as post-processing. By the time you have a composite, half the information is gone. Estimate lighting parameters from each source photo first, then use them to constrain generation or pick which subject gets relit toward the other. Rough but effective estimators, no ML required:

import numpy as np

def estimate_color_temp(skin_rgb):
    """Crude CCT proxy. Higher R/B => warmer light. Relative, not Kelvin."""
    r, g, b = skin_rgb.mean(axis=0)
    return r / max(b, 1e-6)

def estimate_key_direction(gray, face_mask):
    """Shading gradient over the face approximates the light azimuth."""
    gy, gx = np.gradient(gray.astype(float))
    # Light comes FROM the direction of increasing luminance.
    return np.degrees(np.arctan2(-gy[face_mask].mean(),
                                  gx[face_mask].mean()))

def estimate_hardness(gray, face_mask):
    """Contrast ratio proxy; hard light => wide spread."""
    v = gray[face_mask]
    p95, p05 = np.percentile(v, 95), np.percentile(v, 5)
    return (p95 - p05) / max(p95 + p05, 1e-6)

Enter fullscreen mode Exit fullscreen mode

The gradient trick works because a face is roughly convex. Under a single dominant light, luminance falls off with the cosine of the angle between surface normal and light — so the mean image gradient across the face points, near enough, along the projected light direction. It fails on flat overcast lighting (no meaningful gradient, itself a useful answer) and on strong backlighting. Good enough to route decisions.

Now you have a comparison, and it tells you what to do:

delta_azimuth = |dir_A - dir_B|

  < 30 deg  -> compatible; harmonize color only
  30-90 deg -> relight the weaker-key subject toward the stronger one
  > 90 deg  -> conflicting keys; do NOT composite as-is. Pick one as
               authoritative and regenerate the other's shading, or
               choose a scene (overcast) that flattens both.

Enter fullscreen mode Exit fullscreen mode

That last branch matters more than any algorithm. If one photo is backlit sunset and the other front-flash indoors, no harmonization pass saves it. The correct engineering answer is to detect the conflict and change the target scene, not grind harder on blending.

Harmonization: match statistics in a perceptual space, not RGB

The classic move is Reinhard-style statistics transfer: shift and scale each channel so the source’s mean and standard deviation match the target’s. Done in RGB it wrecks skin tones, because RGB channels are heavily correlated and you rotate hue while trying to fix brightness. Do it in a decorrelated, roughly perceptual space — Lab or Ruderman’s lαβ:

def match_stats(src_lab, ref_lab, mask):
    out = src_lab.copy()
    for c in range(3):
        s, r = src_lab[..., c][mask], ref_lab[..., c]
        sd_s = s.std() or 1e-6
        out[..., c] = (src_lab[..., c] - s.mean()) * (r.std() / sd_s) + r.mean()
    return out

Enter fullscreen mode Exit fullscreen mode

Two refinements:

Clamp L harder than a and b. Chroma can move a lot before it looks wrong; luminance cannot. Rescaling L aggressively flattens facial modeling into that waxy, cut-out look. Keep L’s scale factor near 1.0 and let a/b do most of the correction.

Match on skin only, apply globally. Compute statistics from the skin mask — hair and clothing have wildly different reflectances and drag your means around — but apply the transform to the whole subject. Skin-only application is a classic bug: a correctly-toned face on a body still lit by the original room.

Shadows are where composites die

Color you can fix. Contact shadows you have to produce. Two subjects on the same ground plane cast shadows in the same direction with consistent softness, and occlude each other at contact points. Without cast shadow, subjects float — the most common tell in generated two-person images.

Minimum viable shadow: project the silhouette along the estimated azimuth, scale by an assumed light elevation, blur proportionally to the hardness estimate, and multiply (never subtract) into the background. Multiply preserves underlying texture; subtraction crushes it to a flat gray patch.

The hard part is consistency of softness. A hard-edged shadow under one person and a soft gradient under the other is as jarring as mismatched color, and it gets skipped because it’s invisible in isolation and glaring in comparison.

Getting the full chain right end-to-end — estimate, decide compatibility, relight, then generate contact shadows — is exactly the pipeline behind DuoPortrait, which composites two people from separate photos; running real user uploads through it made clear that the estimator’s conflict detection branch earns its keep more than the harmonizer does. Most failures aren’t bad blending. They’re two photos that should never have been blended in the first place.

So the rule I’d hand anyone starting this: measure lighting compatibility before committing to a composite, and treat incompatibility as a routing decision rather than a problem to blend away. A pipeline that says “these two photos won’t work together, here’s why” produces better output than one that always returns something.