slperez/LP-Distilled0.1-ONNX

LP-Distilled0.1 ONNX

LP-Distilled0.1.onnx is an experimental/proof-of-concept browser-oriented distillation of a LivePortrait-style portrait animation pipeline. It takes a static portrait crop, a LivePortrait appearance feature volume, normalized head-pose controls, and normalized expression controls, then renders a 256x256 RGB animation frame.

This release is intended for interactive portrait-animation research and demos. It is not a full replacement for the original LivePortrait pipeline and should not be used in production.

Model Details

Field	Value
Model file	LP-Distilled0.1.onnx
Format	ONNX
ONNX opset	17
ONNX IR version	8
Size	74,573,121 bytes, about 71.1 MiB
SHA-256	335F98B44E5FAB3577B15C3EC53EDDD39BE228F072D98D7CE11232BCF1C42AFD
Output resolution	256x256
Teacher	Internal LivePortrait-based teacher pipeline
Runtime target	Browser ONNX Runtime Web / WebGPU, with WASM fallback where needed

The released ONNX graph is a fused two-stage model:

Stage1: source portrait + feature_3d + pose controls -> posed neutral RGB
Stage2: Stage1 RGB + same source feature_3d + expression controls -> final RGB

The fused graph routes the Stage1 rgb output into Stage2's image input. The external caller only has to feed the original aligned portrait, the source feature_3d, and the two control vectors.

Inputs And Outputs

Name	Shape	Type	Description
feature_3d	[1, 32, 16, 64, 64]	float32	LivePortrait appearance feature volume extracted once from the aligned source portrait.
pose_control_vector	[1, 3]	float32	Normalized [yaw, pitch, roll] controls.
input_rgb	[1, 3, 256, 256]	float32	Aligned source portrait crop in CHW RGB format.
expression_control_vector	[1, 30]	float32	Normalized expression, blink, eye-drift, mood, and viseme controls.
rgb	[1, 3, 256, 256]	float32	Rendered output frame in CHW RGB format.

The model does not include face detection, crop/alignment, or appearance feature extraction. A browser demo can prepare those inputs with compatible LivePortrait-derived ONNX source-preparation models:

retinaface_det_static.onnx
face_2dpose_106_static.onnx
appearance_feature_extractor.onnx

The fused model does not require motion_extractor.onnx.

Control Contract

All controls are normalized numeric values. They are not image-space landmark coordinates.

Pose controls:

pose.yaw
pose.pitch
pose.roll

Expression controls:

kp1.x
kp1.y
kp1.z
kp2.x
kp2.y
kp2.z
kp3.y
kp5.x
kp5.y
kp5.z
kp6.y
kp6.z
kp11.x
kp11.y
kp13.y
kp14.x
kp14.y
kp15.x
kp15.y
kp16.x
kp16.y
kp17.x
kp17.y
kp17.z
kp19.x
kp19.y
kp19.z
kp20.x
kp20.y
kp20.z

The control schema deliberately excludes many LivePortrait keypoint axes. Runtime code should use the exact schema order above rather than assuming keypoint * 3 + axis layout.

Keypoint expression controls are normalized deltas, not absolute keypoints. The training convention was:

raw_delta = clamp(raw_delta, -0.08, 0.08)
normalized_delta = raw_delta / 0.08

Pose controls are also normalized to [-1, 1]. The training convention was:

yaw_norm   = yaw_degrees / 20
pitch_norm = pitch_degrees / 14
roll_norm  = roll_degrees / 14

Those ranges are part of the model contract. Sending much larger values can produce degraded or unstable output.

Pose Vocabulary

The pose model was trained primarily around a finite vocabulary of Catmull-Rom head-pose trajectories, plus single-axis paths. This is intentional: the runtime motion system is expected to follow, stop, reverse, or blend from learned trajectories rather than freely sampling every possible yaw/pitch/roll combination.

Inputs outside the trained trajectory vocabulary may still work, but quality can degrade, especially for extreme rotations or unusual pose/expression combinations.

Limitations

• Output is fixed at 256x256.
• This proof-of-concept model was trained on only a few hundred portrait style images. In adhoc testing it can generalize some, but might not look great, depending on the specific source image.
• Quality depends heavily on source crop/alignment quality.
• The model can blur or distort hair, facial details, accessories, and unusual backgrounds.
• Extreme poses can be softer than neutral or moderate poses.
• Controls are semantic approximations learned from a teacher; they are not a physically meaningful face rig.
• The model card does not assert compatibility with all upstream licenses. Check the relevant LivePortrait and dataset licenses before redistribution or commercial use.

Ethical Considerations

Portrait animation models can be misused to create misleading or non-consensual media. Applications should clearly disclose generated animation, obtain consent for likeness use, and avoid representing outputs as authentic footage.

Example Runtime Flow

1. Load a portrait image.
2. Detect and align the face to a 256x256 RGB crop.
3. Run a compatible LivePortrait appearance feature extractor once:
   input_rgb -> feature_3d
4. For each frame:
   feature_3d
   pose_control_vector
   input_rgb
   expression_control_vector
     -> LP-Distilled0.1.onnx
     -> rgb
5. Draw rgb to a canvas or video frame.

For lip-sync, map phonemes or visemes to the 30-wide expression control vector, then smooth the control curve over time. Directly snapping between discrete viseme vectors will look jerky.

Release Notes

This is an early research release. The current graph is useful for interactive demos, but quality is still evolving. Future releases may change the control schema, model capacity, training data mix, or source-prep assumptions.