PhysTwin

PhysTwin turns one real video clip into a metric 3D motion observation, fits an executable rigid-body scene with NVIDIA Newton / Warp on CUDA, then resimulates that scene under different gravity.

Video
  → SAM2 tracking
  → DA3 camera
  → known-radius sphere reconstruction
  → PhysicalMotionObservation
  → Newton + Warp inverse fit
  → PhysicalScene
  → counterfactual rollout

The current evaluation is IRIS Falling_ball/big/01.mp4, 1.40-1.90 s, 16 frames. Scale comes from the measured ball radius, 0.11 m. DA3 supplies camera intrinsics and poses. DA3 depth is not used as metric moving-ball depth.

What I built

• SAM2 mask tracking from a click prompt.
• Camera K / pose reconstruction through Depth Anything 3.
• Metric lift from a known 0.11 m radius.
• Project-owned contracts: observation, physical scene, and simulated world state.
• Newton 1.5.1 / Warp 1.16 simulator-in-the-loop gravity fit on CUDA.
• Moon-gravity counterfactual from the fitted scene (1.62 m/s²).
• React / Three.js viewer for committed artifacts.

The optimizer search is CPU/NumPy. Each objective evaluation invokes the GPU simulator. The repo does not include custom @wp.kernel CUDA kernels. C++20 V1 is retained image-space CTest evidence. The IRIS gravity result uses Newton/Warp.

Measured result

IRIS falling-ball clip, NVIDIA GeForce RTX 4080 SUPER:

• Recovered gravity: 9.91 m/s² versus IRIS 9.81 m/s² (1.0% error)
• Trajectory RMSE: 0.064 m (normalized 0.107)
• 88 Newton evaluations, 18.6 s committed fit
• Evaluation gate: acceptable, not ideal (nRMSE > 0.10)
• IRIS gravity was unread during the fit

The purple Moon path is the same fitted drop with gravity set to 1.62 m/s². It was not observed.

Limits

• Monocular reconstruction needs a measured length or radius.
• Physics is one rigid sphere plus gravity. No contact, drag, or articulated humans.
• Counterfactuals are simulated hypotheses, not observations.
• Evaluated on one IRIS falling-ball clip.
• An earlier IRIS pendulum experiment completed a Newton run and failed as physics (RMSE 0.616 m). That residual is kept in the repo.