XR video pipeline engineer: stereoscopic camera to VR headset, low latency
Budget: $25 – $50 CAD
We’re building a prototype for robot teleoperation where an operator wears a VR headset and sees the robot scene through a stereoscopic VR camera with very low latency and strong depth perception—as close to “being there” as possible.
Goal
Phase 1 (this project): Get a working end-to-end live stream from an existing stereoscopic VR camera to a VR headset with measurably low latency and acceptable visual quality. A custom app is fine (Unity/Unreal/native). We want something we can put on and experience.
Why this is hard
We care about:
• Motion-to-photon latency
• Stereo sync (left/right alignment)
• Stable framerate + low jitter
• Headset-friendly rendering (proper stereo projection / distortion handling depending on platform)
⸻
Scope (Phase 1)
You will:
• Propose the lowest-latency architecture (e.g., WebRTC / GStreamer / native RTP/UDP / NVENC / hardware decode).
• Implement an end-to-end pipeline:
• Camera capture (stereo)
• Encoding + transport (LAN to start; internet optional)
• Decode + render in VR headset
• Provide:
• A runnable build + setup instructions
• A latency measurement method and results (even if approximate at first)
• Recommendations for improvements / next steps
We’re fine starting on a local network first. Reliability + experience matters, but latency is priority #1.
Future Roadmap (optional / follow-on)
If Phase 1 goes well, Phase 2+ could include:
• Selecting better cameras and placement for teleop depth perception
• Integrating neck movement / pose and camera stabilization
• Adding extra streams (e.g., wrist cameras) and compositing them into the VR app UI
• QoS tuning, bitrate adaptation, foveated strategies, etc.
Goal
Phase 1 (this project): Get a working end-to-end live stream from an existing stereoscopic VR camera to a VR headset with measurably low latency and acceptable visual quality. A custom app is fine (Unity/Unreal/native). We want something we can put on and experience.
Why this is hard
We care about:
• Motion-to-photon latency
• Stereo sync (left/right alignment)
• Stable framerate + low jitter
• Headset-friendly rendering (proper stereo projection / distortion handling depending on platform)
⸻
Scope (Phase 1)
You will:
• Propose the lowest-latency architecture (e.g., WebRTC / GStreamer / native RTP/UDP / NVENC / hardware decode).
• Implement an end-to-end pipeline:
• Camera capture (stereo)
• Encoding + transport (LAN to start; internet optional)
• Decode + render in VR headset
• Provide:
• A runnable build + setup instructions
• A latency measurement method and results (even if approximate at first)
• Recommendations for improvements / next steps
We’re fine starting on a local network first. Reliability + experience matters, but latency is priority #1.
Future Roadmap (optional / follow-on)
If Phase 1 goes well, Phase 2+ could include:
• Selecting better cameras and placement for teleop depth perception
• Integrating neck movement / pose and camera stabilization
• Adding extra streams (e.g., wrist cameras) and compositing them into the VR app UI
• QoS tuning, bitrate adaptation, foveated strategies, etc.