Immersive Multimedia Technologies

Immersive 3D video and audio rendering technologies described in the Google Starline paper [Starline_ref], as well as volumetric audio-visual rendering (e.g. 3D video holography, 3D surround audio) recently published in the MPEG-I standards (Motion Picture Expert Group – Immersive) will be revisited.

This course presents techniques that are image-based (e.g. giving a perspective 3D illusion by blending 2D images) and is therefore complementary to INFO-H502 that focuses on 3D graphics modeling. Both approaches can be used in VR applications like the metaverse, but the former is preferred for free navigation in movies.

Understand volumetric rendering techniques based on images (and video), as well as discrete sound sources.

Required and Corequired knowledge and skills

C/C++ programming skills are recommended, though the exercises can be done in Python only (but the case study examples will mostly be in C, often without object-oriented concepts).

Theory lessons will explain the main ingredients of immersive 3D audio-visual rendering technologies, complemented with hands-on exercises for mastering their basics.

References, bibliography, and recommended reading

[Starline_ref] Jason Lawrence, et. al., “Project Starline: A high-fidelity telepresence system,” ACM Trans. Graph., Vol. 40, No. 6, Article 242, December 2021, https://doi.org/10.1145/3478513.3480490

[VR_imm_video_ref] G. Lafruit, M. Teratani, “Virtual Reality and Light Field Immersive Video Technologies for Real-World Applications,” Institution of Engineering & Technology, ISBN 978-1785615788, 2022.

Course notes

• Syllabus
• Université virtuelle

Additional information

Slides and notes inspired by [Starline_ref], [VR_imm_video_ref] and their bibliographic references, as well as video clips for some subjects.

Contacts

Prof. Gauthier Lafruit, LISA-VR

Campus

Solbosch

Method(s) of evaluation

• Oral examination

Oral examination

A report on the exercises (to be submitted by the exam period), as well as an oral exam on the theory part.
Oral examination about the theory, with punctual questions on the exercises report.
Except for using a copy of [Starline_ref], no open book exam, though a single A4 page with formulae is allowed.

Mark calculation method (including weighting of intermediary marks)

40% exercises, 60% theory oral exam

Language(s) of evaluation

• english
• (if applicable french, Dutch )