Computer games and virtual reality systems rely on spatial audio to provide the desired level of realism and immersion. If a player/user is positioned in a virtual room with other sounding objects, the realism of perception, localization of these sound sources, and the perception of the virtual environment requires auditory feedback from the modeled virtual environment. Similarly expressive synthetic audio is necessary to improve the realism of such applications. Sound objects in computer games are typically canned (i.e. pre- recorded) sounds, also known as foley sounds. This approach is useful, but insufficient in that repetition of canned sound effects cause boredom and reduce realism. This course aims to provide students with the necessary background and skills together with an in-depth understanding of virtual and synthetic audio.

The course will first aim to bring all students to the same level on fundamental concepts of digital audio, including, sampling, frequency-domain representations, filtering, and time-frequency analysis. Human auditory perception, including concepts such as critical bands, perception of pitch, timbre, loudness, rhythm, and texture; as well as masking will be covered. Spatial auditory perception will be covered specifically including its aspects relevant to game audio such as binaural hearing, localization of sound sources in 3D space, and distance perception. Concepts related to room acoustics modeling, simulation and auralization will be covered from game audio perspective. Artificial reverberators which are strongly related to perception-based simulation of room acoustics will be covered in detail. Most important existing and emerging spatial audio reproduction techniques will be covered. Subjective evaluation of spatial audio systems and algorithms including experiment design, ITU recommendations, and statistical analysis of results will be discussed. Coding of spatial audio for use in computer games and virtual reality applications will be given with existing MPEG standards. Discussion of sound synthesis will include a brief review of

existing speech, musical instrument, and environmental sound synthesis methods. Special emphasis will be given to expressive synthesis and morphing techniques that allow using the same sound synthesis method to generate perceptually similar but not identical sounds. Finally, existing game audio APIs will be reviewed.