Everyone’s a Critic: Understanding Audio Content Reviews

Published April 24, 2015

Introduction

Audio content reviews are offered as a service by the Advanced Technology Group (ATG) for game titles in any stage of development—from pre-production to “post-mortem” (that is, after a title has shipped). Similar in concept to ATG performance and networking reviews, audio content reviews offer an opportunity to have a title evaluated by a professional engineer who has an intimate knowledge of not only the technical specifications and capabilities of the target system, but also the broad spectrum of offerings in your title’s genre. During an audio content review, attention is given to objective metrics and to less tangible subjective perceptions. Audio content reviews offer a confidential, external view into the audio implementation of your title. These reviews are not tied to the title certification process; all recommendations are just that—recommendations. Developers may implement or ignore suggestions as they deem appropriate and as their schedules and priorities allow.

This white paper covers the basic methodologies used for audio content reviews to help you better understand what the service offers. Additionally, a developer can implement some of these processes to further evaluate and improve their title’s audio. Many of the questions answered by an audio content review are not platform dependent, and can be used to raise the quality bar and to provide tangible measurements to justify further investment in audio.

If you are interested in pursuing a formal ATG audio content review of your game, speak with your developer account manager. If you do not know who your developer account manager is, contact Games Rep (gamesrep@microsoft.com).

Audio content review process

Audio content reviews are performed on titles at any stage in their development. Reviews focus on offering topical data based on the title’s schedule. If a title is shipping imminently, the review most likely will focus on high priority issues that can be addressed in the remaining time. Conversely, an already released title, or one in early production, might generate callout areas that are more detailed and further-reaching in terms of potential implementation and content creation work.

After a title’s developer account manager has initiated the audio content review process, generally the only requirement is to submit a playable build of the title, a compilation of audio assets (with either an accompanying audio middleware project or a test framework for triggering them), or, depending on the stage of production, an audio specification document. Also appreciated is additional information such as pointers to more complete audio implementation areas of the title, saved games that jump to specific areas of the title where audio evaluation focus is to be placed, and “cheat” codes for invincibility and triggering enemies. For titles in early pre-production, audio specifications, planning documents, sketches, and other rough audio source assets (and playback mechanisms) can be submitted. If the title is based on an existing audio engine, the developer might submit previously released titles (such as prequels) for audio content review to point out potential areas for enhancement.

The review methodology is well defined, rigorous, and can be replicated by game developers who wish to perform their own internal content reviews. Various aspects of the title are explored and reported and, typically, are separated into categories (some of which may not apply to all titles).

Through general gameplay and specific exploration, various quantitative and qualitative questions are asked about each topic. Results are coalesced into an audio content review report. The real-time 7.1 mix (and, where appropriate, stereo downmix) is captured, analyzed, and included in the report as illustrations (both visual and auditory), to assist in monitoring such areas as speaker usage and reference mix levels.

Audio content review reports consist of both commendations and concerns. Areas of outstanding or innovative audio implementation are called out in the commendations section. Suggested areas for further implementation or adjustment are discussed in the concerns section, prioritized by perceived impact on the title.

Audio content review categories

The following sections of this white paper examine each review category, discussing the areas that are evaluated, as well as Boilerplate issuess that are frequently encountered and recommendations for addressing them.

• Overall game audio
• 7.1/5.1 Usage (Xbox One) / Multichannel usage (Windows)
• In-game sound effects
• In-game dialogue
• In-game ambience
• In-game music
• Voice
• Startup/Menus
• Implementation

Overall game audio

Questions

• Is the overall audio effect providing a positive experience for the player?
• Are game-critical sounds always heard distinctly?
• Does the game make good use of dynamic range?

Details

• Mixing: Are all elements well balanced, such that the most important elements can be isolated and understood by the player?
• Dynamic range: Does the game leave enough headroom for the most impactful moments to be louder? Is objectionable clipping present? In general, is there concern with “listener fatigue”?
• Reference levels: Does the game appropriately abide by industry recommendations for playback loudness, and match the Xbox One startup sound, other titles, and general recommended practices for audio level balancing?

Boilerplate issues

• Mixing:Automated mixing “intelligence” remains a challenge for many titles. Even simple notification-driven ducking (amplitude reduction) of categories of sounds when an important sound begins to play can have a great impact here. More extensive and sophisticated practices include the use of compressor/limiters, multiband equalization digital signal processing (DSP) effects, high dynamic range (HDR) audio, and real-time monitoring.
• Dynamic range: A relic of early game audio, where assets needed severely restricted dynamic range for technical reasons, is still encountered occasionally in modern titles. While this varies with genre, games generally seek a mix that allows for substantial dynamic range. This not only allows “loud” parts of the title to have an emotional impact, but also provides room for crucial dialogue to come through in the game mix. In rare cases, titles go to the other extreme, providing a wider range than is practical for players’ typical, less-than-ideal, playback environments.
• Reference levels: See the Xbox One Audio Mixing and Monitoring Best Practices white paper on the Xbox Game Developer, which documents industry standard measurement methodologies and recommended target playback levels. In particular, titles should note the perceived volumes of each initial gameplay transition element, starting with the Xbox One boot sound, through corporate logos, menu sequences, cut-scenes, and gameplay. If users perceive any of these elements as objectionably loud, they will tend to adjust their volume control, and effectively reduce the available dynamic range across the rest of the title. For this reason, consistency is an important goal to strive for, particularly in initial title screens.

7.1/5.1 Usage (Xbox One) / Multichannel usage (Windows)

Xbox and Windows titles alike can address unique speakers for specific sounds, as well as 3D positioning sounds around the user and applying other spatial and environmental effects.

Questions

• Are the speakers used appropriately and effectively?
• Does the soundscape immerse the player in the game?

Details

• Logos: Do they present 7.1 audio?
• Menu: Does the menu system use multichannel audio?
• Cinematics and cut-scenes: Is 7.1 used appropriately and effectively?
• Center channel: Any exclusive uses? Dialogue where appropriate?
• Surround channels: Any exclusive uses? Used aggressively enough?
• LFE channel: Any exclusive uses? Supports sound “weight”?
• Immersion: Do elements of 7.1 soundscape change with player/camera orientation?

Boilerplate issues

• General 7.1 vs. stereo usage: Often gameplay itself uses a nicely rendered 7.1 mix because sounds are being 3D-positioned and thus implicitly use the surround speakers. It is in the opening logos, menus, and cut-scenes that audio reverts to stereo. Particularly because these are the first (for logos and menus) and potentially most scrutinized (for non-interactive cut-scenes) audio experiences the player encounters, drawing them into the game from the beginning with dynamic and aggressive 7.1 usage can be significantly more effective than “saving” it for game play.
• Center channel: Some titles choose to treat the center channel as another 3D-pannable speaker. While a perfectly acceptable choice, the center channel also can be used for unique content treatments. Dialogue is the most traditional element to place in the center, even as a speaker doubling in scenarios where the speech is positioned simultaneously in 3D space. Screen-anchored sounds, such as first-person Foley, weapons, menu, and user interface sounds, can all be placed in the center channel as well. As an added bonus in memory-restrictive scenarios, a sound using the center channel can be a mono rather than a stereo sound for the left and right speakers.
• Surround and back channels: Although surround channels are used by 3D-positioned sounds, using this technique alone often leaves the speakers fairly underutilized, particularly in environments that are thinner on dynamic or moving objects. Ambience and room tone often are placed effectively in the surrounds, as are nonvisual corresponding elements such as wind, rain, and so forth.
• LFE channel: Often the most underused (or incorrectly used) speaker, the low frequency effects (LFE) channel is intended for bass elements. By not placing elements of explosions, impacts, and massive objects in this speaker, you certainly miss a chance to add true impact to your visuals. A common (and potentially problematic) title technique is to simply double sounds meant for the directional speakers in the LFE channel. In addition to creating muddy and ill-defined LFE responses, this technique can lead to issues in home theaters with bass management or bass enhancement technologies. These two technologies frequently do the same thing as doubling main speaker audio in the subwoofer, which leads to further muddiness and distortion. Specifically authored “rumble” tracks can be the most effective use for the channel. Because the LFE channel is dedicated to low frequencies, tracks played on this channel can use exceptionally low sampling rates and thus be quite small.
• Immersion: In addition to effectively using all speakers in a multichannel environment, titles using a free or moving camera should attempt to reflect changes in camera orientation to avoid seemingly “disconnected” sound effects. For example, a stereo wind effect playing in front of players can be distracting if, after turning around, the players still hear the wind effect in their face. The X3DAudio positioning library supports rotation of multichannel emitters. When elements may seem disconnected if statically positioned, titles also may consider quad, 5.1, or 7.1 ambiences, which still can be rotated to give the listener the perception of changing orientation as their camera view is panned.

In-game sound effects

Questions

• Are sound effects realistic, immersive, and consistent?
• Do all visual events have corresponding sound events?
• Do sound effects positively influence player response/reaction?

Details

• Variation: Do common or repetitive sound effects have variations in wave, pitch, volume, and so on?
• Fidelity: Are sound effects perceived as high quality?
• Balance: Are sound effects balanced with each other?
• Physics: Do sounds respond appropriately to game state variables such as force and distance?
• 3D implementation: Do sound effects take advantage of distance and position cues?
• Environmental simulation: Do sounds respond to the environment the player is in? Is reverberation used effectively? Is advanced geometry simulation (such as occlusion or obstruction) used?
• Detail: Is an appropriate noise associated with every visual?

Boilerplate issues

• Variation: One of the greatest challenges in all areas of game audio is providing variation. It may sound obvious, but the most frequently heard sounds require the most variation and, potentially, the most subtlety in the implementation of that variation. In particular, sounds that are triggered in direct response to a user action (such as pressing a button, moving the controller, and so on) tend to become the most repetitive. Variation in audio content can be achieved by using an array of basic and advanced techniques. Simply combining basic (waves, volume, pitch) and advanced (randomized filtering, placing sounds in smaller component sources) techniques produces a range of possibilities.
• Fidelity: Fidelity issues most generally are found in previous generation content pipelines and ports from other platforms, usually manifesting themselves as low sampling rate content or as lossily compressed and decompressed source material. Where possible, maintain the highest fidelity source assets as far into the pipeline as possible. Note that XMA is a perceptual codec, so raw sample rate reduction may be less effective than using more aggressive XMA quality settings. In scenarios where XMA quality settings begin to introduce artifacts, consider using space-saving techniques such as pre-processing (gentle filtering of higher frequencies that could otherwise encode with “warble” artifacts) or other size reduction techniques.
• Balance: Occasionally, some classes of sound effects make it into a game build unmixed—that is, non-attenuated. Audio content reviews can call out sounds that clearly are playing full scale and potentially causing audio saturation or “clipping.”
• Physics: Particularly in object collision systems, reviews evaluate whether sounds change in timbre, volume, and quality based on force applied. Reviews also evaluate sound playback that simulates elements of real-world audio propagation simulation, including appropriate Doppler shifting for high velocities, properly responding to world geometry (see also 3D implementation), and level of detail shifting as distance is adjusted.
• 3D implementation: Titles can explore multiple avenues for sound spatialization that move beyond basic attenuation with distance (that is, sounds become quieter as they are positioned farther away). Potential areas for achievement in game titles include filtering and adjusting a wet/dry mix with distance, appropriate use of LFE to help sell massive objects near the player, and use of spatial cues to help tell the player the location of various objectives. Audio content reviews frequently spot potential 3D implementation “problem” areas such as randomized 3D positioning that can place objects in impossible locations (for example, ambient highlight elements emanating from walls).
• Environmental simulation: Reviews evaluate and frequently call out opportunities for more advanced techniques, including occlusion and obstruction filtering, distance-based filtering, geometry interaction (reverberation, sound diffusion, and portaling), and other techniques that can add realism for minimal processing impact. Directionality and orientation to reverberation (rather than a “preset” that is set once) are other advanced techniques now seen more frequently in titles.
• Detail: Visuals without apparent audio are called out, because these may point to incomplete animation and scripting implementations, incorrectly tagged game triggers, and potential balance issues.

In-game dialogue

Questions

• Does dialogue support the current state of the game?
• Are transitions between lines and characters smooth and seamless?
• Is dialogue always clearly articulated?

Details

• Variation: Do lines of dialogue sufficiently avoid repetition?
• Fidelity: Are character lines intelligible and perceived as high fidelity?
• Balance: Are individual lines of dialogue well balanced with each other?
• Interactive: Does the dialogue respond appropriately to game state?

Boilerplate issues

• Variation: Dialogue repetition is among the most noticed audio issues in titles. This is particularly true when characters have a very small “vocabulary,” and player actions cause them to cycle through it rapidly. Repetition can become a significant concern; repetition noticed within 5 to 10 minutes of gameplay is among the most frequently highlighted.
• Fidelity: With dialogue often taking a significant amount of disc space, aggressive compression is frequently applied. Where this causes dialogue fidelity to suffer, alternative space-saving techniques are recommended. These techniques include using band-limiting of dialog (perceptual codecs such as XMA may compress better when band-limited than when performing brute-force sample rate reduction); and using the xWMA software compression format.
• Balance: Focus is placed on whether interacting characters have unnatural differences in volume levels, as well as if an individual character’s dialogue becomes unexpectedly loud or quiet. Additionally, where present in isolation, dialogue can be measured to ensure it abides by recommended standards for reference playback levels as defined in the Xbox One Audio Mixing and Monitoring Best Practices white paper on the Xbox Game Developer. See also “Reference levels” in the Overall Game Audio section earlier in this paper.
• Interactive: Dialogue implementation is evaluated for realism in several areas: if scripted lines reflect current game conditions and assist the player with accomplishing objectives; if the transitions between lines of dialogue are realistic (characters do not interrupt each other or themselves—speaking multiple lines at once—and delivery/interaction of lines is natural); and if lines are delivered appropriately and dramatically (at a volume and an emotional intensity to match the game state).

In-game ambience

Questions

• Does ambience give the player more information about the environment?
• Is ambience repetitive or organic?

Details

• Variation: Does ambience avoid noticeable looping and repetition?
• Fidelity: Is ambience believable to the player and perceived as high fidelity?
• Balance: Are various ambient elements well balanced?
• Interactive: Does ambience transition smoothly as characters move around and between environments?
• Environmental simulation: Does ambience respond to the environment that the player is in? Is reverberation used effectively? Is advanced geometry simulation (such as occlusion or obstruction) used?

Boilerplate issues

• Variation: Ambient implementations are evaluated for several minutes to note any obvious cadences, rhythms, or other developing artificial patterns that may become objectionable when heard for extended periods. The review evaluates whether the ambience has highlight elements, and whether those elements are sufficiently varied.
• Fidelity: Players are particularly likely to notice fidelity issues in the base environmental tone. High quality ambience often successfully masks lower quality highlight elements, dialogue, and Foley.
• Balance: Generally, an issue associated primarily with dynamically mixed ambiences, the levels throughout the title can be evaluated to ensure reasonable balance, both with foreground and background elements. Perhaps more importantly, ambience is contrasted with plot-advancing dialog played on top of it, and can be highlighted where inadvertently competing.
• Interactive: Attention is particularly focused on transition areas between environments to note how successfully blending is accomplished. The use of player-relative rotational ambience also can be evaluated; the X3DAudio library supports these kinds of multipoint emitters natively.
• Environmental simulation: When focusing on ambience, a frequent issue is the presence of highlight elements in unlikely or impossible locations (wind whistling, for example, while the player is next to a wall). Considering game geometry for ambience playback and impact are important emerging challenges for audio implementation.

In-game music

Questions

• Does music support the current state of the game?
• Are transitions between music states smooth and seamless?

Details

• Variation: Does the score sufficiently avoid repetition?
• Fidelity: Is the score perceived as high fidelity?
• Balance: Are individual music tracks well balanced with each other?
• Interactive: Does the score respond appropriately to game state changes?
• Multichannel: Does the score take advantage of 5.1/7.1 creatively?

Boilerplate issues

• Variation: Score repetition is more prevalent perhaps than even dialogue and sound effects repetition because it is frequently an omnipresent element of gameplay. The presence of verbatim loops for frequent game states (for instance, “battle music”) quickly can become tiring to the player and lead to the user replacing the game’s music score. Recommendations here include composing multiple stitched or crossfaded variations of assorted lengths and musical textures; saving music for crucial moments and considering soundscapes or ambiences for more general, long-term usage; considering musical timeouts; and sufficiently using what music is composed. Rather than starting to play a piece of looped music from the beginning, consider starting from a random location. Music toward the end of the piece, for example, may never be heard.
• Fidelity: As the emotional foundation for the title, a score of low fidelity can be distracting and lack emotional impact. If in-memory size is a concern, evaluate the benefit of streaming and using compression to minimize bandwidth. If on-disc size is a challenge, consider using smaller fragments of music and dynamically stitching them together at runtime; or, depending on the kind of score, a more sample-based music playback system may be appropriate. Audio for attract movies and cut-scenes presents special fidelity challenges as compared to in-game audio; video codecs often offer different audio compression formats.
• Balance: Music tracks used throughout the game are compared for reasonable balance.
• Interactive: Frequently, the most important role for a game score is to correspond appropriately with current game state. In titles with more rudimentary music playback systems, this is not always the case; music may remain energetic long after the last enemy is defeated, or act unresponsive during heavy combat. Audio content reviews can evaluate and present game locations and scenarios wherein the music appears disconnected from the on-screen activity in a manner that is distracting or frustrating. Interactivity also covers more dynamic elements of playback, including allowing important dialog to come through in the mix through musical techniques (such as ducking or temporarily transitioning to less active music).
• Multichannel: Music no longer has to be strictly stereo, though this may be appropriate and creatively desired by some titles. Games with minimal or passive surround mixes may wish to explore music placement in the surrounds and center channel. Diegetic music uses (that is, music that emanates from an in-game source) can certainly take advantage of 3D positional elements just like any other sound element.

Voice

Questions

• Is voice chat well integrated into the overall audio soundscape?

Details

• Audible/Intelligible: Can voice be heard clearly in standard scenarios?
• Integration: Is voice, compared with other sounds, appropriately implemented?

Boilerplate issues

• Audible/Intelligible: While voice communications overall are often implemented by networking rather than by audio development teams, maintaining intelligibility frequently is an audio challenge to address. When other players are talking, titles must ensure that their dialogue—whether over speakers or voice chat headset—is heard clearly by the gamer. Titles that excel in this area use combinations of ducking, frequency filtering, and intelligent playback/culling to keep the game immersive while allowing player communication to succeed.
• Integration: Headset to headset communication often is appropriate and expected in FPS titles where characters are expected to be carrying a two-way radio. Completing the illusion of radio communications, titles often simulate and integrate other radio elements such as squelches and static, and even “failed” communications when jamming or obstacles are between characters. However, in voice chat scenarios where the concept of walkie-talkies is less appropriate, other more appropriate and interesting ways exist to handle voice. For example, can voice be 3D-positioned in the world, as if truly emanating from other players’ visual representations and virtual locations, or can there be a one-to-many system built into the world, where players have to jockey to use a virtual public address system? Audio content reviews can call out achievement in voice chat integration and innovation.

Startup/Menus

Questions

• Does the initial experience prepare the player for game audio?
• Do menus have audible cues to indicate successful navigation?

Details

• Initial volume levels: Are they reasonable compared to startup sound, other titles, and best practices?
• Balance: Are all logos, cinematics, and initial menu audio well balanced?
• Feedback: Does audio provide appropriate feedback for menu UI navigation?
• Load screens: Is prolonged silence avoided?
• Transitions: Does music/ambience avoid abrupt stops when switching from menus to gameplay?
• Easy to adjust audio options (if present)

Boilerplate issues

• Initial volume levels: Game logos and technology trailers/stingers often are authored for other platforms where dynamic range may not be as wide, or where reference levels are unavailable. For suggested levels on Xbox One, see the Xbox One Audio Mixing and Monitoring Best Practices white paper on the Xbox Game Developer. In particular, players typically use the Xbox One boot sound to calibrate their systems. A game logo perceived to be substantially louder can lead to a user adjusting the volume, which effectively reduces the available dynamic range across the rest of the title.
• Balance: Menu system, attract movie, cinematics, and in-game sound levels are evaluated and compared. Frequently, one or more of these is substantially louder than others are and can lead to the player adjusting their system volume and, therefore, reducing available dynamic range.
• Feedback: Menu navigation can use subtle to dramatic elements, but should give the user some idea that their input is being accepted and causing an action to occur.
• Load screens: Given the number of titles that have successfully eliminated prolonged silence during static load screens, it is nearly inexcusable that a current title not use that wait time to set the mood, provide a back story, or otherwise entertain gamers. Taking just one half of a second to load audio from the game disc, even assuming conservative seek and drive throughput, a title can load and use more than a minute of XMA-compressed stereo CD-quality audio.
• Transitions: Even where silence is masked, transitions from menu to gameplay and from level to level of a game should strive for seamless and smooth audio changes. Generally, it is objectionable for a title to clamp off its audio (causing clicks and other artifacts) as a scene changes.
• Easy to adjust audio options: Titles that make audio options available (an aesthetic choice) must ensure that these are easy to change and provide audible feedback to the player. Top Boilerplate issuess include not providing “auditioned” sounds or other feedback mechanisms to accompany audio sliders; defaulting all sliders to a 100 percent value (so making one element louder actually requires turning down everything else); only providing volume sliders at the main game menu, rather than as an in-game option, when adjustment most likely is requested by the user.

Implementation

Questions (most effectively evaluated when source assets and audio engine code are provided for evaluation)

• Does the title make use of the hardware and software resources at its disposal?
• Are there significant performance bottlenecks in the audio graph?

Details

• Creativity: Is audio used as a gameplay device? As a reward?
• Responsiveness: Do sounds sync with visuals? Does player feel like audio is connected with actions?

Boilerplate issues

• Creativity: Since this topic, by its very nature, is objective, most comments in this area are limited to positive highlighting of unusual and innovative audio implementation choices, including the use of tight audio integration with elements of gameplay, physics, AI, and user customization.
• Responsiveness: Perceived latency is not a commonly seen issue with current generation titles. When encountered, it often highlights unresponsiveness in other elements of gameplay or game audio engine design issues. However, a sound’s connection to gameplay frequently is tied directly to the dynamic and interactive elements of that sound. Integration with the environment (such as reverberation or geometry interaction) and game state (DSP manipulation of source assets) become the crucial elements for marrying sounds to the virtual world in which they are expressed.

Summary

This white paper presents a process for evaluating what can be a very objective element of interactive entertainment: their audio implementations. With a detailed list of questions and areas of inquiry, an ATG engineer can evaluate your title’s audio implementation and provide a knowledgeable and impartial third-party perspective. Again, addressing the suggestions and recommendations presented in an audio content review is strictly voluntary. We encourage you to investigate this service, and perhaps more importantly, for game audio directors, designers, producers, and QA engineers to consider using the above topics and questions as a guide for discussion when developing and implementing their audio vision for a project.

If you are interested in pursuing a formal ATG audio content review of your game, speak with your title’s developer account manager. If you do not know who your development account manager is, or if you have questions or comments about the topics presented or referenced here, you can submit an inquiry to the Microsoft Third-Party Account Management team at gamesrep@microsoft.com.