|
Introduction
The purpose of this article is to provide an approach to test multimedia frameworks and chips on mobile devices. The approach is based along three lines of action:
- Audio and Video Testing
- Identification of the types of bugs common in embedded systems software
- Discussion of the various methods used to find bugs
Some of the approach paths detailed in the sections below have been executed on desktops in a windows environment while some pertain to mobile and handheld devices. The baseline philosophy of testing remains agnostic of the platform under test.
Multimedia Audio Testing
It is extremely difficult to define a term as vague as audio quality; everybody has a different view of the subject. Standardizing external and integrated audio around established technology specifications is very important for mobile audio to succeed. The quality of audio in a mobile device is extremely important for a rich multimedia experience on a handheld device.
Internal audio can affect playback on stereo Bluetooth headphones, Bluetooth headsets, wired headphones, wired headsets and externally amplified players and hence control of audio settings via software is very important from an end-user perspective.
We believe that the multimedia platform (audio portion only) will be used in (but not limited to) the following environments:
- Playback of audio files (MP3, AAC, WAV, WMA, Ogg-Vobis, MIDI)
- Playback of polyphonic ringtones including iMelody, MIDI, MP3, WAV
- Playback of game audio
- Speech Recognition – useful in voice-based dialing
- Conversation Recording on device
- Voice Memo Recording
Aztecsoft’s approach to testing multimedia audio consists of these steps:
Understanding all the APIs pertaining to audio of the multimedia chipset.
Parameterizing what needs to be tested.
Understanding the acceptance criteria.
Determining testing tools that are required for the task.
Understanding automation needs and writing test automation code.
Executing the tests and logging test results.
Assisting in the Acceptance Testing process.
The above approach includes the following:
Audio User Control Testing
The following user audio controls need to be tested:
User Control (Audio) |
Description of Test |
Test Type |
Volume |
Change in levels will impact the output volume measured in decibels (dB) |
Automated |
Equalizer |
Change in frequency of output audio by changing the octave response which is frequency based between 31.25Hz and 16KHz maintaining the SNR level high |
Automated |
Bass Boost |
This setting increases the value of the lower frequencies thus increasing the bass level output of the audio |
Automated |
Mute |
This setting disables voice signal output to the speaker |
Automated |
Microphone Level |
Measurement of sensitivity of onboard microphone for audio capture |
Automated |
Audio Parameter Testing
As the above settings are altered by software, corresponding audio parameters need to be measured to ascertain that the change in audio settings accurately changes the audio output and its characteristics. Audio parameters that need to be measured include:
Audio Parameter |
Description of Parameter under Test |
Test Type |
Dynamic Range (DR) |
The output can accommodate both soft and loud signals if this value is high |
Automated |
Signal-to-Noise Ratio (SNR) |
For freedom from noise, SNR level needs to be high |
Automated |
Total Harmonic Distortion (THD+N) |
For freedom from distortion, THD level should be low |
Automated |
Frequency Response |
This should match the full range of human hearing |
Automated |
Channel-to-Channel crosstalk |
This value needs to be low to minimize leakage between audio channels |
Automated |
Tonal Balance |
This is a measure of balance of sound levels across the full range of audio frequencies |
Automated |
Audio Device State Testing
Audio device states need to be measured via API during the course of audio testing. These are:
Audio Device State |
State Definition |
Playing |
Audio is playing |
Recording |
|
- Foreground |
Normal application is recording |
- Background |
Speech recognition or speech activity detection is running |
Full duplex |
Device is simultaneously playing and recording |
Paused |
File handle is open. Only devices that are playing, foreground recording or in full duplex operation may be paused. Background recording may not be paused. The paused state assumes that a device must transition to the resumed state rapidly. No audio samples may be lost between the device is paused and later resumed. |
Closed |
No file handle is open |
Audio Device Driver DRM
With the proliferation of audio streams, podcasts, and digital-music capable mobile devices, there is a growing adoption of Digital Rights Management. DRM tags are included in audio content. If the audio device driver does not support DRM, then it is difficult to playback audio content that is intended for trusted devices. Aztecsoft will test for DRM support using the platform APIs.
- Drivers with DRM support: Devices with DRM-enabled drivers are considered "trusted." The DRM system will play all encrypted and non-encrypted content on these devices.
- Drivers without DRM support: Drivers without DRM signatures can render unencrypted content. These drivers can also play DRM content that does not require a "trusted audio device." However, if the usage rules in the content require that it play only on trusted devices, the DRM system will not play it on a driver without a DRM signature.
Conditions for testing DRM drivers
If DRM support is implemented in the multimedia platform, then it is essential to test the driver to determine full and correctly implementation of the DRM interface as follows:
When requested, the audio driver must disable the ability to capture the stream currently being played back. That is, whenever Content Copy Protection rights are asserted through the DRM interface:
- The driver does not save the unprotected digital content in any non-volatile storage. This may be EEPROM, hard disk, memory card, memory stick, or any other similar storage medium.
- The driver disables the capture multiplexer on an output D/A converter or otherwise prevents loop-back of digital content.
|
When requested, the audio driver must disable the digital audio output on the device. That is, whenever the content asserts Digital Output Disable rights through the DRM interface, the driver must disable all digital audio outputs that could transmit content over a standard interface through a standard interconnection scheme. |
The audio driver must rely only on other components that also contain DRM signatures. The driver must never facilitate the transfer of audio data to any component that does not have a DRM signature.
For instance, if an audio file is transmitted from the mobile device to a Windows-based computer, the driver must utilize the Windows DRM kernel APIs to make the Windows DRM system aware of the movement of digital content. |
The audio device and driver must not include user-selectable options to defeat or subvert the DRM components. Specifically, the driver must not provide user control panels, or other methods of disabling the DRM functions. |
Multimedia Video Testing
Multimedia video quality is easier to assess as compared to audio, which is very subjective.
We believe that the multimedia platform (video portion only) will be used in (but not limited to) the following environments:
- Playback of video files (MPEG-2, MPEG-4, 3GPP)
- Playback of static image files including WBMP, BMP, JPG, GIF, PNG formats
- Playback of video audio
Aztecsoft recommends a testing approach for video multimedia as below:
Understanding of the video-related APIs of the multimedia platform. |
Definition of testing parameters |
Setting Video-specific Acceptance Criteria |
Determining Testing Tools |
Writing Test Cases, Scripts and Automation Code |
Executing Tests and Logging Test Results |
Assisting the Acceptance Testing Process |
Video quality can be determined by measuring many parameters such as resolution, refresh rate, support of video file formats, video memory etc.
Video Resolution Testing
Most mobile video devices are capable of reproducing 160x160 or 320x320 resolution. For the video to appear unaffected by the digitization process, a video capture chip should capture all of the supported resolution samples known in the industry as D1.
The following tests will be conducted to test the Video Resolution:
- Current Resolution Value
- Display of video with all supported resolutions
- Change of Resolution Parameters
- Display of video with changed resolution parameters
Frame Rate Testing
The number of frames that can be refreshed per second determines the frame rate of the playback board on the multimedia chipset. This is usually measured in fps. Higher the fps, better is the display of the video.
Determining frame rate is an automated process that requires code to be written to divide the number of frames rendered with the time it took to render them. One can choose to measure the FPS over as many frames as one wants. If it is measured over just the last frame one can get very accurate FPS but it will vary too fast for most applications' need.
|
