1[Home](README.md) 2 3# Automated Testing 4 5OboeTester can be used to measure the round trip latency and glitches. 6It can be launched from a shell script by using an Android Intent. 7 8Before running the app from an Intent, it should be launched manually and a Round Trip Latency test run. Then you can give permission for using the microphone to record the looped back sound, and give permission to write to external storage for saving the test result. 9 10## Requirements 11 12All tests require: 13 14* host computer 15* ADB installed 16* ADB USB cable 17 18The latency, glitch and data_paths tests also need: 19 20* [loopback adapter](https://source.android.com/devices/audio/latency/loopback) 21* a 3.5 mm jack on the phone* 22 23\* If you don't have a 3.5 mm jack then you can use a USB-C to 3.5mm adapter. 24In order to use ADB at the same time you will also need a USB switching device 25or use ADB/Wifi. 26 27## Start App from Intent 28 29The app can be started by sending a Start comment to the OboeTester class. 30The app will run and the results will be written to a file. 31 32 adb shell am start -n com.mobileer.oboetester/.MainActivity {parameters} 33 34String parameters are sent using: 35 36 --es {parameterName} {parameterValue} 37 38For example: 39 40 --es test latency 41 42Integer parameters are sent using: 43 44 --ei {parameterName} {parameterValue} 45 46For example: 47 48 --ei buffer_bursts 8 49 50Boolean parameters are sent using: 51 52 --ez {parameterName} {parameterValue} 53 54For example: 55 56 --ez use_input_presets false 57 58## Parameters 59 60There are two required parameters for all tests: 61 62 --es test {latency, glitch, data_paths, input, output} 63 The "latency" test will perform a Round Trip Latency test. 64 It will request EXCLUSIVE mode for minimal latency. 65 The "glitch" test will perform a single Glitch test. 66 The "data_paths" test will verify input and output streams in many possible configurations. 67 The "input" test will open and start an input stream. 68 The "output" test will open and start an output stream. 69 70 --es file {name of resulting file} 71 72The file will be stored in a directory that can be written by OboeTester without any special permissions. 73This is typically "/storage/emulated/0/Android/data/com.mobileer.oboetester/files/". 74 75There are some optional parameter in common for all tests: 76 77 --ef volume {volume} // normalized volume in the range of 0.0 to 1.0 78 --es volume_type {"accessibility", "alarm", "dtmf", "music", "notification", "ring", "system", "voice_call"} 79 Stream type for the setStreamVolume() call. Default is "music". 80 --ez background {"true", 1, "false", 0} // if true then Oboetester will continue to run in the background 81 82There are several optional parameters in common for glitch, latency, input, and output tests: 83 84 --ei buffer_bursts {bursts} // number of bursts in the buffer, 2 for "double buffered" 85 --es in_api {"unspecified", "opensles", "aaudio"} // native input API, default is "unspecified" 86 --es out_api {"unspecified", "opensles", "aaudio"} // native output API, default is "unspecified" 87 --es in_channel_mask {"mono", "stereo", "2.1", "tri", "triBack", "3.1", "2.0.2", "2.1.2", "3.0.2", "3.1.2", "quad", "quadSide", "surround", "penta", "5.1", "5.1Side", "6.1", "7.1", "5.1.2", "5.1.4", "7.1.2", "7.1.4", "9.1.4", "9.1.6", "frontBack"} 88 --es out_channel_mask {"mono", "stereo", "2.1", "tri", "triBack", "3.1", "2.0.2", "2.1.2", "3.0.2", "3.1.2", "quad", "quadSide", "surround", "penta", "5.1", "5.1Side", "6.1", "7.1", "5.1.2", "5.1.4", "7.1.2", "7.1.4", "9.1.4", "9.1.6", "frontBack"} 89 --ei in_channels {samples} // number of input channels, default is 2. This is ignored if in_channel_mask is set. 90 --ei out_channels {samples} // number of output channels, default is 2. This is ignored if out_channel_mask is set. 91 --ei sample_rate {hertz} 92 --es in_perf {"none", "lowlat", "powersave"} // input performance mode, default is "lowlat" 93 --es out_perf {"none", "lowlat", "powersave"} // output performance mode, default is "lowlat" 94 --es out_usage {"media", "voice_communication", "alarm", "notification", "game"} // default is media 95 --es in_sharing {"shared", "exclusive"} // input sharing mode, default is "exclusive" 96 --es out_sharing {"shared", "exclusive"} // output sharing mode, default is "exclusive" 97 --ez in_use_mmap {"true", 1, "false", 0} // if true then MMAP is allowed, if false then MMAP will be disabled 98 --ez out_use_mmap {"true", 1, "false", 0} // if true then MMAP is allowed, if false then MMAP will be disabled 99 100There are some optional parameters in common for glitch, input, and output tests: 101 102 --ei duration {seconds} // glitch test duration, default is 10 seconds 103 104There are several optional parameters for just the "glitch" test: 105 106 --ef tolerance {tolerance} // amount of deviation from expected that is considered a glitch 107 // Range of tolerance is 0.0 to 1.0. Default is 0.1. Note use of "-ef". 108 // input preset, default is "voicerec" 109 --es in_preset ("generic", "camcorder", "voicerec", "voicecomm", "unprocessed", "performance"} 110 111There are several optional parameters for just the "data_paths" test. Note the Note the use of "-ez" for the boolean parameters. 112 113 --ez use_input_presets {"true", 1, "false", 0} // Whether to test various input presets. 114 --ez use_all_sample_rates {"true", 1, "false", 0} // Whether to test all sample rates. Note use of "-ez". Default is false 115 --ei single_test_index {testId} // Index for testing one specific test 116 117These parameters are used with the "data_paths" test starting with v2.5.11. 118 119 --ez use_input_channel_masks {"true", 1, "false", 0} // Whether to test the reported input channel MASKS. Default is false. 120 --ez use_all_channel_counts {"true", 1, "false", 0} // Whether to test all the supported channel COUNTS. Default is true. 121 --ei output_channel_masks_level {0, 1, 2} // Whether to test NONE=0, SOME=1, or ALL=2 channel masks. Default is false. 122 123These parameters were used with the "data_paths" test prior to v2.5.11. 124 125 --ez use_input_devices {"true", 1, "false", 0} // Whether to test various input devices. 126 --ez use_output_devices {"true", 1, "false", 0} // Whether to test various output devices. 127 --ez use_all_output_channel_masks {"true", 1, "false", 0} // Whether to test all output channel masks. Default is false 128 129There are some optional parameters for just the "output" test: 130 131 --es signal_type {sine, sawtooth, freq_sweep, pitch_sweep, white_noise} // type of sound to play, default is sine 132 133For example, a complete command for a "latency" test might be: 134 135 adb shell am start -n com.mobileer.oboetester/.MainActivity \ 136 --es test latency \ 137 --ei buffer_bursts 2 \ 138 --ef volume 0.8 \ 139 --es volume_type music \ 140 --ei buffer_bursts 2 \ 141 --ei out_channels 1 \ 142 --es out_usage game \ 143 --es file latency20230608.txt 144 145or for a "glitch" test: 146 147 adb shell am start -n com.mobileer.oboetester/.MainActivity \ 148 --es test glitch \ 149 --es in_perf lowlat \ 150 --es out_perf lowlat \ 151 --es in_sharing exclusive \ 152 --es out_sharing exclusive \ 153 --ei buffer_bursts 2 \ 154 --ei sample_rate 48000 \ 155 --ef tolerance 0.123 \ 156 --ei in_channels 2 \ 157 --es file glitch20230608.txt 158 159or for a "data_paths" test: 160 161 adb shell am start -n com.mobileer.oboetester/.MainActivity \ 162 --es test data_paths \ 163 --ez use_input_presets true \ 164 --ez use_input_devices false \ 165 --ez use_output_devices true \ 166 --es file datapaths20230608.txt 167 168## Interpreting Test Results 169 170Test results are simple files with "name = value" pairs. 171After running the test you can determine where the results file was written by entering: 172 173 adb logcat | grep EXTFILE 174 175The test results can be obtained using adb pull. For example: 176 177 adb pull /storage/emulated/0/Android/data/com.mobileer.oboetester/files/glitch20230608.txt . 178 179The beginning of the report is common to latency and glitch tests: 180 181``` 182build.fingerprint = google/cheetah/cheetah:14/MASTER/eng.philbu.20230518.172104:userdebug/dev-keys 183test.version = 2.5.1 184test.version.code = 72 185time.millis = 1686156503523 186in.channels = 2 187in.perf = ll 188in.preset = voicerec 189in.sharing = ex 190in.api = aaudio 191in.rate = 48000 192in.device = 22 193in.mmap = yes 194in.rate.conversion.quality = 0 195in.hardware.channels = 2 196in.hardware.sampleRate = 48000 197in.hardware.format = i32 198in.burst.frames = 96 199in.xruns = 0 200out.channels = 1 201out.perf = ll 202out.usage = game 203out.contentType = music 204out.sharing = ex 205out.api = aaudio 206out.rate = 48000 207out.device = 3 208out.mmap = yes 209out.rate.conversion.quality = 0 210out.hardware.channels = 2 211out.hardware.sampleRate = 48000 212out.hardware.format = float 213out.burst.frames = 96 214out.buffer.size.frames = 192 215out.buffer.capacity.frames = 1920 216out.xruns = 0 217``` 218 219### Latency Report 220 221Each test also adds specific value. For "latency". If the test fails then some values will be unavailable. 222 223Here is a report from a good test. The '#' comments were added for this document and are not in the report. 224 225``` 226confidence = 0.892 # quality of the latency result between 0.0 and 1.0, higher is better 227result.text = OK # text equivalent of the result 228latency.msec = 23.27 # round trip latency in milliseconds 229latency.frames = 1117 # round trip latency in frames 230latency.empty.msec = 19.27 # round trip latency if the top output buffer was empty 231latency.empty.frames = 925 # same but translated to frames 232rms.signal = 0.03142 # Root Mean Square of the signal, if it can be detected 233rms.noise = 0.00262 # Root Mean Square of the background noise before the signal is detected 234correlation = 0.975 # raw normalized cross-correlation peak 235timestamp.latency.msec = 10.35 # latency based on timestamps 236timestamp.latency.mad = 0.05 # Mean absolute deviation 237timestamp.latency.count = 12 # number of measurements 238reset.count = 1 # number of times the full duplex stream input underflowed and had to resynchronize 239result = 0 # 0 or a negative error 240``` 241 242Here is a report from a test that failed because the output was muted. Note the latency.msec is 243missing because it could not be measured. 244 245 rms.signal = 0.00000 246 rms.noise = 0.00048 247 reset.count = 3 248 result = -96 249 result.text = ERROR_CONFIDENCE 250 confidence = 0.009 251 252### Glitch Report 253 254Here is a report from a good test. The '#' comments were added for this document and are not in the report. 255 256 tolerance = 0.123 257 state = LOCKED 258 unlocked.frames = 2528 # frames spent trying to lock onto the signal 259 locked.frames = 384084 # frames spent locked onto a good signal with no glitches 260 glitch.frames = 0 # frames spent glitching or recovering from a glitch 261 reset.count = 208 # number of times the full duplex stream input underflowed and had to resynchronize 262 peak.amplitude = 0.057714 # peak amplitude of the input signal, between 0.0 and 1.0 263 signal.noise.ratio.db = 96.3 264 time.total = 9.96 seconds # close to your specified duration 265 time.no.glitches = 9.96 # time we have been running with no glitches 266 max.time.no.glitches = 9.96 # max time with no glitches 267 glitch.count = 0 # number of glitch events, actual number may be higher if close together 268 269Here is a report from a test that failed because the output was muted. Note the glitch.count is 270missing because it could not be measured. 271 272 state = WAITING_FOR_SIGNAL 273 unlocked.frames = 0 274 locked.frames = 0 275 glitch.frames = 0 276 reset.count = 1 277 time.total = 9.95 seconds 278 279### Data Paths Report 280 281The report first goes through the info about the specific device before going through input preset tests, 282input devices tests, and output tests. 283Each will show the specific configuration of a test before showing whether it passed or failed. 284At the end of the report, an analysis of the failed tests will be given 285followed by the number of passed, failed, and skipped tests. 286