Re: Objective measurements of portable players using df-metric
Reply #5 – 2018-09-14 21:04:36
I like the test instrumentation. 32 ohm load will reveal a lot of problems with typical hardware in my experience. However, I see a few potential confounders: 1) You're using 44.1khz when almost anything android (which is an awful lot of things these days) is going to default to 48khz and then very often resample. Android has a relatively high quality windowed sinc resampler, but if you look at very broadband signals like the white noise example, you're probably going to detect the effect of the resampler and penalize pretty heavily when in reality the output is likely to be fine for real music. Trying two different sampling rates, or else detecting resampling may be more accurate. Certainly if you tested some of the devices you are proposing in rockbox, you would see very different results depending on the sampling rate setting in our software. Still most of audio files that we listen are 44.1. That was the reason. Internal resampling is a good question. I'm thinking about small research of various resamplers in order to see how different signals are being distorted by them. My current understanding is that Df values for resampling are significanly higher than for other problems in audio circuits. One example of Df measurements for resampling is in my old article [http://soundexpert.org/news/-/blogs/visualization-of-distortion] Fig.7. For glockenspiel audio Df values are -74dB and -84dB depending on resampler. So I think this will not affect resulting Df values greatly as the latter are around -30dB for real-life signals and audio circuits of portable players. I hope the research of resamplers using df-metric will help to understand this better. And particularly, what signals are most sensitive to it. What resamplers do you recomend for the test?2) The square and triangle tests looks questionable to me. I see you're doing this correctly and generating band limited square/triangle signals (e.g. via fourier synthesis), but they're still going to have all sorts of weird interactions with the high pass filtering, imaging filter and resampler that probably have no relevance to real audio quality. In fact, I suspect that you are seeing this in many of your tests given the large difference between the triangle and square results. Is there a reason to consider such artificial signals at all? Triangle signals are important for my another psychoacoustic research (don't want to dig into details now). Square wave was intended for measuring slew rate but, to be honest, I didn't touch the issue yet.3) The use of a common scale is a little confusing. -6dB on the -92dbfs test is a much better result than it would be on the 1 khz sin result, but the average person is not going to grasp that from the way the data is presented. Showing the -92dBfs results relative to the noiseless case (so that the -6dB value maps close to 98 dB) would make it a lot more clear what that test means. Yes, in df-metric many Df values have sense in comparison to another ones (sine 12.5k and DFD for example). Dependency of Df values on the type of signals is inherent in df-metric and this must be accounted while interpreting results of measurements in any case. Taking into account that new signals will be added in future and probably some current ones - removed, I think it's better just to show values that were measured (without further complications). At this stage of the research at least. 4) The bandlimiting you're using isn't really explained, and I didn't want to register to download the utility you're using for noise generation. In general, I like the RMAA approach of doing things like frequency and IMD sweeps rather than just testing broadband signals like white noise because effects around 100-10,000Hz matter vastly more than very high or very low frequencies. If you want to do broadband noise measurements anyway, I think you should band limit them to some set of relevent ranges and test independently (e.g. 50-500 Hz, 1000-5000 Hz) and leave out frequencies above 15kHz entirely. Otherwise you are going to end up measuring a lot of things that do not matter such as the high pass filter transition band and the imaging filter while obscuring things that do matter (e.g. lower frequency bass roll off). IMD sweeps and noise divided by ranges look interesting. At the moment the only heavily band-limited noise signal, used in testing, is BS EN 50332-1. Usually technical signals help to reveal some particular features of audio device performance (often without clear understanding how the feature affect listening experience). As the measurement procedure is always the same, the only two questions, when adding new tech. signals, are: what exactly we need to test and what signal can help to do this. For example I expect that with wide-spreading of VR-audio the very important audio parameter will be phase accuracy and some wide-band or high-frequency noise signals could help to measure it. I also recommend aggressively band limiting the real music samples as well just to make sure things like transient bits of clipping (in the recording, not your hardware) do not introduce artifacts. I would prefer to avoid preprocessing music material before feeding into DUT as it is against one of the main features of df-metric - testing as close as possible to real-life scenario of DUT use. Actually the tracks used for testing are very different by sound, have different RMS levels and frequency ranges. At least two of them have severe clipping. I plan to publish diffrograms of all sound tracks for all tested devices. May be influence of clipping better to examine comparing diffrograms of different tracks and different devices ... And most of other tracks are clipping-free. What happens if you loopback the the recording device by the way? How close to 16 bit limited are the individual tests? This would serve as a great control. You mean to test the recording device with the same set of test signals? For the purpose I would need some playback device of high quality (lab. quality). My recording device has playback capability but its quality is mediocre (as it is less important in recorders), so loopback will show only quality of playback+recording chain. Or may be I didn't understand your thought ... Thanks for your detailed comments and proposals, very helpful and insightful indeed.