I have come to believe that our testing method is dubious and I urge you to read the responses in this thread and not take my findings for granted.
Hi,
First post, but a lengthy one. I have been a lurker for some time and after spending several days on this test which I documented extensively, I decided to make a write up and share it with you guys.
I hope you enjoy reading this as much as we had during the tests. But above all, I hope you find it useful too.
Enjoy!
Quality Bitrate Test
Song used for the test: Holly Cole’s ‘I Can See Clearly Now’ from the album ‘Don’t Smoke In Bed’.
Why? Because we wanted to know if lossy audio compression is possible without an audible difference compared to the original from an audiophile’s perspective.
Some ripping details:
[blockquote]• Exact Audio Copy v.0.95 beta4 from 21 February 2006
[blockquote]o Secure Mode
o Read Sample Offset Correction of +30
o Enabled overread into Lead-In and Lead-Out
o Enabled Speed Reduction During Extraction[/blockquote]
• Plextor PX-716A, Firmware 1.09
• Core2Duo E6600, 2Gb, not overclocked and configured for stability
• Windows XP Professional, 32bit
• FLAC Encoder 1.1.2
• Ogg Encoder 2.83
• Lame Mp3 Encoder 3.96.1[/blockquote]
The audio rig:
Pre Amp: McIntosh C29
power Amp: McIntosh MC150
CD Player: Luxman D105U
Speakers: Magnepan MG 3.5
Interlinks: Siltech SQ-G3 series
Speaker cable: FTM-G3 series
The codecs:
Encoding details for ‘I Can See Clearly Now’:
Lame --preset Encoding Time Bitrate File size in bytes
Medium 21 143 4.519.605
Standard 18 163 5.151.548
Extreme 17 207 6.537.945
192 10 198 6.275.804
320 10 320 10.143.868
Oggenc2 -q Encoding Time Bitrate File size in bytes
2 2,8 92 2.905.079
4 2,5 122 3.871.195
6 2,5 173 5.496.585
8 2,7 227 7.193.392
10 3,2 447 14.171.303
After encoding the files were decoded using the same proggie that encoded them and burned in random order to an audio CD. The CD contained the original uncompressed wave and decoded FLAC for reference. The track order and relevant info was kept on a separate piece of paper.
The Samples Used
The following samples were used in the test:
1. Ogg Vorbis Q2, 4, 6, 8 & 10
2. Lame Mp3 VBR Standard & Extreme
3. Lame Mp3 CBR 192 & 320
4. Uncompressed Wave and FLAC
We listened to Q0, 64kbps, 128kbps Mp3 and the Mp3 Vbr medium, but after seconds into the tracks we determined it was a waste of time. The quality degradation was too obvious and in our opinion not worth using for high quality audio.
So, this resulted in 9 tracks to compare (Wave and FLAC not counted).
The Testing Stages
The test was taken in 3 stages. In the first stage the uncompressed wave was tested against the original CD. During the second stage the uncompressed wave was used as a reference and played on request of the listener to do a back to back comparison of a compressed track. In this stage the listener was informed that the reference track was playing. Never did the listener know which type of compression was used on the test track.
The reason for this is simple. We had too many tracks to compare and after switching several times the listener becomes fatigued. Even the best trained ear will become confused. Your brain will start to ‘correct’ what you hear, much like with vision. Put on glasses that turn everything upside down and after a while your brain will correct the image and turn it back. Similar things happen with hearing. Therefore, a reference point is needed for the listener during the second stage.
During the third stage, the best Ogg and Mp3 track are played with the reference track. This time however the listener will not know which track is what. He is left completely in the dark to determine which track sounds best.
During every stage, the display of the CD player is not visible to the listener, and the player is controlled by a person not taking part in the test.
Also, the tests were done for each listener separately and have not been able to influence each other.
Stage 1, comparing the uncompressed wave to the original CD
I am glad I did not skip this step because I took for granted that the uncompressed wave and FLAC track on my CD-R were identical to the original. It was almost impossible to believe and a great disappointment for me, but confirmed by all listeners during several blind tests; the uncompressed wave and the FLAC track on the CD-R did NOT sound identical to the original CD. This was a great setback as I hadn’t expected this. I attempted to do binary comparisons of the tracks but I couldn’t figure out what to do as we didn’t have the tools or the time to get into this. I had three guys breathing in my neck to listen to the music and not watch me operate a mouse and keyboard. So we went back to the sofa to do another test to determine if we could continue with the reference track as it was. One of the guys reminded us that several weeks before the laser was replaced in the CD player. Knowing a thing or two about CD players, he explained that it could be possible that the laser was not focused correctly and therefore was not able to read the CD-R correctly. The players’ error correction would be unable to keep up with the errors resulting in the audible difference. We also noted that if the CD-R was playing for a couple of minutes, we could hear static artifacts like it was skipping. The player was ripped apart and we experimented with the focus of the laser. We realize this can’t be done optimally without the proper tools, but after some experimenting we actually found a good setting to continue with. The static was gone and the listeners agreed that the reference track sounded identical to the original CD so we could continue the test.
The Second Stage
What I found most convincing about this test is that throughout this stage, the results were very consistent among the listeners.
Track Compression Sound quality Dynamics
1 Ogg Vorbis Q2 Missed detail in low, midrange is sticky, Dynamics are gone, sounds flat (2D),
high is lacking overall image is messy
2 Ogg Vorbis Q4 Identical to the original Very close to original, slightly compressed,
slightly less open. Consistently considered
the best track with the exception of track 5
3 Ogg Vorbis Q6 Low misses finesse very close to original Lacking in musicality, slightly less open,
considered less than track 2
4 Ogg Vorbis Q8 Identical to the original Very close to original, but slightly less open
5 Ogg Vorbis Q10 Identical to the original Identical to the original
6 Mp3 Vbr Normal Low detail missing, low lacking and Image distorted and sticking, Dynamics are gone,
distorted, high missing and distorted, flat image, completely uncontrolled.
high artificial Considered worse than 1
7 Mp3 Vbr Extreme Near identical to the original, Less Dynamic, slightly sticky,
different but hard to define, less open and vague positioning
high sounds artificial
8 Mp3 Cbr 192 Missing detail in low and high, Image is flat, sticky, very messy,
midrange empty Considered similar to 1 and 6
9 Mp3 Cbr 320 Near identical to original Slightly uncontrolled, compressed,
Considered less than 7
Mp3 track 7 and Ogg track 5 were selected for the blind test in Stage Three.
The Third Stage
This didn’t take long. The Mp3 track was consistently identified and the Ogg track was considered identical to the original.
Because of the interesting results with the Ogg tracks, the listeners agreed to test Ogg tracks again against the original reference, in a blind test. Q8 and Q10 were considered identical to the original and interestingly, Q4 was considered better than Q6.
What is also interesting to note is that Mp3 has been consistently considered less dynamic as Ogg, and sounds messy and uncontrolled. High has consistently been identified as ‘artificial’.
Savvy details:
F:\media\test>lame --preset medium UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 17960 Hz - 18494 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=4) j-stereo MPEG-1 Layer III (ca. 10x) qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
9704/9706 (100%)| 0:21/ 0:21| 0:21/ 0:21| 11.765x| 0:00
32 [ 47] %*
40 [ 12] *
48 [ 23] %
56 [ 34] %
64 [ 17] %
80 [ 71] **
96 [ 751] *******************
112 [2023] **************************************************
128 [2281] ********************************************************
160 [2703] %%****************************************************************
192 [1345] %%%%*****************************
224 [ 307] %%******
256 [ 87] %**
320 [ 6] %
average: 142.8 kbps LR: 246 (2.534%) MS: 9461 (97.47%)
Writing LAME Tag...done
ReplayGain: -4.1dB
F:\media\test>lame --preset standard UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=2) j-stereo MPEG-1 Layer III (ca. 7.3x) qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
9704/9706 (100%)| 0:18/ 0:18| 0:19/ 0:19| 13.702x| 0:00
32 [ 33] *
128 [4113] %*****************************************************************
160 [2634] %******************************************
192 [1550] %%%**********************
224 [ 888] %%%%***********
256 [ 346] %%****
320 [ 143] %**
average: 162.7 kbps LR: 619 (6.377%) MS: 9088 (93.62%)
Writing LAME Tag...done
ReplayGain: -4.1dB
F:\media\test>lame --preset extreme UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz VBR(q=0) j-stereo MPEG-1 Layer III (ca. 5.7x) qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
9704/9706 (100%)| 0:17/ 0:17| 0:17/ 0:17| 14.550x| 0:00
32 [ 33] *
128 [ 175] %***
160 [2121] %%%**************************************
192 [3493] %%%%%%%%%%%%%%%%%%%***********************************************
224 [2019] %%%%%%%%%%%%%%%%%%%%%%%%***************
256 [1228] %%%%%%%%%%%%%%%%%%%%****
320 [ 638] %%%%%%%%%%%**
average: 206.5 kbps LR: 3907 (40.25%) MS: 5800 (59.75%)
Writing LAME Tag...done
ReplayGain: -4.1dB
F:\media\test>lame --preset 192 UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz average 192 kbps j-stereo MPEG-1 Layer III (7.3x) qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
9704/9706 (100%)| 0:09/ 0:09| 0:10/ 0:10| 26.639x| 0:00
32 [ 33] *
40 [ 0]
48 [ 0]
56 [ 1] %
64 [ 1] %
80 [ 0]
96 [ 1] %
112 [ 0]
128 [ 4] *
160 [ 797] %%******
192 [6790] %%%%%*************************************************************
224 [1612] %***************
256 [ 306] %**
320 [ 162] %*
average: 198.2 kbps LR: 635 (6.542%) MS: 9072 (93.46%)
Writing LAME Tag...done
ReplayGain: -3.9dB
F:\media\test>lame --preset 320 UncompressedWave.wav
LAME version 3.96.1 (http://lame.sourceforge.net/)
CPU features: MMX (ASM used), SSE, SSE2
Using polyphase lowpass filter, transition band: 20094 Hz - 20627 Hz
Encoding UncompressedWave.wav to UncompressedWave.wav.mp3
Encoding as 44.1 kHz 320 kbps j-stereo MPEG-1 Layer III (4.4x) qval=3
Frame | CPU time/estim | REAL time/estim | play/CPU | ETA
9704/9706 (100%)| 0:09/ 0:09| 0:10/ 0:10| 26.904x| 0:00
average: 320.0 kbps LR: 6723 (69.26%) MS: 2984 (30.74%)
Writing LAME Tag...done
ReplayGain: -4.1dB
F:\media\test>oggenc2 -q 2 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
"UncompressedWave.ogg"
at quality 2,00
[100,0%] [ 0m00s remaining] /
Done encoding file "UncompressedWave.ogg"
File length: 4m 13,0s
Elapsed time: 0m 2,750s
Rate: 92,184242
Average bitrate: 91,6 kb/s
F:\media\test>oggenc2 -q 4 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
"UncompressedWave.ogg"
at quality 4,00
[100,0%] [ 0m00s remaining] \
Done encoding file "UncompressedWave.ogg"
File length: 4m 13,0s
Elapsed time: 0m 2,485s
Rate: 102,014755
Average bitrate: 122,0 kb/s
F:\media\test>oggenc2 -q 6 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
"UncompressedWave.ogg"
at quality 6,00
[100,0%] [ 0m00s remaining] |
Done encoding file "UncompressedWave.ogg"
File length: 4m 13,0s
Elapsed time: 0m 2,485s
Rate: 102,014755
Average bitrate: 173,3 kb/s
F:\media\test>oggenc2 -q 8 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
"UncompressedWave.ogg"
at quality 8,00
[100,0%] [ 0m00s remaining] |
Done encoding file "UncompressedWave.ogg"
File length: 4m 13,0s
Elapsed time: 0m 2,672s
Rate: 94,875250
Average bitrate: 226,9 kb/s
F:\media\test>oggenc2 -q 10 UncompressedWave.wav
Opening with wav module: WAV file reader
Encoding "UncompressedWave.wav" to
"UncompressedWave.ogg"
at quality 10,00
[100,0%] [ 0m00s remaining] /
Done encoding file "UncompressedWave.ogg"
File length: 4m 13,0s
Elapsed time: 0m 3,172s
Rate: 79,920135
Average bitrate: 447,1 kb/s