Skip to main content

Topic: Public Listening Test [2010], Item Selection (Read 34667 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
This week I started going through the vast amount of listening test material available (or linked to) in this forum. I noticed that many of the critical items recently used in ABX tests can be categorized as "transient (non-stationary) noise". The items I found which can be assigend to this category are:

- Amnesia
- Autechre (Artefact+Context version in 01__DCT_Killer_Samples__Lossless_.zip), also ae_gg from the same artist
- Closer to God
- Convulsion
- Emese
- Harlem
- Human Disease
- NiN Quake OST
- 2nd_vent_clip.flac and drone_short from http://ff123.net/samples/.

Any important ones I missed?

Now, with the public test coming up, I propose we use only two of those items in the test. As the first item, I propose the Harlem sample, since that's a natural sound (applause recorded at a live performance of the Harlem Gospel Singers). All the other items are more ore less artificial sounds and as such, do not represent typical listening material. Now, which one should we take as the second item? Should we make a poll?

Chris
  • Last Edit: 06 January, 2010, 05:51:20 PM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #1
I propose to use online randomizer to select samples. Than the list of  numbers of random  samples will be sent automatically to all interested people who want to be sure that's selection was totally random.

Let's collect all samples from HA members and upload section and then public it. Samples from different styles and/or with different issues (not specifically for one competitor in particular).

I would avoid some of the samples here like "Closer to God" because they were submitted as Nero's issues only.
  • Last Edit: 06 January, 2010, 07:40:30 PM by IgorC

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #2
I can't make a suggestion for transient noise samples but I agree very much to do include non-artificial samples.

I'm more into non-transient noise.
I suggest to use harp40_1 as a serious harpsichord problem (maybe it falls into the transient noise problem class partially - in this case we have another non-artificial sample for it).
From mp3 experience problems that strike me most are samples which are hard to beleive that they bring problems to encoders.
trumpet_MyPrince is such a sample which tends to get encoded with kind of a tremolo effect. An even worse sample for this encoding problem - of artificial nature - is lead-voice.
The very first problem sample I ran upon trumpet is prone to distortion at rather low bitrate.
herding_calls is also a tonal problem sample at least to low/moderate bitrate mp3.
I once ran upon subtle issues on a very simple song Là Ou Je Suis Née when using mp3.

The samples are here:
harp40_1
trumpet_My Prince
lead-voice
trumpet
Là Ou Je Suis Née
  • Last Edit: 18 January, 2010, 05:03:04 PM by halb27
lame3995o -Q1

  • halb27
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #3
I just looked up my lossyWAV problem sample collection and found a transient noise prone sample keys_1644ds.
Not just music, but non-artificial.

Oh, I forgot the link for herding_calls in my last post.
  • Last Edit: 18 January, 2010, 05:38:54 PM by halb27
lame3995o -Q1

  • /mnt
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #4
I find the ptp_robocop, musique_non_stop and the_robots samples, to be suitable for the test. I have tried them with Nero at q 0.40 and QuickTime TVBR at 60, and they can be ABXed effortlessly.

Suggested samples:
- Linchpin
- The Robots
- Musique Non Stop
- Show Me Your Spine (3:00 - 3:30 Edit)
- Human Disease
- Hexonxonx
  • Last Edit: 18 January, 2010, 06:01:06 PM by /mnt
"I never thought I'd see this much candy in one mission!"

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #5
Yep, linchpin has presented issues for all AAC encoders in my personal test.

Samples from previous public test and from well know http://ff123.net/samples.html to avoid biased decisions.
  • Last Edit: 18 January, 2010, 06:12:15 PM by IgorC

  • /mnt
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #6
Yep, linchpin has presented issues for all AAC encoders in my personal test.


It sure is noticable issue with Nero AAC and LAME V5 - V0 and a big one for Helix Mp3. I haven't tried it on iTunes or QuickTime in a while though.
  • Last Edit: 18 January, 2010, 06:22:27 PM by /mnt
"I never thought I'd see this much candy in one mission!"

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #7
Vorbis and Apple LC-AAC performed very bad on linchpin at 80 kbps http://www.hydrogenaudio.org/forums/index....c=74781&hl=
  • Last Edit: 18 January, 2010, 06:27:04 PM by IgorC

  • /mnt
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #8
Vorbis and Apple LC-AAC performed very bad on linchpin at 80 kbps http://www.hydrogenaudio.org/forums/index....c=74781&hl=

I see, am suprised that Vorbis struggled with it aswell. For some reason this sample does sound better with LAME at 130kbps, even if i can ABX it at V2.

The Spill The Blood sample sounds bad on my cheap Logitech PC speakers.

QuickTime True VBR at 60 is just as bad:

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v1.0
2010/01/18 23:35:06

File A: C:\Temp\128 Test\Linchpin.m4a
File B: E:\Music\Lossless\Fear Factory - Digimortal\05. Linchpin.flac

23:35:06 : Test started.
23:35:22 : 01/01  50.0%
23:35:30 : 02/02  25.0%
23:35:40 : 03/03  12.5%
23:35:59 : 04/04  6.3%
23:36:20 : 05/05  3.1%
23:36:29 : 06/06  1.6%
23:36:40 : 07/07  0.8%
23:36:47 : 08/08  0.4%
23:36:56 : 09/09  0.2%
23:37:08 : 10/10  0.1%
23:37:20 : 11/11  0.0%
23:37:29 : 12/12  0.0%
23:37:30 : Test finished.

 ----------
Total: 12/12 (0.0%)
  • Last Edit: 18 January, 2010, 06:49:51 PM by /mnt
"I never thought I'd see this much candy in one mission!"

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #9
Nice work, guys! Yes, the Linchpin sample today was promoted to test sample for tuning Fraunhofer's encoder

An addition to the "tonal items" halb27 uploaded: About half a year ago, I downloaded a sample called 06___Mandylion_shrt.flac from HA. It was supposedly uploaded by shadowking, as a quick search indicates. Can anyone post the link to it here? I don't want to upload it again because it's probably still available.

Chris

P.S.: It seems the item used to be in this thread? Where did it go?
  • Last Edit: 18 January, 2010, 08:01:55 PM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #10
Here are more killer samples:
Creuza (tonality issue)
Spill the blood. I submitted this quiet sample as issue for LAME -V5... V0 sometimes ago. One of my most tested samples
Waiting was tested in every public test.
Fatboy. MP3 encoders perform very bad on it (see last public MP3 test at 128). Would be interesting to see what happens with AAC encoders.
Aquatisme from 48 kbps AAC test
Herbie Hancock (old record,jazz sample+ stereo separation test) was submitted sometimes ago as Apple issue but it's actually very hard sample for many codecs.
Descending Darkness. Very loud and reach drums content

These samples are the most interesting for me.
  • Last Edit: 18 January, 2010, 09:32:27 PM by IgorC

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #11

  • muaddib
  • [*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #12
Why not taking in consideration organizing a database of samples as I've proposed here:

http://www.hydrogenaudio.org/forums/index....showtopic=64991

and then use public lottery for random choice of samples (also described in the thread)?

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #13
It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.

  • hellokeith
  • [*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #14
Have all the item selections been already decided? I have a 10-20 second section of a track that I use which features some hi-hat hits quickly moving through the stereo field, very easy to identify if something is wrong.

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #15
Not at all! If you think it's a critical sample, please upload it in this thread.

Thanks!

Chris
If I don't reply to your reply, it means I agree with you.

  • muaddib
  • [*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #16
It's good idea.
My plan is a little bit limited to this particular test. The same methodology but only for 128 kbps. The organization of test is already time intensive.
The collaboration is welcome if someone else will take care of it.

We could start creating the database that I propose from the samples that are prepared for this aac test. 3a, 3b, 4 and 5 subsets are adequate for this.
Maybe in the upload thread for the current test, a sticky could be created where each proposed sample is given number and where it is decided in which of subsets 3a, 3b, 4 or 5 it should go to.
Creating database during the preparation of the test is contradictory to what I wrote in the thread for database in post #8, but I guess that there is no other way to start organizing an "objective" database for listening tests.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #17
[a href='index.php?showtopic=73970']Bibilolo[/a] hasn't been mentioned, yet, but should be in the selection.
  • Last Edit: 15 February, 2010, 04:47:53 PM by rpp3po

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #18
I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #19
I think we should also include a high quality applause sample with a nice stereo image. Any recommendations?

See the first post.

From the link in Igor's post (#6) I suggest "ATrain" and "SinceAlways". In that selection, there is also an "applaud00" (from an Eagles live recording), but that is so loud that it clips at several passages.

Chris
If I don't reply to your reply, it means I agree with you.

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #20
Took a look at http://www.listening-tests.info/mp3-128-1/results.htm before it closes down. Final Fantasy, Chariots Of Fire, Tom's Diner, and Velvet are a problem for at least one coder in the test. Final Fantasy we don't need in this test because we already have the SQAM harpsichord item (halb27, post 3 above). Chariots Of Fire I can't find on HA, but I got a CD which contains it, so I can upload a critical passage if necessary. Tom's Diner is available in sample16.zip on http://www.mp3-tech.org/tests/aac_48/samples/. By the way, quite a few of the other samples suggested are available in ZIP files there. Velvet and some other potential candidates are at http://lame.sourceforge.net/download/samples/.

If time permits, I'll go through the last two linked collections to see if there are other samples worth including.

Chris
  • Last Edit: 20 February, 2010, 02:30:44 AM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #21
Quote
Chariots Of Fire I can't find on HA

http://www.hydrogenaudio.org/forums/index....st&p=601657

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010], Item Selection
Reply #22
Wonderful, thanks a lot! Somehow missed that thread 

Chris
If I don't reply to your reply, it means I agree with you.

  • hellokeith
  • [*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #23
Sample to test encoder stereo imaging transparency:

http://www.hydrogenaudio.org/forums/index....showtopic=78889

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010], Item Selection
Reply #24
Due to absence of random selection mechanism and time limits it will be more realizable if Chris will be the person who will choose the samples with public agreement. Some most discussed samples are already included like Linchpin.
Personally I don't have doubts about Chris.
  • Last Edit: 09 March, 2010, 04:48:06 PM by IgorC