Skip to main content

Topic: Public, Multiformat Listening Test @ 64 kbps - OPEN (Read 87727 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #100
To illustrate the debate about noise and clicks I uploaded a new harpsichord sample (recorded in the 60') with much more mechanical clickety-clack :
http://www.megaupload.com/?d=WEJXUM18

I also have a fresh and expensive opera live recording with several stage noise (feet or object on the parquet floor) which were so close to ripping artefacts that I assimilated them first to mastering issues. I also have an excellent opera in Hybrid-SACD with weird pops on one track I can't explain. And to finish, a new CD re-issue (licensed by a label to a second one) with a ripping click noise: the CD is new, pressed but with a flaw typical of amateur ripping...


I don't think we should include samples that weren't correctly ripped unless with explicitely want to test how will react different encoders to a common issue. Better use a good rip of interesting samples rather than poor ones. But those clicks are far to be unsignificant: millions peoples have the same ripping artifacts on their library and the exemples I listed above are showing that even EAC-maniac can have the same or similar ones.

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #101
12.8. is near... How is the test going? How many results are expected?

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #102
Will it be possible to get the key file for the result files when the test has finished? It would be interesting to compare your own results with the averages.
davidnaylor.org

  • Alex B
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #103
Will it be possible to get the key file for the result files when the test has finished? It would be interesting to compare your own results with the averages.

Normally it has been published after the test has finished. Also, the complete test data must be publicly available for anyone who wants to verify the results or further analyze the data.

BTW, I mailed my result files today.

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #104
Will it be possible to get the key file for the result files when the test has finished?


Absolutely, yes!

I received both results, BTW. So far, I haven't been able to sort the results, but I hope to do so tomorrow. After that, I can also tell you which samples have a higher priority. What I can tell is that I have 22 mails in my inbox.

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #105
It will be very interesting to see the results. I felt I was finding more serious artifacts in the lower-numbered samples, ~1-5. Will be interesting to see if other people were doing the same.
  • Last Edit: 08 August, 2007, 02:21:01 PM by naylor83
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #106
Sample 8 is the only one with less than 10 results. For the other samples, here is the number of (valid) results so far:

Sample 01: 18
Sample 02: 18
Sample 03: 16
Sample 04: 17
Sample 05: 15
Sample 06: 14
Sample 07: 11
Sample 08: 9
Sample 09: 12
Sample 10: 10
Sample 11: 11
Sample 12: 11
Sample 13: 13
Sample 14: 11
Sample 15: 10
Sample 16: 10
Sample 17: 12
Sample 18: 11

Unfortunately, there were several invalid results with ranked references which I am going to ignore.

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #107
Unfortunately, there were several invalid results with ranked references which I am going to ignore.


Oops. Hope none where mine
davidnaylor.org

  • Whelkman
  • [*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #108
Unfortunately, there were several invalid results with ranked references which I am going to ignore.

Can you describe invalidities? I read ABC/HR's documentation, but it is out of date. I got through five rounds of samples or so before I understood the behavior of the ABX "Testing Mode" but went back and corrected or reran previous trials (good thing I wrote notes for each sample). I should be okay now, but one never knows.

EDIT: I see that I've downloaded the Java version yet documentation points to the Win32 version, so it's not out of date as much as mildly incompatible.
  • Last Edit: 09 August, 2007, 08:51:11 PM by Whelkman

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #109
Can you describe invalidities?


As I said, results with ranked references.

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #110
Unfortunately, there were several invalid results with ranked references which I am going to ignore.
Can you describe invalidities? I read ABC/HR's documentation, but it is out of date. I got through five rounds of samples or so before I understood the behavior of the ABX "Testing Mode" but went back and corrected or reran previous trials (good thing I wrote notes for each sample). I should be okay now, but one never knows.EDIT: I see that I've downloaded the Java version yet documentation points to the Win32 version, so it's not out of date as much as mildly incompatible.

Whenever you are not sure whether you can hear differences or not, you should do ABX in testing mode. When you do enough correct trials (5 out of 5, or 7 out of 8) then reference is grayed out and you can only move slider for coded sample (you can only rank coded sample). Before doing ABX in testing mode it is advisable first to do it in training mode until you are sure that you can hear differences.

IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.

And one more thing. Come on people please do a bit more on samples 7-18!  At least sample 7 is very easy even for high anchor.

  • Whelkman
  • [*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #111
Whenever you are not sure whether you can hear differences or not, you should do ABX in testing mode. When you do enough correct trials (5 out of 5, or 7 out of 8) then reference is grayed out and you can only move slider for coded sample (you can only rank coded sample). Before doing ABX in testing mode it is advisable first to do it in training mode until you are sure that you can hear differences.

IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.

Great summary. This information would be helpful to newcomers were it included somewhere in the test summary or linked elsewhere.

  • beto
  • [*][*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #112
IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.


That is precisely what I do. I have no problem in giving 5 to an encode. If I cannot spot any obvious difference after listening to the sample two or three times it gets a 5 rating.

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #113
That is precisely what I do. I have no problem in giving 5 to an encode. If I cannot spot any obvious difference after listening to the sample two or three times it gets a 5 rating.


I try a bit harder than that, because I like to know that I have all the detail in the music that I can possibly hear when I dish out a big 5.0.
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #114
Editorial comment:
Your testing procedure is just too complicated for a senior citizen old codger like me.  Too many installs, too many directories, to many apps...  Too much....  If anyone comes up with a simple web based user interface count me in.... 


Note to moderators:  If my comment seems overly harsh just delete it.

Woody

  • Martin F.
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #115
I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests
FLAC.

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #116
Well, all you need to do is simply download the ABC/HR ZIP file provided on the test page, unpack it to a folder of choice, then download the samples you want to test, unpack them in the ABC/HR folder, run the corresponding DecodeXX.bat files from the "bin" directory and that's it. All you need to do then is to start ABC/HR, load the corresponding "*.ecf" file, test, save as "*.erf" and send me the file. That's not too complicated.

I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests


Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.

  • Martin F.
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #117
Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.


I was a bit confused … the readme says "Place the sample package ("SampleXX.zip") in the same folder as ABC/HR and uncompress it." I thought #18 was the "latest version"
  • Last Edit: 10 August, 2007, 07:47:51 PM by Martin F.
FLAC.

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #118
Well, all you need to do is simply download the ABC/HR ZIP file provided on the test page, unpack it to a folder of choice, then download the samples you want to test, unpack them in the ABC/HR folder, run the corresponding DecodeXX.bat files from the "bin" directory and that's it. All you need to do then is to start ABC/HR, load the corresponding "*.ecf" file, test, save as "*.erf" and send me the file. That's not too complicated.


I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests


Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.


It still takes a bit of work to get started on these tests, although it's not difficult.  I can think of other ways to do this.  For example:  have the java app start from a click on a webpage.  The application could have another dialog box specifically made for such tests which lists the samples the test administrator has selected, has buttons for downloading the compressed files for each sample (either by torrent or http), and has another button for starting the test once the download has completed (the app would automatically take care of decompressing/decoding everything).  The only thing you would need to specify is where to download the compressed file to, and where to load it from when you want to start/continue the test.  Instead of emailing the .erf file, you could choose to submit it directly by pressing a button (perhaps with IP logging to identify possible multiple submissions from the same person).

The only downside to this is people may not like the abchr application to connect to the internet, because then there are always questions about what it does when it connects.

  • Whelkman
  • [*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #119
The only downside to this is people may not like the abchr application to connect to the internet, because then there are always questions about what it does when it connects.

The people who care about their the nature of their Internet traffic probably aren't the same who want the most convenient experience possible. I assume the application would still work manually for those inclined to run it thusly.

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #120
why can't an installer with ALL ready be made? (not that i care, but for others like the prev poster)

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #121
Someone would have to write an installer in Java or something that runs on Windows, Linux and Mac.

  • kdo
  • [*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #122
I've got two comments about the current abc/hr java version:

1) in ABX mode there are keyboard shortcuts Q,W,E,D to play/stop samples -- this is VERY helpful.
However, in the ABC/HR mode, the same shortcuts are available but which sample do they refer to? That is, when I press W or E - which sample of the 5 it is going to play? Had to click the buttons with the mouse to make sure it's playing the ones I wanted.

2) on my laptop the playback via the java sound engine skips every so often. Setting high priority in the task manager (windows2000) doesn't help.

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #123
Well, I cannot help in those cases since I didn't write the program, sorry. Also, there is no better thing we can use that is also available for multiple platforms.

Anyways, today is the last day you can test and submit results. I will close the test tomorrow at around 12 PM German / Central European Time (GMT +1 with daylight saving time). Once the test is closed, no results will be accepted, so please don't wait until the very last minute.

Important! If you decide to start testing now, please focus on samples 7+ since they are the ones with the least results. This only applies to people who start testing now - if you already tested samples 1 to 6 and your results are pending, submit them, of course. Again, if you start testing now, please do so for samples 7 and higher - do not start testing now while choosing samples 1 to 6!
  • Last Edit: 11 August, 2007, 04:50:48 AM by Sebastian Mares

  • beto
  • [*][*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #124
I try a bit harder than that, because I like to know that I have all the detail in the music that I can possibly hear when I dish out a big 5.0.


That of course makes sense. Maybe I am just not patient enough.