Skip to main content

Topic: ABX on iPod (Read 5278 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • sehested
  • [*][*][*][*]
  • Members (Donating)
ABX on iPod
I plan to make a program to ABX music samples on an iPod.

My idea for this program is:

- Take a FLAC file with tags as input
- Convert this song using a number of predefined encodings
- Song tags should be "Title 1A", "Title 1B", "Title 2A", "Title 2B", "Title 3A", "Title 3B" in case three encoding options are selected.
- For every pair of AB one would be lossless.
- Encoder track number would be randomly selected.
- Create "mapping file with details on encoding and track mapping"
- Make play list in iTunes "Listening Test - Title" with all songs

Now this play list can be downloaded to an iPod and the user can do a ranking of the files.

After ranking the files he can view the mapping file and compare his results.

This would allow people to perform an unbiased AB test on their iPod.

I'm getting an iPod mini in two weeks and will use this program to do ABX testing to find the optimal encoding option for me.

I just hope that the iPod mini is not having the same problem as I discovered on its older sibling:

3G iPod Badly distorts high bit rate MP3

Now before I go ahead and do this program and maybe reinvent the wheel. Please let me know if such a program already exist out there.

  • KikeG
  • [*][*][*][*][*]
  • Developer
ABX on iPod
Reply #1
http://www.kikeg.arrakis.es/fileabx/

But it only accepts wav files as input, I'm afraid.

Edit: also, this program implements "pure" abx testing, whilst your proposal would be more like "abc/hr" testing.
  • Last Edit: 19 July, 2004, 03:38:30 AM by KikeG

  • sehested
  • [*][*][*][*]
  • Members (Donating)
ABX on iPod
Reply #2
Quote
I have now given the iPod ABX idea some more thought and realised that
the key feature of the program should be to mask the origin of the
samples and take care of the hassle of keeping sample files organised.

Encoding of files are better done completely outside this
program. The number of options for converted a file is simply more
than I would like to handle and besides the iTunes OLE interface only
allow you to select the encoder not to change its settings.


I anticipate the basic use of ipod-abx will be to compare a number of
encodings with a reference as follows, in this example three files:

1. Start out by preparing the files you want to compare.
2. Run ipod-abx <name> <reference> <file1> <file2> <file3>
  to prepare random playlist.
  The playlist would look like this:
     Artist     Song
     iPod-ABX  <name> 1 A
     iPod-ABX  <name> 1 B
     iPod-ABX  <name> 2 A
     iPod-ABX  <name> 2 B
     iPod-ABX  <name> 3 A
     iPod-ABX  <name> 3 B
3. Download the playlist "iPod-ABX <name>" to your iPod.
  A song will appear on the iPod play screen as:
     <name> 1 A
     iPod-ABX
4. Listen to the songs and rate them 1-5, but don't rate the
  references (use 0 stars). If you want decimals for your rating,
  e.g. 4.2, note it down and make other notes as you please during the
  tests.
5. Upload the ratings to iTunes.
6. Enter any notes you made as comments for the songs.
  If a song comment contains a decimal number, e.g. 4.2 this number
  will be used by ipod-abx as the rating, rather than the stars.
7. Run ipod-abx -v <name> to compile and display the results:
  sample  : <name> 1 A
  filename : <reference>
  encoder  : CD (EAC)
  rating  : 4.8

  sample  : <name> 1 B
  filename : <file2>
  encoder  : CD (EAC) lame aps

  sample  : <name> 2 A
  filename : <file3>
  encoder  : CD (EAC) lame api
  rating  : 1
  note     : Unbeleivably badly distorted piano

  sample  : <name> 2 B
  filename : <reference>
  encoder  : CD (EAC)

  sample  : <name> 3 A
  filename : <file1>
  encoder  : iTunes 4.2.0.72, QuickTIme 6.5
  comment  : CD (EAC) aac 128
  rating  : 4.2
  note     : Piano lacking definition

  sample  : <name> 3 B
  filename : <reference>
  encoder  : CD (EAC)

  Reference identified 2 out of 3 times.



ipod-abx can also be used to ABX two samples a number of times,
e.g. to compare LAME 3.90 with LAME 3.96 on iPod:

1. Start out by preparing the two files you want to compare.
2. Run ipod-abx -r <number> <name> <reference> <file1>
  to prepare random playlist with the two same files repeated <number> times.
  The playlist would look like this if <number> is 5:
     Artist     Song
     iPod-ABX  <name> 1 A
     iPod-ABX  <name> 1 B
     iPod-ABX  <name> 2 A
     iPod-ABX  <name> 2 B
     iPod-ABX  <name> 3 A
     iPod-ABX  <name> 3 B
     iPod-ABX  <name> 4 A
     iPod-ABX  <name> 4 B
     iPod-ABX  <name> 5 A
     iPod-ABX  <name> 5 B
3. Download the playlist "iPod-ABX <name>" to your iPod.
  The songs will appear on the iPod play screen as:
     <name> 1 A
     iPod-ABX
4. Listen to the songs and rate the non-reference.
5. Upload the ratings to iTunes.
6. Run ipod-abx <name> to compile and display brief results:
  Reference identified 5 out of 5 times.


<reference> or <file> can be any one of .flac, .wav, .mp3, .m4a, or .m4p.

.flac, and .wav are converted to Apple lossless.
.mp3, .m4a, and .m4p are added as they are.

The following tags are set:
Name: <name> <sample> <A/B>
Artist: iPod-ABX
Album: iPod-ABX <name>
Track Number: <tracknumber in playlist>
Track Count: <total tracks in playlist>
Discnumber: 1
Disccount: 1

The following tags are left as is:
Encoder

The following tags are cleared:
Year, Genre, Grouping, Compilation, Rating, Composer, BPM

Soundcheck must be enabled in iTunes and on the iPod to have the songs
play back at similar volume.


Usage: ipod-abx <name> <reference> <file> [file]...
       Prepares iTunes playlist for abx listening test.
       Randomly renames files, add files to iTunes, update their tags
       and create playlist.

       Use -r <number> to repeat test <number> times
       Use -s <mm:ss>  to set start time of .m4p file. Length of sample
                       will be limited to length of reference.

       ipod-abx <name>
       Compile result of listening test based on ratings and comments
       in iTunes of the tested songs.

       Use -v          for verbose output.


ipod-abx will complain about:
file type not supported
files not existing
iTunes not starting
album / playlist already in iTunes
length of songs differs by more than a few seconds
tracks not present in iTunes when compiling result
any error codes from iTunes

So that's the plan. I will probably start coding in about two weeks when
I'm back from my holidays.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
ABX on iPod
Reply #3
Quote
http://www.kikeg.arrakis.es/fileabx/

But it only accepts wav files as input, I'm afraid.

Edit: also, this program implements "pure" abx testing, whilst your proposal would be more like "abc/hr" testing.
[a href="index.php?act=findpost&pid=227128"][{POST_SNAPBACK}][/a]


Thanks, for the input.

My main concern is that I want to perform the ABX on the iPod itself.
First I considered to convert all the test files to Apple lossless to completely mask the files origin, but I have gone away from that idea.
I want to be able to reveal defects in the iPod decoding which would not be obvious if the iPod was only to playback lossless files. E.g. some high bitrate mp3 files are distorted on the iPod, but not if the file is converted to wave or lossless first.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
ABX on iPod
Reply #4
I have now completed iPod-ABX.

Please enjoy:

ipod-abx

Here's the usage notes:

Quote
Usage: ipod-abx <name> <reference> <sample> [sample]...

       Make iTunes iPod playlist for listening test.
       Randomly renames files, add files to iTunes, update their tags
       and create ABC/HR playlist with A as reference. Rate B and C. One being
       the reference and the other a sample.

       Use -x <number> to make ABX playlist with <number> sets with A as
                       sample, B as reference, and X as unknown. Rate X 5 if
                       reference or 1-4 if sample.
       Use -s <scale> set lowest rating of scale:
                          0 : 0-5 stars equals  - , 1.0, 2.0, 3.0, 4.0, 5.0
                          1 : 0-5 stars equals 1.0, 1.8, 2.6, 3.4, 4.2, 5.0
                          2 : 0-5 stars equals 2.0, 2.6, 3.2, 3.8, 4.4, 5.0
                          3 : 0-5 stars equals 3.0, 3.4, 3.8, 4.2, 4.6, 5.0
                          4 : 0-5 stars equals 4.0, 4.2, 4.4, 4.6, 4.8, 5.0
                          5 : 0-5 stars equals 4.5, 4.6, 4.7, 4.8, 4.9, 5.0
       The scale can be altered on the iPod by changing the rating of the
       reference.


       ipod-abx -u <name>

       Update song names in iPod with rating for making ranking between
       samples easier.


       ipod-abx <name>

       Prepare result file of listening test based on ratings and comments
       in iTunes of the tested songs.

       Use -i          to update playlist with rating and play info from iPod.
       Use -v          for verbose output.


       ipod-abx -c <heading>

       Compile result of all listening tests whose name start with heading.
       Songs not played will be excluded from the results.
       The folder name of each songs file name will be use to determine the
       encoder. See the example below for a suggested folder structure for
       test samples.


Hint:  Commmands in comment tag are treated as follows:
           finish( mm:ss) set finish time of song.
           start( mm:ss) to set start time of song.

Hint:  <reference>/<sample> can either be a file or a folder name. In case
       file do not exist it is assumed to be a folder name. ipod-abx will look
       for this folder in the current folder and in the previous folder used
       by ipod-abx. When specifying a folder name only part of the folder name
       needs to be given and "-" will match any part of the folder name. In
       case several folder names match the alphabetically latest will be
       choosen.

Examples:
       Assuming test samples are organised in this folder structure:

           Test Samples
             FLAC
               iTunes 4.2 aac 128
               iTunes 4.2 mp3 112 vbr
               iTunes 4.2 mp3 128 cbr
               iTunes 4.6 aac 128
               iTunes 4.6 mp3 112 vbr
               iTunes 4.6 mp3 128 cbr
               lame 3.96 mp3 128 v5

       To generate the playlist "128kbps - Waiting" with reference from
       "FLAC" and samples from "iTunes 4.6 mp3 112 vbr",
       "iTunes 4.6 mp3 128 cbr" and "lame 3.96 mp3 128 v5" folders:

  ipod-abx "128kbps - Waiting" flac itunes-mp3-112 itunes-mp3-128-cbr lame-v5


       To create result file after completing iPod listening test:

       ipod-abx -iv "128kbps - Waiting"


       To compile results of a number of test all named "128kbps - <song>":

       ipod-abx -c 128kbps


       To generate the playlist "Waiting" with eight ABX tests between samples
       from folders "iTunes 4.6 mp3 112 vbr" and "lame 3.96 mp3 128 v5":

       ipod-abx -x8 Waiting itunes-mp3-112 lame-v5

  • sehested
  • [*][*][*][*]
  • Members (Donating)
ABX on iPod
Reply #5
Some design notes:

Quote
Basic use of ipod-abx will be to compare a number of
encodings with a reference as follows, in this example three files:

1. Start out by preparing the files you want to compare.
2. Run ipod-abx <name> <reference> <sample1> <sample2> <sample3>
  to prepare random playlist.
  The playlist would look like this:
     Artist     Song
     <name>     A (reference)
     <name>     1 - B (?)
     <name>     1 - C (?)
     <name>     A (reference)
     <name>     2 - B (?)
     <name>     2 - C (?)
     <name>     A (reference)
     <name>     3 - B (?)
     <name>     3 - C (?)

3. Download the playlist "<name>" to your iPod.
  A song will appear on the iPod play screen as:
      2 - B (?)
       <name>
4. Listen to the songs and rate them 1-5, but don't rate the
  references (use 0 stars). If you want decimals for your rating,
  e.g. 4.2, note it down and make other notes as you please during the
  tests.
5. Upload the ratings to iTunes.
6. Enter any notes you made as comments for the songs.
  If a song comment contains a decimal number, e.g. 4.2 this number
  will be used by ipod-abx as the rating, rather than the stars.
7. Run ipod-abx -v <name> to compile and display the results:
  sample * : A (reference)
  filename : <reference>
  info     : Lossless, 919 kbps, 44.1 kHz
  encoder  : CD (EAC)

  sample * : 1 - B (?)
  filename : <reference>
  info     : Lossless, 919 kbps, 44.1 kHz
  encoder  : CD (EAC)
  rating  : 4.8

  sample  : 1 - C (?)
  filename : <sample2>
  info     : MP3, 192 kbps, 44.1 kHz
  encoder  : CD (EAC) lame aps

  sample  : 2 - B (?)
  filename : <sample3>
  info     : MP3, 320 kbps, 44.1 kHz
  encoder  : CD (EAC) lame api
  rating  : 1
  note     : Unbeleivably badly distorted piano

  sample * : 2 - C (?)
  filename : <reference>
  info     : Lossless, 919 kbps, 44.1 kHz
  encoder  : CD (EAC)

  sample  : 3 - B (?)
  filename : <sample1>
  info     : AAC, 128 kbps, 44.1 kHz
  encoder  : iTunes 4.2.0.72, QuickTime 6.5
  comment  : CD (EAC) aac 128
  rating  : 4.2
  note     : Piano lacking definition

  sample * : 3 - C (?)
  filename : <reference>
  info     : Lossless, 919 kbps, 44.1 kHz
  encoder  : CD (EAC)

  Missed 1 reference out of 3.



ipod-abx can also be used to ABX two samples a number of times,
e.g. to compare LAME 3.90 with LAME 3.96 on iPod:

1. Start out by preparing the two files you want to compare.
2. Run ipod-abx -x <number> <name> <reference> <sample>
  to prepare random playlist with a refence and a sample repeated <number> times.
  The playlist would look like this if <number> is 8:
     Artist     Song
     <name>     A (sample)
     <name>     B (reference)
     <name>     1 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     2 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     3 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     4 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     5 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     6 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     7 - X (?)
     <name>     A (sample)
     <name>     B (reference)
     <name>     8 - X (?)
3. Download the playlist "<name>" to your iPod.
  The songs will appear on the iPod play screen as:
      1 - X (?)
       <name>
4. Listen to the songs and rate the non-reference.
5. Upload the ratings to iTunes.
6. Run ipod-abx <name> to compile and display brief results:
  Sample/reference identified 8 out of 8 times.




The result of several individual listening tests can be combined to produce an overview of the results:

"128" listening tests compiled results:
                         A=iTunes 4.2 m4p 128
                                B=iTunes 4.6 aac 128
                                       C=iTunes 4.6 aac 160
                                              D=iTunes 4.6 mp3 112 vbr
                                                     E=iTunes 4.6 mp3 128 vbr
                                       lame 3.96 mp3 128 v5=F
                         A      B      C      D      E      F
-------------------------------------------------------------------------------
           BigYellow     -     4.9    5.0    5.0    5.0    5.0
                Echo    3.4    5.0    4.2    3.0    4.6    4.6
                Gone     -     4.8    5.0    5.0    4.9    5.0
           Hungarian     -     5.0    5.0    5.0    5.0    5.0
      ItCouldBeSweet     -     5.0    5.0    4.8    4.8    4.8
           Kraftwerk     -     4.9    5.0    4.9    5.0    5.0
       OrdinaryWorld     -     5.0    5.0    5.0    5.0    4.5
            RedHouse     -     5.0    5.0    4.5    4.8    4.9
            Rosemary     -     5.0    5.0    5.0    5.0    5.0
            Stairway     -     4.6    5.0    5.0    3.8    5.0
             Waiting     -     5.0    5.0    4.5    4.8    5.0
-------------------------------------------------------------------------------
             AVERAGE    3.4    4.9    4.9    4.7    4.8    4.9
===============================================================================
Prepared 2004-07-29 19:39 with iPod-ABX v0.4




<reference> or <file> can be any one of .flac, .wav, .mp3, .m4a, or .m4p.

.flac, and .wav are converted to Apple lossless.
.mp3, .m4a, and .m4p are added as they are.

The following tags are set:
Name: <sample>
Artist: <name>
Album: <name> [iPod-ABX]
Track Number: <tracknumber in playlist>
Track Count: <total tracks in playlist>
Discnumber: 1
Disccount: 1
Rating: 100 (five stars)
Grouping: iPod-ABX / iPod-ABC/hr

The following tags are left as is:
Encoder

The following tags are cleared:
Year, Genre, Compilation, Composer, BPM, Artwork

Soundcheck must be enabled in iTunes and on the iPod to have the songs
play back at similar volume.

ABX on iPod
Reply #6
Thanks a lot sehested, this is greatly appricated.  I've always wanted to be able to conduct accurate abx tests on my iPod, a lot of people did see the importance of doing abx on the iPod itself and told me to just test the formats on my computer.  But it is important that such tests be done on the iPod and what you use as its output.  My 3G iPod occassionaly does have distortion issues and formats/bitrates sound a lot different coming out of car audio than they do a pair of tiny computer speakers.  Thanks a lot!