Skip to main content

Topic: Interesting Histograms. (Read 25746 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
I've been doing histograms of audio stuff, as well as calculating spectral flatness measure, so I can reproduce some of the information I had back when I was starting on the PAC/NBC/AAC work.

I've found an album that a fair number of missong codes.

I've also found an album that has a max under +-16384 and has over 75% missing codes inside of that.

I didn't originally set up to calculate ratio of missing codes to all codes. But I think I shall have to.

I'll put up some photos of the various histograms when I get a bit more done.

Wow, batman, doesn't anyone look at this stuff before pressing!?
  • Last Edit: 28 May, 2011, 07:43:05 PM by Woodinville
-----
J. D. (jj) Johnston

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #1
http://s238.photobucket.com/albums/ff228/j...l%20Histograms/

Some histograms.  My ghu is octave SLOW.
-----
J. D. (jj) Johnston

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #2
ETA:

octave wavread returns numbers 1 level out of 32768 below -1 and and exactly +1 on the positive side.

Does anybody have a WORKING wavread fer bleep's sake?  Anyone? Help?  wavread should never go below -1, or above 1 - 2^-(nbits-1)

Also, try this in the latest octave 

round(-.0001)
ans=-0

Yep, minus zero.

I guess I'll spring for my own personal copy of ratlab and signal processing package, that's intolerable.

So, all of these histograms are "almost right", it's pretty clear that they are right except that they might be missing more missing codes, since one can't count on the actual wavread scaling.

Bleepity blip bleep blip.

Your in annoyance

jj
  • Last Edit: 28 May, 2011, 11:54:56 PM by Woodinville
-----
J. D. (jj) Johnston

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #3
Ok, if you're using wavread from octave, be sure to change the constants for 16 and 24 bit files from 2^(nbits-1)-1 to 2^(nbits-1)

Otherwise you waste a lot of time getting incorrect results.

Plots above will be changing as they get done. Expletive.
-----
J. D. (jj) Johnston

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Interesting Histograms.
Reply #4
Thanks for debugging Octave's wavread, JJ!

Hmm, looking at the Robert Lee (piano) histogram, it looks like someone, as the last step before pressing, applied a positive-dB gain on the 16-bit data? 

Chris
If I don't reply to your reply, it means I agree with you.

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #5
Thanks for debugging Octave's wavread, JJ!

Hmm, looking at the Robert Lee (piano) histogram, it looks like someone, as the last step before pressing, applied a positive-dB gain on the 16-bit data? 

Chris


It's a nearly-homemade recording of a really good stride pianist from the middle of Indiana.

If you want to hear youtube-quality, listen to "Allen Dale" on Youtube. But try to ignore it when he talks politics. Listen to 12th St Rag, and just watch the hands fly.

Oh, and you know, I'm not under any IP restrictions right now, I could post the histogram script for Octave if anyone gave two whits. It's not like it's even remotely close to rocket science, I think anyone here could write it in a half-hour as long as you also fix the wavread constant. &*(&( that was irritating. Oh, and the Spectral Flatness Measure is for 2048 Hann Window.  Green Day and Avril Lavigne now up, Chumbawama coming up soon.
  • Last Edit: 29 May, 2011, 04:49:12 AM by Woodinville
-----
J. D. (jj) Johnston

  • xnor
  • [*][*][*][*][*]
  • Developer
Interesting Histograms.
Reply #6
Interesting plots, thanks.

I've also found an album that has a max under +-16384 and has over 75% missing codes inside of that.


So it's basically a 15-bit album? 

I'm interested in the script.
  • Last Edit: 29 May, 2011, 06:49:50 AM by xnor
"I hear it when I see it."

  • romor
  • [*][*][*][*][*]
Interesting Histograms.
Reply #7
Ok, if you're using wavread from octave, be sure to change the constants for 16 and 24 bit files from 2^(nbits-1)-1 to 2^(nbits-1)

Otherwise you waste a lot of time getting incorrect results.

Plots above will be changing as they get done. Expletive.

Excuse my ignorance and if I understand you right, but why should Octave (or anything else) scale normalized data on 2^(n-1)-1 instead 2^(n-1)?

  • Paulhoff
  • [*][*][*]
Interesting Histograms.
Reply #8
I've been doing histograms of audio stuff, as well as calculating spectral flatness measure, so I can reproduce some of the information I had back when I was starting on the PAC/NBC/AAC work.

I've found an album that a fair number of missong codes.

I've also found an album that has a max under +-16384 and has over 75% missing codes inside of that.

I didn't originally set up to calculate ratio of missing codes to all codes. But I think I shall have to.

I'll put up some photos of the various histograms when I get a bit more done.

Wow, batman, doesn't anyone look at this stuff before pressing!?


I don't have anything, and I'm not at all sure what you want to begin with.


My Bad???

Paul

   
"Reality is merely an illusion, albeit a very persistent one." Albert Einstein

  • xnor
  • [*][*][*][*][*]
  • Developer
Interesting Histograms.
Reply #9
Excuse my ignorance and if I understand you right, but why should Octave (or anything else) scale normalized data on 2^(n-1)-1 instead 2^(n-1)?


because 2^15-1 = 32767 decimal = 7FFF hex = highest possible value for a signed, 16-bit integer

and -32768 decimal = 8000 hex

see two's complement
  • Last Edit: 29 May, 2011, 01:22:38 PM by xnor
"I hear it when I see it."

  • romor
  • [*][*][*][*][*]
Interesting Histograms.
Reply #10
If the problem is about histogram function, and that's why the need for +1 bin (number of samples + 1), maybe that can make some sense to me

Data presented by wavread from Octave is same as text output by Cool Edit Pro, or output from libsndfile library on my PC
  • Last Edit: 29 May, 2011, 01:39:56 PM by romor

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Interesting Histograms.
Reply #11
OT:

OH MY FUCKING GOD JJ you read my mind about octave.

I tried my *best* to like it a month ago, when I attempted to use it for "real" audio work. I really did. Right before I discovered
  • wavread does not do blocked WAV input, only file-at-a-time. I actually got most of the way through rewriting this, but
  • Whoever wrote wavread does not comprehend two's complement arithmetic, and maps [-32768, 32767] to [-1,1]. That's right. Try reading a 16-bit WAV of all zeroes in. Can't do it! The samples will read as -2^-17. (EDIT: Ah yes you saw this too)
  • wavread has not been materially updated since 2001. Or maybe 1996. Or possibly 1993. Needless to say it has not gone through a process of continuous improvement
  • Octave itself requires upwards of 4 seconds on my P4 to populate a 128k sample array with the evaluation of a window function. Which is approximately 3 orders of magnitude slower than it ought to be.


At which point I concluded that: it is plainly obvious that nobody actually does useful audio processing work in Octave, and it would likely be a waste of my time to attempt to do so, rather than look at other free numerical processing solutions. (I'm thinking mostly about F# nowadays.)

Then I posted "GNU Octave fucking blows" on Facebook. Now see jj, if I friended you, you would have seen that comment, and you wouldn't have had to go through all that trouble, eh
  • Last Edit: 29 May, 2011, 03:38:46 PM by Axon

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #12
Ok, if you're using wavread from octave, be sure to change the constants for 16 and 24 bit files from 2^(nbits-1)-1 to 2^(nbits-1)

Otherwise you waste a lot of time getting incorrect results.

Plots above will be changing as they get done. Expletive.

Excuse my ignorance and if I understand you right, but why should Octave (or anything else) scale normalized data on 2^(n-1)-1 instead 2^(n-1)?


I don' t know, but that's what it did. !?

You can go into the library and change it to the proper 2^(n-1) without much trouble.
-----
J. D. (jj) Johnston

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Interesting Histograms.
Reply #13
I don' t know, but that's what it did. !?

You can go into the library and change it to the proper 2^(n-1) without much trouble.


I vaguely remember doing this too.

Yet another reason to give Octave the Viking funeral it richly deserves, by dousing it with lighter fluid, and setting it on fire.

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Interesting Histograms.
Reply #14
I've also found an album that has a max under +-16384 and has over 75% missing codes inside of that.

I didn't originally set up to calculate ratio of missing codes to all codes. But I think I shall have to.

In some meaningful sense, couldn't this be considered a "13-bit" recording, rather than a "15-bit" recording?

And I suppose, more generally: when we are evaluating coding efficiency, are we "literally" evaluating Komolgorov complexity, and using that as an estimator for sound quality?

  • xnor
  • [*][*][*][*][*]
  • Developer
Interesting Histograms.
Reply #15
Forgive my previous reply, I didn't read properly.

in the Octave 3.4.0 sources, wavread.m, line 201 onward:

Code: [Select]
    ## Normalize samples.
    switch (bits_per_sample)
      case 8
        yi = (yi - 128)/127;
      case 16
        yi /= 32767;
      case 24
        yi /= 8388607;
      case 32
        yi /= 2147483647;


This seems to be the problem you're talking of and should be corrected to 32768 for 16-bit for example.

Or patch it to don't normalize at all.
  • Last Edit: 29 May, 2011, 03:58:55 PM by xnor
"I hear it when I see it."

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #16
Forgive my previous reply, I didn't read properly.

in the Octave 3.4.0 sources, wavread.m, line 201 onward:

Code: [Select]
    ## Normalize samples.
    switch (bits_per_sample)
      case 8
        yi = (yi - 128)/127;
      case 16
        yi /= 32767;
      case 24
        yi /= 8388607;
      case 32
        yi /= 2147483647;


This seems to be the problem you're talking of and should be corrected to 32768 for 16-bit for example.

Or patch it to don't normalize at all.


I patched it to 32768 and 2147483648

Then I could get the results I expected on a file that was half -max and half +max without using wrong multipliers

I think 8-bit is mulaw, and if I'm right, that's just totally wrong, but I'm not entirely sure. Been a while since I used 8-bit.
-----
J. D. (jj) Johnston

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #17
Interesting plots, thanks.

I've also found an album that has a max under +-16384 and has over 75% missing codes inside of that.


So it's basically a 15-bit album? 

I'm interested in the script.


Actually, as Axon points out, it's 13 bits, because there are so many missing levels, as well as it running under half.
-----
J. D. (jj) Johnston

  • xnor
  • [*][*][*][*][*]
  • Developer
Interesting Histograms.
Reply #18
I think 8-bit is mulaw, and if I'm right, that's just totally wrong, but I'm not entirely sure. Been a while since I used 8-bit.


Nah it's just that 8-bit PCM is unsigned so the range of values is 0 to 255 with a centerpoint of 128.
So if you substract 128 you get the same result you'd get with signed 8-bit integers (-128 to 127). Of course the division by 127 should be corrected to 128 like in the other cases.

edit: oh you already filed a bug, great!
  • Last Edit: 29 May, 2011, 04:20:14 PM by xnor
"I hear it when I see it."

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Interesting Histograms.
Reply #19
If anybody's interested, I still have my mostly-complete-and-working rewrite of wavread to implement buffered WAV file reading (along with other mucho bug fixes of course).

EDIT: Ah, I saw jj's bug too. I guess I should stop being so antisocial and submit a few of my own too.
  • Last Edit: 29 May, 2011, 04:32:11 PM by Axon

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #20
If anybody's interested, I still have my mostly-complete-and-working rewrite of wavread to implement buffered WAV file reading (along with other mucho bug fixes of course).

Well, I'd be interested if it was working.



I suppose I should just attach the histogram/sfm calculation here.

Nothing fancy, I assure you.

--- this is freeware, nothing is guaranteed at all---
Code: [Select]
clear all
close all
clc

fname='02.wav';
x=wavread(fname);
x=x';

len=length(x)

his(1:65536)=0;

low=32768;
high=32768;
windd=hann(2048)';

sfmmean=0;
nmeas=0;

for ii=1:len
for jj=1:2

t=x(jj,ii);
if ( t < -1)
t=-1
fflush(stdout);
end
if (t >65535/65536)
t=65535/65536
fflush(stdout);
end

t=round(t*32768+32769);


his(t)=his(t)+1;
if (t < low)
low=t;
end
if (t > high)
high=t;
end
end
if (mod(ii,1024)==0)
if (ii>1024)
for kk=1:2
w=x(1,ii-2047:ii);
w=w .* windd;
wt=fft(w);
wt= wt .* conj(wt);
wt=max(wt,.0000000001);
am=sum(wt)/2048;
gm= exp (sum(log(wt))/2048);
sfm=gm/am;
if ( am > .5)
nmeas=nmeas+1;
sfm=10*log10(sfm);
sfmmean=sfmmean+sfm;
end


end
end
if (mod(ii,1024*16)==0 )
ii/len
fflush(stdout);
end
end

end

tot=sum(his);
his=his/tot;
his=max(his, .000000000001);
xax=-32768:32767;

semilogy(xax,his);
axis([-40000 40000 1e-10 1e-1]);
nmeas=nmeas
if (nmeas >0)
sfmmean=sfmmean/nmeas
end
low=low-32769
high=high-32769
fname
---end of freeware---

Change fname as you will

Its slow. The time is spent not in the fft part, but in the very simple histogram part, too. No idea why. It's slower than a snail glued to a rock.
  • Last Edit: 30 May, 2011, 02:18:46 AM by Canar
-----
J. D. (jj) Johnston

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Interesting Histograms.
Reply #21
Mods, could one of you please add *.m as an allowed upload extension?

  • ExUser
  • [*][*][*][*][*]
  • Read-only
Interesting Histograms.
Reply #22
I'll bug the admins. I can't do this. In the interim, zip the .m and post that?

  • googlebot
  • [*][*][*][*][*]
Interesting Histograms.
Reply #23
Then I posted "GNU Octave fucking blows" on Facebook. Now see jj, if I friended you, you would have seen that comment, and you wouldn't have had to go through all that trouble, eh


I somewhat associate 'Axon' with wooing Woodinville for quite some time now, but I think that's a new level.

  • Woodinville
  • [*][*][*][*][*]
Interesting Histograms.
Reply #24
I'll bug the admins. I can't do this. In the interim, zip the .m and post that?


Well, screen-copying the text above into octave works like a champ

It's not like there's any special character stuff in it.
-----
J. D. (jj) Johnston