Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Proposal Of Testing Framework (Read 13208 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Proposal Of Testing Framework

Hi,

I'm considering creating a unified testing framework for LAME and perhaps other encoders in future. 

Dibrom, I'd like to hear your personal opinion about the idea. I'm making this post public, because I'd like to know what the other members of this forum think about it, too.

Here is an excerpt from the letter I sent to the lame-dev list (the relevant part):
Quote
The developers and everyone having the will to test the encoder will have the opportunity to share their results using standard web forms. The features will include:
1. availability of standard samples which are known to cause problems as losslessly compressed files with a brief description of the problem
2. for each sample, there will be seperate test result sets for each version of the encoder
3. under each sample and encoder version there will be a forum-like "thread" where anyone could post their test results. Discussions will not be allowed and the users will be asked to discuss the results on dedicated forums and mailing lists.
(NOTE: I could add the possibility to rate the sample in 0-5 scale and/or a poll with fixed answers instead of writing a comment in the thread)
4. for anyone who tested the encoder on a custom sample, a seperate page for upload and results.
(NOTE: or only a possibility to send a link to the custom sample, instead of uploading it, in order to avoid getting copyrighted files)
5. the uploaded samples which turn out to address a yet unaddressed issue of the encoder will eventually be moved to the standard-samples area by the admins.

(thanks to Alexander for the suggestions)

It's a very simple scheme. Any suggestions are welcome. Once it's complete, the main advantage of it will be that empty sample/encoder-version fields will indicate that they haven't been tested yet. It will be clear what is yet to be done. It will also be clear in which version a certain increase or decrease in quality occured. Results of the tests which have already been done may be easily moved to the database by myself with some help from the testers.

Please let me know what do you think of the idea. Do you find the framework useful?


I would like to combine this tool with your work at the Hydrogen Audio forum. In my opinion, users and testers should submit their results to this unified database. On the front page of the database's interface, you and other developers will be able to announce which samples are currently needed to be tested. You could also link to them from HA threads. I don't really want to start creating this tool without your will to adapt it. I think that another advantage over the current system (posting results to the forum) will be the ease of searching within already done listening tests.

I want to make this tool as flexible as possible in order to make it useful for testing unstable releases, releases available only as binary and so forth.

In the beginning, the samples I'll use can be the ones you've published here on the forum.

Let me know what do you think.

Bye and keep up the great work, everyone  ,
Olcios

Proposal Of Testing Framework

Reply #1
The idea is nice, although iirc it's already part of the AudioIllumination concept.

I'm not Dibrom, but I'd guess that Dibrom is not very eager to encourage the test result submission of for example alt-preset development outside HA.
The simple fact is that most Lame devs (except Takehiro) haven't taken the testing or tweaking very seriously at all (at least for over a year or so). Thus the alt-presets were born and developed here.

Then again, Vorbis already has its bugzilla, which is more general though.

I'm interested in this too, but I would rather see it integrated to HA/AI rather than lame.sourceforge.net. It could then serve other formats like MPC,AAC and possibly Vorbis (at least for example Garf Tuned versions) as well.

I'm sure also Dibrom would be interested in working with you..
Juha Laaksonheimo

Proposal Of Testing Framework

Reply #2
Also I think the community is more behind Takehiro-Dibrom-alt-preset -axel, than Alexander,  regarding the LAME tweaking.

Takehiro is the only developer anyway, who has done some noticeable quality improvements during the months after 3.92 (especially for 3.94a). Then again alt-preset is the only "tested by organized community and verified quality -preset".

It also seems that Takehiro is really the only developer of Lame who is really interested in tweaking and fixing the quality in major ways in the future also...
Juha Laaksonheimo

Proposal Of Testing Framework

Reply #3
Just wanted to point that I personnaly do not like some assumptions that are made....

Proposal Of Testing Framework

Reply #4
I'm not ignoring this thread, but the reason I haven't responded yet is because I've been to busy to really lay out all my thoughts on this matter.  I'll try and respond sometime later tonight.. though I'm going to be quite busy through most of the entire day, so I can't promise for sure..

However, I do wonder what you mean with your post Gabriel.  If you want someone to be able to provide any sort of useful response, you're going to have to be more specific.

Proposal Of Testing Framework

Reply #5
Do no worry, I do not need any answer.
I just wanted to point that:

"It also seems that Takehiro is really the only developer of Lame who is really interested in tweaking and fixing the quality in major ways in the future also... "

is the kind of sentence that is annoying me a little. Of course due to the word "seems" it is not an affirmation but only an impression, so it is ok.
But please do not transform those sentences to affirmations.

I do not need any answer because I am just indicating my opinion, that's all...

Proposal Of Testing Framework

Reply #6
OK.

Well, I'm not answering, but I'm asking a question.

Since it "seems" this is a mistaken assumption, can you tell us which other developers currently are working on coding big improvements to the psymodel in LAME?  Which developers besides Naoki (who is no longer working on LAME it seems) and Takehiro have been putting significant effort into fixing quality related bugs in the psymodel which seem to have been around (and known about, reported many times by users) for a long time?  Which developers besides Naoki and Takehiro are working on finally adding new features (mixed blocks, IS, improved block switching, new tonality estimation, etc) which again have been talked about for a very long time but which nothing has ever been done about?

Maybe you know something we dont.  On the other hand, sometimes the truth hurts I guess..

Proposal Of Testing Framework

Reply #7
Well, we are moving to off-topic here...

To answer your question, I received some test samples from Robert mainly related to VBR and tonality estimation. I also worked on preliminary IS stuff...

Sometimes we try things that do not work well, sometimes we try things in order to have indication about validity of further work. Some things are preliminary/partial work and can not be put directly into cvs.

However, there is one fact that is sure: Takehiro is doing an amazing work.

Proposal Of Testing Framework

Reply #8
Ok, well this is good news then...

Unfortunately, it's not so difficult to see why it's easy to make these assumptions above... once again, there's really no centralized development plan, so who ever really knows what the other person is working on at any given time?  It seems to me that development is too erratic to ever expect any one thing at a given time and that we can only go on: a) what has or hasn't happened in the past and, b) only assuming that the most visible developmental efforts are actually going to continue to fruition.

I think this underlines a possible issue with implementing an idea like what olcios proposed.  Unless there is going to be better organization and structure in solving the problems which will be pointed out via such a system, is it really going to be worth the effort actually making it?  It's not going to do any good if we have this nice structure for submitting bugs, but there's no organizational structure for how those bugs should be solved or who certain tasks should be delgated to.

Anyway, I'll try to respond more directly to this again later.. but I think the above point should be considered very carefully..

Proposal Of Testing Framework

Reply #9
lol.. I was gonna answer something, but decided it's better not to.
I say just this: Takehiro is the man.
Juha Laaksonheimo

Proposal Of Testing Framework

Reply #10
I totally agree on:

"development is too erratic to ever expect any one thing at a given time" (but isn't it the difference between a full-time job and a project your only working on free time?)

and on:

"Takehiro is the man"

It seems that we agree on many things this morning....


Btw: Good morning Europe...

Proposal Of Testing Framework

Reply #11
Well, it's fine if IS is added, although it's of course not gonna help mid/high bitrates.
Anyway, the impression I've got from the current Lame development (since 3.92) by following the sourceforge dev mailing list is that Takehiro does most of the innovations and new features which would have true impact on audio quality, and Robert or you fix some errors in these or other things now and then.

Though I understand, it's also a loss that Naoki has left the project, and Robert and you seem to be quite busy. Well, I know you all have done a good job overall regarding Lame.. It's just that the community has this never ending hunger for even better quality.. and at least for me it seems that Takehiro is nowadays the only one delivering real audible improvements. At least I haven't noticed anything remarkable from anybody else, which is not so nice because it's a long time from the last big quality improvement release (Lame 3.90).
http://www.mp3dev.org/mp3/history.html

If this is not the case, please share your info/audio samples.

Also one great thing about Takehiro is that he's in constant contact with the community, and pretty much always available for a chat if needed. Then again Alexander prolly doesn't have much idea about the high quality minded lossy audio community at all..
Juha Laaksonheimo

Proposal Of Testing Framework

Reply #12
Quote
"development is too erratic to ever expect any one thing at a given time"

(but isn't it the difference between a full-time job and a project your only working on free time?)

I don't think so really.  There are a ton of Open Source projects that have clear developmental schedules and organizational structure even though most of the people working on them only do so in their free time.  I just don't think this is a good excuse for a complete lack of structure.  I think the problem now is that the LAME userbase has grown so large that it needs a responsible party to drive development.  It seems to me that many of the LAME developers like the prestige of working on a project which is used by so many people but at the same time don't want to be held responsible and manage it like the big project that it really is (or should be at least).  IMO, this creates kind of a conflict of interest, and I think you can clearly see the results of this...

I just can't really see a system like this working out given all of this.  It would clearly be nice to have a good, organized, and efficient system in place to report bugs and issues, but I just don't think it can work right now.  There are other far more basic problems with the whole developmental process that need to be addressed first in my opinion.

For example, who exactly is going to be held responsible for the problems reported to this system?  Who is going to decide who is responsible?  Who decides which bugs are relevant and which aren't?  How are we going to make sure that efforts to solve these problems don't overlap (a possibility in my mind given poor communication and the "every man for himself" developmental philosophy)?  If everyone right now only works on this in their free time, and we've seen that progress in developing LAME has been rather slow lately (aside from some of the most recent work), then it would seem that there's really not even enough manpower to handle this kind of developmental load... can we even be sure that the problems reported ever will be fixed at all?  And, I think most importantly, are the rest of the LAME developers prepared to work on bugs which may be beyond the threshold of most people's hearing?  Are they going to be willing to continue to push LAME when they may even feel itself that it is "good enough"?

I think it's hard to answer these questions without strong leadership and organization.

Maybe I'm just being overly skeptical, but I'm just not so convinced that this can really work out very well unless other things change first...  but as I said, I'm very much interested in something like this (I've had very similar ideas lately, but for other formats mostly..) if it will be worth the effort put into it.

Proposal Of Testing Framework

Reply #13
What about a new thread like "Lame organization" where you could exlain you proposals about organization and where we could discuss this?
(I think that we should keep this topic on-topic)

Proposal Of Testing Framework

Reply #14
Ouch.  You sound like you're being awfully harsh on the LAME developers (other than takehiro)

I think the bug submission is a good idea, it would help provide a communal area for all the developers, add to it a wiki for the developers to communicate easily with eachother and it could work.  Each of them just needs a line, I'm working on : __ __ __ __ __ __

That way, Gabriel could be working on the preliminary work, testing and whatnot, then his results get put on the wiki for other people, helping them to know what can be done and where it could go.  And with users posting the initial problems that drive the prelim work, it could all work smoothly.

I've had similar problems.  5 people working independantly on some code, although it ended up with nothing getting done.  I tried having a wiki for communication, and it seems it could work.  It's simpler, but slower having just 1 person working on everything, which is my case now.  People can go on my forums though, posting bugs and suggestions, those go to me and the ideas people, then to the coder.  That works quite well tbh
< w o g o n e . c o m / l o l >

Proposal Of Testing Framework

Reply #15
Quote
Ouch.  You sound like you're being awfully harsh on the LAME developers (other than takehiro)


It might seem this way to someone who is less familiar with the situation, but I don't believe anything I'm saying is unfair really.  There are many issues with this whole process which have been this way for over a year now (or longer).....  Nobody else seems to speak up about it, so when I do, it might seem that I'm just out for blood or something, but this isn't true.  I honestly feel that there are serious roadblocks caused by the way things are run right now which are preventing the progress we could be making if things were different.

Quote
I think the bug submission is a good idea, it would help provide a communal area for all the developers, add to it a wiki for the developers to communicate easily with eachother and it could work.  Each of them just needs a line, I'm working on : __ __ __ __ __ __


Yes, it's a good idea by itself.  It's a good tool.  But like all tools, it's only as good as the use that people make of it.  That's kind of the point I'm trying to emphasize.  Right now, I don't see a lot of indication that simply creating this tool will solve much of anything in regards to the more key problems.  A bug submission system needs to be very closely monitored and managed for it to really be of use.  Similarly, a wiki needs to be updated rapidly and often if it is going to be useful... right now, the LAME website isn't even kept up very well IMO (it could be organized a lot better, a lot more informative, etc..).  The website is in CVS also, so it should be nearly as easy to update as something like a Wiki would be...

Quote
That way, Gabriel could be working on the preliminary work, testing and whatnot, then his results get put on the wiki for other people, helping them to know what can be done and where it could go.   And with users posting the initial problems that drive the prelim work, it could all work smoothly.


And why is having this system in place going to allow for this?  There's already a mailing list, so I don't think the problem is a lack of a system for communication, it's a lack of desire for strong communication and coordination by the developers.  How is having a wiki going to make things significantly different than communicating via a mailing list or web forum?  He could be posting his results here already, or on the LAME mailing list... but as far as I can see, there's no mention of what he informed us of earlier in this thread.  The same seems to go for a lot of developments in LAME... the actual development process itself isn't really mentioned, it's after the fact when all of a sudden a new switch appears in the encoder out of the blue...

Quote
I've had similar problems.  5 people working independantly on some code, although it ended up with nothing getting done.  I tried having a wiki for communication, and it seems it could work.  It's simpler, but slower having just 1 person working on everything, which is my case now.  People can go on my forums though, posting bugs and suggestions, those go to me and the ideas people, then to the coder.  That works quite well tbh


Yeah, but like I said, there has to be a desire to actually coordinate efforts in the first place.  I've actually talked about these issues in other threads here before, and the impression I got was that the developers (or at least Gabriel IIRC), actually preferred the extremely loose and non-structured method of development.

Proposal Of Testing Framework

Reply #16
Yeah.. I admit sounding harsh.. sorry about that.

Anyway, another issue..
There are practically 2 levels of Lame audio quality development.
The preset development and the non-preset development (psychoacoustics/blockswitching/joint-stereo/noise-shaping/etc). These are on the other hand separate, on the otherhand walk hand in hand (depends also who you ask).

Now that I have thought this more, I think that at least Tak would be happy about this kind of sourceforge system, which would help him to develope and test his new stuff. Then again that may or may not have an impact on presets, which then again needs specific testing.

Thing is that there's not much history for non-preset large scale testing. I somehow was immediately thinking only the presets because A-Ps was done with close co-operation with Naoki.

Well anyway, I don't see that HA has much to say to non-preset testing anyway. Now that I think of it, it might be even a good idea, and naturally non-preset core psychoacoustic testing would be best at lame.sourceforge.net. If anything it would probably help Takehiro, because I have a feeling that he needs more testing help.
Juha Laaksonheimo

 

Proposal Of Testing Framework

Reply #17
I do not prefer non-structured method, but I would not like a strong leadership.
I prefer a kind of peer-leadership.

I also have nothing against organization, but organization takes time to manage. So who is going to manage that? I can not be me because I only have about 3 hours a week for Lame.
I am open to any proposition...

Proposal Of Testing Framework

Reply #18
You're right dibrom, I'm just going on what I read here, and what little I know about the LAME development, sorry for stickin the ol' nose in :o)

If the developers are willing to communicate more about their progress and have someone coordinate things, that should help?  I think the key idea is to not step over the line between coordinating and managing.  If the developers get into regular contact, they can arrange what to do amongst eachother :o)  Peer-leadership as Gab sais..
< w o g o n e . c o m / l o l >

Proposal Of Testing Framework

Reply #19
Quote
So who is going to manage that?

Why not Dibrom ?

Proposal Of Testing Framework

Reply #20
Olcios:
I read your latest comment at Lame dev mailing list. What would you see as HA's role in this?

Anyway, there should also be developers opinion about this system. Would they see this as a good thing. And do they agree that this kind of system should be created.

I talked with Takehiro, and asked what he thinks, and I got a bit surprising answer.. If I was you, I'd first try to find out are all the developers even enthusiastic about this kind of system.

I'll try to get Takehiro to write his opinion here.
Juha Laaksonheimo

Proposal Of Testing Framework

Reply #21
Quote
What would you see as HA's role in this?


Every alpha version released on HA to be released and tested on the framework, not on the forum.

I don't want the framework to become "dead" but it seems probable that this is exactly what would happen. I'm currently investigating some issues from the TODO file, which hold a higher "usefulness/amount of work" ratio.

I mailed Takehiro today, I'll see what he'll reply.

Bye,
Olcios

Proposal Of Testing Framework

Reply #22
Quote
Quote
So who is going to manage that?

Why not Dibrom ?

Agree.
Dib, you are the person who knows what is needed and what is the leadership, and you feel some trouble with that there's no leader.

I don't feel any trouble at all. I am not good at communication in English. I am not good at listening. I have no leadership. I am only a weekend programmer, enjoying writing code.

And even if someone lead the project and tell me some problem to fix, I cannot grantee the answer. Maybe if I feel it interesting to fix the problem, I will try. But if not, I never want to try. BANZAI "free" software

And anyone (not only Dibrom, but also ANYONE) who feel the problem, I am welcomed to  be as the leader (but I cannot grantee I will follow you ). If YOU are not sutisfied about LAME development structure, why not you lead the discussion on lame-dev ?

Again, I am only the weekend programmer, not the project leader. I have no idea how to lead the hobbiest developpers and realize the "STRUCTURE".

Sorry about quite off topic reply. I am deeply getting my idea's into shape about on topic discussion (the testing framework) and it will take one more moment...
May the source be with you! // Takehiro TOMINAGA

Proposal Of Testing Framework

Reply #23
I think that some of you would be interested by reading:
http://www.bcg.com/opensource/BCGHackerSur...4July02v073.pdf

It is the result of a survey about hackers motivation and culture.

It could explain to some of you the motivation of some people for open source programming. This documents is a good description of people like me, and I think that I am not alone in this case.

Proposal Of Testing Framework

Reply #24
I think the quality of the upcoming medium preset will let everyone know how successful the collaboration is.  The medium preset will be raw at first but with proper testing should yield a great quality with a low bitrate after a while with the new code and the ideas for reduced lowpass frequency. 

mp3