Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Calling for ideas (Read 6941 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Calling for ideas

The current situationis that there is no such thing as ABX for video. There are some non-subjective video quality measuring tools like Video Quality Studio. The problem of these tools is that their measurements aren't the same thing as true video abxing. The solution for these video quality measurement problems would be to create some kind of abx(/hr) tool for video. Here are several of my own ideas for such tool:


1.  Still frame rating (the user would have to rate a still frame without knowing the codec)

2. Clip rating (the program would play a little clip from the film like the audio abxing tools do)

3. Calculating PSNR/something else (just suggestion)


Please, feel free to post your own ideas for a tool like this.
Some general info about video artifacts: http://www.everwicked.com/content/QualityMetrics/

Calling for ideas

Reply #1 –
How do you ABX video?  I mean, you need to have an "original" which is perfect quality.  But what do you use for the "original"?  DVD-quality?  That's compressed using MPEG-2 and has its own distortions.

Calling for ideas

Reply #2 –
Quote
How do you ABX video?  I mean, you need to have an "original" which is perfect quality.  But what do you use for the "original"?  DVD-quality?  That's compressed using MPEG-2 and has its own distortions.

Umm.. I wouldn't call many CDs "perfect quality" either (over-compression etc effects).
Original would be "the source", in this case DVD.

Subjective comparison has also its place in video coding, but the issue is how to interpret results like PSNR, VQM, SSIM. Simply taking the average is far from ideal, and I see people using the average values almost everywhere. It really needs some more complex statistics.
Juha Laaksonheimo

Calling for ideas

Reply #3 –
Some ideas:
* Sharpen Filter (with this filter we could sharpen the image to see which video present less artifacts, for example)
* A system to identify motion scenes (to jump straight to them)
* Frame position memorization
* Zoom

I don't know if these ideas are valid or not, I hope they're useful
[ Commodore 64 Forever...! ]

Calling for ideas

Reply #4 –
@Oge_user: Sorry, but I don't think any of these are very good ideas.

* Sharpening will bring forth artifacts, but it doesn't really say anything about the quality of the video. You might as well subtract the compressed video from the original, and say that "the greater difference, the worse quality", which is not always the truth.

On a similar note PSNR and SSIM measures can give 'hints' to the video quality, but should never be used for judging a codec.

* High-Motion-Scenes: Also not a good idea. 2-pass VBR is a tradeoff of the quality of the ENTIRE movie. A codec could assign many bits to high-motion scenes and get great scores at the expense of low-motion scenes.

* Zoom. IMO it is a common mistake to Zoom when comparing. Quality should be compared as a whole, not just how a single codec handles a particular part of a frame.

I usually use side-by-side comparison for somparing quality, by loading both compressed movies into AviSynth and doing a "stackhorizontal". That way I can do a frame-by-frame or sequence-by-sequnce comparison while the movie is playing.

However - judging video quality is not something everyone can agree at. There are many tradeoffs. The main tradeoff points are Sharpness vs Blockiness vs Picture stability. For example, many people like the DivX4+5, while my personal preference is that is smooths too much.

An ABX-like application could however be done. It should be possible to open any number of sources and comparing some sequences two sources at the time.
Regards, sh0dan // VoxPod.  AviSynth 2.5 developer.

Calling for ideas

Reply #5 –
Do you think that a program would also need a hidden reference or even a non-hidden reference. Non-hidden reference would be good because the source material may also contain some errors.

Calling for ideas

Reply #6 –
The main issue with an abx video tool is that we would need losslessly compressed sources to prevent some cheating. And lossless compression in video is something very expensive.

About the options, no still picture, no zoom, no sharpening.

Video is something that works in motion. It's quite useless to check still pictures. as you generally don't pause a movie. And some artifacts seen in still frames are completely hidden in motion. This also why most video quality metrics can't be used here as they don't take motion into account.
It's a 'Jump to Conclusions Mat'. You see, you have this mat, with different CONCLUSIONS written on it that you could JUMP TO.

Calling for ideas

Reply #7 –
I could start to program something in two weeks. (I do have the exam weeks starting at next week) The two programming languages that have got my interest are Delphi and Java. The Delphi version should be pretty simple to program using Direct Show components from http://www.progdigy.com . The Java version would require more time and I'm not sure is there any need for a cross-platform solution.

Calling for ideas

Reply #8 –
for starters, it would be quite easy to change my abxing javascript into a video version. 

(zoom, still frames and stuff like that are pretty useless imho, you need to compare two motion videos running without any dsping.)
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Calling for ideas

Reply #9 –
(Thread Resurrection)

Any progress on blind video comparators?

I think there are two useful test modes:

1. Comparision of obvious differences and rating. Display the reference and up to 3 (depending on processor power and display resolution) different codecs at the same time.
Blind testing is probably only needed in so far, that the sample<->codec-name relation is invisible (You don't know whether the blury picture is Xvid or RV9 ).

2. Test if difference is perceivable at all. Probably most useful for comparing different settings of the same codec.
I wonder if ABX is the best technique in this case. A simple is equal/is different decision would allow the reduction to two videos at the same time (thus saving screen space) and would give the same statistical significance.

Calling for ideas

Reply #10 –
And another resurruction.

I have something for subjective evaluation in the works since a long time ago but never had time to "finish" it.

Will try to spend some time in the weekend.

everwicked

Calling for ideas

Reply #11 –
Quote
You don't know whether the blury picture is Xvid or RV9


Calling for ideas

Reply #13 –
Thank you!

Not exactly what I expected though. From what I see the files are played RARA RBRB and so on, with an option to vote after each group. Is it possible to jump between the files somehow?  I fear, that evaluating a sample without seeing the others is rather difficult. I think, the user should be able to choose, what files are being played.

Calling for ideas

Reply #14 –
Im not sure what you mean with jump between files...

Yes, the sequence is RARA etc, it's actually a formal assessment method called Double Stimulus Continuous Quality Scale Method.

Of course, I am going to add more.

You evaluate the sample against the reference, not against the others...

Calling for ideas

Reply #15 –
everwicked is right;no point in seeing all of them at the same time....or switching between samples.....

you must look one sample at the time and compare it to source(and with marks,and not by direct side-by-side comparison against it)...

that's that method anyway...

Calling for ideas

Reply #16 –
There is another method that involves playing the samples side by side...

I will implement that in the future I hope

By the way, I re designed the webpage and added a bit more of info, have a look.

Calling for ideas

Reply #17 –
I come from the audio side, so maybe I'm completely off here, but I can't see much reason for the current test scheme. How am I supposed to give consistent ratings, if I can't compare between contenders?

Example: I have a reference (DVD, DV, TV-capture) and want to compare different encodings (Divx, Xvid, with/without b-frames, etc.). It's not sufficient to compare each of those just to the reference, my ratings might change over the length of the test, and I have no way to compensate for that.

I think something like ff123's ABC-HR would be more useful, where the play-buttons start the video. (A hidden reference is probably not needed in most cases, though.)

Quote
no point in seeing all of them at the same time....or switching between samples.....

This is one of the great advantages and simplifications with video comparison! You can't do that with audio, but here, the only reasons against it are, what I wrote above: screen resolution and processor power.

Calling for ideas

Reply #18 –
i was just describing THIS method...

Quote
my ratings might change over the length of the test

doesn't this mean that you really didn't saw any difference....i mean when the brain starts to toy with you,you know every codec looks ok.....
heh no such troubles here,i can spot xvid/divx4/5 from miles....(wink)

i usually don't look sources at all,as i know  exactly what sources look like;as it happens they look sharp and don't have blocking....ie.VERY different than encodings....so i try to see which encoding is sharpest,and with less blocks....that encoding is closest to the source,can't miss it really...(i wanna say this;comparing encoding to source will always get you SAME result;ie. encoding won't trick you into believeing you're seeing source.......i'm talking about mjpeg or huff versus mpeg's here...so one should concentrate on encodings more than source vs. encodings....)

Calling for ideas

Reply #19 –
Quote
Quote
my ratings might change over the length of the test

doesn't this mean that you really didn't saw any difference....i mean when the brain starts to toy with you,you know every codec looks ok.....
heh no such troubles here,i can spot xvid/divx4/5 from miles....(wink)

But the purpose of this test is not to spot the Xvid, etc. encoding in the sample suit! It's to find out, what looks better! If both codecs make different errors, I'd like to swap to and fro, and then decide.

And what about testing different settings, where the quality difference is rather subtle? If answering is too easy for you, you have to ask more difficult questions .

Calling for ideas

Reply #20 –
Quote
I come from the audio side, so maybe I'm completely off here, but I can't see much reason for the current test scheme. How am I supposed to give consistent ratings, if I can't compare between contenders?

Example: I have a reference (DVD, DV, TV-capture) and want to compare different encodings (Divx, Xvid, with/without b-frames, etc.). It's not sufficient to compare each of those just to the reference, my ratings might change over the length of the test, and I have no way to compensate for that.

I think something like ff123's ABC-HR would be more useful, where the play-buttons start the video. (A hidden reference is probably not needed in most cases, though.)

Quote
no point in seeing all of them at the same time....or switching between samples.....

This is one of the great advantages and simplifications with video comparison! You can't do that with audio, but here, the only reasons against it are, what I wrote above: screen resolution and processor power.

Formal methods like the DSCQS are used by the MPEG to validate the validity of video coding and they are not that suitable for multiple codec comparisons but rather single codec multiple sample ones. There are other methods in my book but they all have the same philosophy.

If you perform the same test for all codecs though it should provide you with pretty "objective subjective" results since you're always comparing against the reference.

Yet, ABC/HR methods cannot be applied to video IMHO. As stated in previous posts, differences in video are much easier to spot than in audio.

Calling for ideas

Reply #21 –
0.32 is out.

Quote
Objective:

    * The video interface used can be chosen while opening the reference/test files
    * Other minor GUI and internal alterations

Subjective:

    * DSCQS is now implemented as described in ITU-R BT.500 standard (order is randomised, two scales of 0-100, score is difference)
    * The user can exit the test using ALT+F4
    * An option to enable/disable resizing was added
    * Other minor GUI alterations


With this version you can effectively attempt to ABX two videos since the order is randomised and the result is the difference of the scores.

Thus, if you get a negative result it means you scored the impaired sequence with a lower score than the reference one

Calling for ideas

Reply #22 –
Hi Everwicked, thanks for working on a blind video testing tool. Such a tool has been conspicuously absent from most codec comparisons.

Here's what I'd really like to see:
  • The program allows me to load an .avs and several recompressed videos generated from the .avs. So far, it looks like your program does this.
  • Allow me to use a timeline bar, like ABC/HR, to choose a specific segment of the video to examine.
  • Set up the play buttons and quality sliders like in ABC/HR, so I can repeatedly switch between each sample video and continually adjust my ratings.
  • Have the option to run either in fullscreen or windowed mode. The fullscreen would have a hovering toolbar with the different play buttons on it, like the one in MPC.
I don't know if side-by-side video comparisons are appropriate. Due to color and geometry variations on monitors, and especially on LCD panels, the videos may not receive equal playback treatment. Also, I wonder whether most video overlays can handle the playback of multiple videos simultaneously. I just tried playing two copies of the same RV9 video clip in two different instances of MPC, and the colors in each clip were distinctly different. Adjusting the color sliders on my video card's overlay control panel seemed to have no effect. (Geforce3, latest DirectX and drivers.)

I also want the option to choose just a small segment from a large video, because IMHO testing movie trailers or small video clips fails to examine each codec's rate-control abilities. I could manually trim segments from each video clip, but this is very time-consuming.

For low-bitrate encodes, such as RV10, Xvid, and Divx, there is no need for ABX against the original, or for hidden references. ABXing one encode against another is necessary, though, especially when testing the same encoder with multiple similar configurations. For testing MPEG-2 recompressors, an ABX against the original may be necessary (I don't know, I haven't used a DVD re-coder before).

Thanks for listening. I'd like to hear your thoughts.

Calling for ideas

Reply #23 –
Quote
Here's what I'd really like to see:
  • The program allows me to load an .avs and several recompressed videos generated from the .avs. So far, it looks like your program does this.
  • Allow me to use a timeline bar, like ABC/HR, to choose a specific segment of the video to examine.
  • Set up the play buttons and quality sliders like in ABC/HR, so I can repeatedly switch between each sample video and continually adjust my ratings.
  • Have the option to run either in fullscreen or windowed mode. The fullscreen would have a hovering toolbar with the different play buttons on it, like the one in MPC.
I don't know if side-by-side video comparisons are appropriate. Due to color and geometry variations on monitors, and especially on LCD panels, the videos may not receive equal playback treatment. Also, I wonder whether most video overlays can handle the playback of multiple videos simultaneously. I just tried playing two copies of the same RV9 video clip in two different instances of MPC, and the colors in each clip were distinctly different. Adjusting the color sliders on my video card's overlay control panel seemed to have no effect. (Geforce3, latest DirectX and drivers.)

I also want the option to choose just a small segment from a large video, because IMHO testing movie trailers or small video clips fails to examine each codec's rate-control abilities. I could manually trim segments from each video clip, but this is very time-consuming.

For low-bitrate encodes, such as RV10, Xvid, and Divx, there is no need for ABX against the original, or for hidden references. ABXing one encode against another is necessary, though, especially when testing the same encoder with multiple similar configurations. For testing MPEG-2 recompressors, an ABX against the original may be necessary (I don't know, I haven't used a DVD re-coder before).

Thanks for listening. I'd like to hear your thoughts.

Hey, thanks for the feedback.

Here's what I think:

- Avs is supported.
- Selection of range will be integrated in some later version
- Windowed mode is a bad idea. Use Options->Do not resize instead.

Also, none of the standards gives freedom to the user to play/pause etc. ABX would not work so well for video, I have stated my thoughts previously in this thread about this. With DSCQS you can however see if you identified the videos correctly. It's not your sole purpose like when ABXing but it does give you the same result if you like. I think that way makes the test less biased too.

There is a metric that displays the videos side by side. I haven't tried to program it to see what it looks like but if worst comes to worst I can use some low-level blitting to display the video. In a later version

As for low-bitrate encodes you're right, ABXing is not any good but DSCQS is more than that and I am planning to add more metrics.

It's always good to exchange opinions with interested users

Calling for ideas

Reply #24 –
Quote
Also, none of the standards gives freedom to the user to play/pause etc. ABX would not work so well for video, I have stated my thoughts previously in this thread about this.

Okay, I finally understand why ABXing videos is inappropriate--you just had to repeat yourself four times before I got it.  Basically, anyone can just look for a slight difference in macroblocks and be able to distinguish two video clips that are very similar.  The omission of a pause feature also makes sense, since the idea is to compare videos, not still images.

Tomorrow I'll make some movie snippets and see if, by running several double-stimulus tests, I can get a meaningful statistic. It will be interesting to see how easily I can distinguish between similar encodings, because I've never done a blind video comparison before. I'll read through your site and forums, now that I see you've been working on this project for some months.

I'm very much looking forward to the addition of a range selection feature...