Sunday 13 April 2014

A short story of interlacing in HEVC

And how it nearly ruined another video coding standard.

Everyone knows that interlacing was a trick in the '80s for pseudo motion compensation with analogue video. This more or less worked because it mimicked how television worked back then. This technique was preserved when flat panels for pc and tv were introduced, for a mix of backward compatibility and technical limitations, and video coding features interlacing in MPEG2 and H264 and similar.

However as with black and white, TACS and Gopher, old technology has to be replaced with modern and efficient technology, as a trade off of users' interests and technology providers' market prospects. In case you are not familiar, interlacing is a mess to support, makes decoding slower and heavily degrades quality. People saying that interlacing saves bandwidth do not know much about video coding and bad marketing claiming that higher resolution is better than higher framerate has an effect too.

So, when ITU and then MPEG set out to establish the mandates for a new video standard capable of superseding H264, it was decided that interlacing was old enough, did more harm than good and it was time for retirement: HEVC was going to be the first video codec to officially deprecate interlacing.

Things went pretty swell during its development, until a few months before the completion of the standard. A group of US companies complained that the proposed tools were not sufficient (a set of SEI messages and treating fields like progressive frames) and heavily protested with both standardisation bodies. ITU firmly rejected the idea (with the video group chair threatening to step down) while MPEG set out to understand the needs of the industry and see if there was anything that could be done.

An ad-hoc group was established to see if there was any evidence that interlaced coding tool would have improved the situation. Things looked really shady, the Requirements group even mentioned that it was the first time that an AhG was established to look for evidence, instead of establishing an AhG because there was evidence. Several liasons from EBU and other DVB members tried to point out this absurdity while the threat of adding interlacing back in HEVC became real. Luckily the first version of the specifications got published in the meantime, so this decision didn't slow down the standardisation process.

Why so much love towards interlacing? Well in the "rebellious" group defence, it is true that interlaced content in HEVC is less performant than in H264; however it is also true that such deinterlaced content in HEVC outperforms H264 in any configuration. Truth is that mass marketed deinterlacers (commonly found in televisions for example) bring a lot of royalty income, so it is normal that companies with vested interests would prefer to have interlacing in a soon-popular video standard like HEVC. Also in markets like US where the network operator (which has control on the encoding but not on the video source) might differ from the content provider, it could be politically difficult to act as a carrier only if you have to deinterlace a video.

However these problems are actually not enough for forcing every encoder, decoder, analyser to support a deprecated technology like interlacing. Technical problems can be solved with good deinterlacers at the top of the distribution chain, while political ones can be solved amending contracts. Plus having progressive only video will definitely improve quality and let the industry concentrate on other delicate subjects, like bit depth, both properties going in favour of users' interests.

At the last MPEG meeting, the "rebellious" group which had been working on reintroducing interlacing for a year provided no real evidence that interlaced coding tools would improve HEVC at all. The only sensible solution was to disband the group over this wasted effort and support progressive video only, which is what happened luckily. So now both ITU and MPEG support progressive video only and this has finally nailed it.

Interlacing is dead, long live progressive.

Written by Vittorio Giovara (projectsymphony@gmail.com)
Published under a CC-BY-SA 3.0 license.

3 comments:

  1. ------------------------------------------------------------------------------
    And how it nearly ruined another video coding standard.
    ------------------------------------------------------------------------------

    ^^ What the hell is that ?

    ------------------------------------------------------------------------------
    Everyone knows that interlacing was a trick in the '80s for pseudo motion compensation with analogue video.
    ------------------------------------------------------------------------------

    ^^ Ouch !!! After reading the subtitle, I knew where I was, but reading this first sentence almost killed me.
    So, for you interlacing is a trick from the 80's ? I suppose you were not born and you think that TV came to life with you... In fact, you are off by at least 40 years... That's a detail for a video coding expert like you.

    ^^ I learned one thing here, "Interlacing" is a "pseudo motion compensation". Grandma should be happy with her CRT, I should tell her that her old tv set, does motion compensation (which is basically interpolating = creating new information between 2 images to make it smoother). Maybe I should take a look at her TV to see what DSPs are in there to do this task, that may be alien tech from the 80's.
    Interlacing in not motion compensation but field dropping. You keep motion smooth by keeping 50 temporal informations but dropping half of spatial resolution. There is no "motion compensation" or any kind of "interpolation"in here...

    ------------------------------------------------------------------------------
    This more or less worked because it mimicked how television worked back then.
    ------------------------------------------------------------------------------

    ^^ More or less ? Let's just say it worked and stills work and it has been working for at least the last 50 years... In fact if you didn't tried to watched your interlaced footage on your progressive screen computer, you may never have noticed that...

    ------------------------------------------------------------------------------
    In case you are not familiar, interlacing is a mess to support,
    ------------------------------------------------------------------------------

    ^^ That's true. Doing things correctly is harder than always choosing the easiest way whatever the result is.
    You find it difficult to support so you want to drop it... We are lucky enough that you don't find it hard to support color or to support 50Hz...

    What about supporting only one format, one framerate, one frame size ? It makes it easier to only have 59.97frames per sec for example in 1920. To make easier for you, let's use RGB color space the encoding, this removes the nasty color matrices. To make it easier for your CPU, let's make it some power of two, what about 2048x2048 ? 16/9 is so 80's. 20148 is a perfect fit for your GPU also, that should make your 8x8 pattern matching easier...

    ------------------------------------------------------------------------------
    makes decoding slower and heavily degrades quality.
    ------------------------------------------------------------------------------

    ^^ Could you elaborate on how decoding 50fields @ 1920x540 is slower than decoding 50Frames @ 1920x1080 (twice the surface, twice the bitrate) ? Or maybe you wanted to compare 50i to 25P which is not really relevant for fast pace actions like sports... I know many broadcasters uses 25P crappy content from Youtube for their content, but you must be kidding me if you want to use 25P has the new TV standard...

    ------------------------------------------------------------------------------
    People saying that interlacing saves bandwidth do not know much about video coding
    ------------------------------------------------------------------------------

    ^^ HD-SDI is 1.5Gbits in 50i, but it is 3.0Gbits in 50P. Period.
    Interlacing saves on bandwidth.
    Video coding is not only about compression.

    ReplyDelete
  2. ------------------------------------------------------------------------------
    and bad marketing claiming that higher resolution is better than higher framerate has an effect too.
    ------------------------------------------------------------------------------

    ^^ If you think that HD is marketing hype, and if you don't see the difference between 25 and 50 frames per second, maybe there is a bigger problem than HEVC to solve first.
    Interlacing trick allows to have the best of both world : Higher definition for slow action because the two fields forms an hi def image (the two 1920x540 fields forms a 1920x1080 image), Higher smoothness than 1080p25 because of it's higher field rate (50).

    ------------------------------------------------------------------------------
    Why so much love towards interlacing?
    ------------------------------------------------------------------------------

    ^^ Why so much hate towards interlacing ?

    ------------------------------------------------------------------------------
    Well in the "rebellious" group defence, it is true that interlaced content in HEVC is less performant than in H264;
    ------------------------------------------------------------------------------

    ^^ How is that possible ? As you said : "People saying that interlacing saves bandwidth do not know much about video". Could you elaborate on this ?

    ------------------------------------------------------------------------------
    however it is also true that such deinterlaced content in HEVC outperforms H264 in any configuration.
    ------------------------------------------------------------------------------

    ^^ You mean that HEVC, when deinterlacing (dropping half of the fields and half the resolution), it can do better that h264 ? What an achievement. Where can I buy this ?

    ------------------------------------------------------------------------------
    Truth is that mass marketed deinterlacers (commonly found in televisions for example) bring a lot of royalty income, so it is normal that companies with vested interests would prefer to have interlacing in a soon-popular video standard like HEVC.
    ------------------------------------------------------------------------------

    ^^ And now comes the conspiracy theory. What bring royalties is not deinterlacing but motion compensation, Mpeg royalties, color processing, sound processing, not just pixel doubling ...

    ------------------------------------------------------------------------------
    However these problems are actually not enough for forcing every encoder, decoder, analyser to support a deprecated technology like interlacing.
    ------------------------------------------------------------------------------

    ^^ Deprecated ??
    Wiki: In the United States, 1080p over-the-air broadcasts still do not exist as of March 2014;
    all major networks use either 720p60 or 1080i60 encoded with MPEG-2.

    ------------------------------------------------------------------------------
    Technical problems can be solved with good deinterlacers at the top of the distribution chain, while political ones can be solved amending contracts. Plus having progressive only video will definitely improve quality and let the industry concentrate on other delicate subjects, like bit depth, both properties going in favour of users' interests.
    ------------------------------------------------------------------------------

    ^^ You are right let's make all videos crappy as web videos (shoppy and blurry), 25P is the way to go and let's focus on bit depth because all people should agree that funny blurry videos should be billions of colors because the 10bits of luma are not enough to encode blue sky gradients after 16 decode/encode cycles.

    ReplyDelete
  3. ------------------------------------------------------------------------------
    Interlacing is dead, long live progressive.
    ------------------------------------------------------------------------------

    ^^ What an achievement! Developers that don't understand a thing to video will have an easier job and will focus on bit depth...

    I think you clearly do not have the big picture in mind. The question is not : "Can we remove interlacing because it's hard to manage" but "what is it the best for final end users and how can we achieve that". I totally understand your point and know exactly how hard things can be when you want to handle fields properly. But the fact is that having files 30% bigger, or having one third less channels on my tv to make your life easier is not really my problem. Do you prefer 1280x720@50P or 1920x1080@50i ? If you are not blind, you know that 1080i is better (but harder to deal with). Better for slow motion because you get finer details, better than 1080p25 with smoothness when watching high speed motion sequences. Human eye don't perceive information's like a CCD. The eye resolution drops when watching moving objects. Eye is less sensible to color informations than to luminance informations. That's why interlacing still exist today (an chroma subsampling) even if some of the initial technical reasons disappeared. If you want to make your job easier, do it progressive, in RGB, 4:4:4, and let's forget about the end client's eyes (or more precisely his brain).

    Thanks for this really funny post.

    Stephane

    ReplyDelete



All the projects here are under a Creative Commons 3.0 licence! You can use and distribute them as you like (just quote the author so he knows his work is not useless)!

If you wish to get in touch with me write at projectsymphony@gmail.com