Using x264’s color coefficient flagging properly

OK so you have a HD transport stream, you encode it and don’t use colormatrix() in Avisynth because you realized back in 2008 or so that doing so was kind of a dumb idea. Now you want to tell decoders what color coefficients the video uses so it will (hopefully) look correct everywhere, even if you downscale it so ffdshow’s/Haali’s resolution-based autodetection fails. What to do?

The answer to this is obviously “flag it in the h264 bitstream” (if you’re one of those scrubs who still use Xvid just ignore the issue, nobody cares about Xvid anymore). The problem with this answer is that nobody understand what the hell vui.txt actually means. However, in some recent conversation in #darkhold Kovensky linked to a paste where pengvado/akupenguin explained it in more detail.

More specifically:

  • –colormatrix specifies what coefficients are used when converting YUV to RGB. This is the one you want to use.
  • –colorprim specifies what color primaries (see for example the Wikipedia page about sRGB) the RGB uses. Setting this properly most likely requires knowledge of the studio equipment used to master the stream and/or its settings, so just leave it undefined. No one without a color calibrated monitor would ever notice or care, even if decoders/renderers actually supported this.
  • –transfer specifies the gamma curve used for the RGB. Like –colorprim, setting this correctly probably requires knowledge you don’t have, so don’t set it at all. Again, only people with calibrated studio monitors will ever notice or care.

TL;DR: Use --colormatrix bt709 if your source is a HD transport stream; --colormatrix bt470bg if it is an SD transport stream.

EDIT 2011-05-31

JEEB informs me all ARIB broadcasts (i.e. Japanese ones) are always supposed to use Rec.709 regardless of resolution. So you should always use bt709 there.

Comments (6)

  1. Thesis wrote:

    Good tip for us that work with TS.
    But I just got curious, this thing is just when working TS@mpeg2 to h264? Or could we use it when encoding lags to h264 when lags comes from a TS? Or it is just useless in that case because we specify yv12 on lags?

    Thanks in advance.

    Tuesday, October 20, 2009 at 07:34 #
  2. uhwhat? wrote:

    heres my 50 cents about this,
    like thefluff have said, “Use –colormatrix bt709 if your source is a HD transport stream; –colormatrix bt470bg if it is an SD transport stream.”

    but if you have a hd transport stream with bt709, and you believe this series is just an upscale and youre just goin to encode it as SD resolution, 480p, youd change it bt470.

    Sunday, October 25, 2009 at 23:05 #
  3. @Thesis: I think you’re completely misunderstanding this whole thing. The source being a .ts or not has absolutely nothing to do with it…

    Basically in simplified terms, RGB -> YUV conversion is done through some formula, but that formula is different for HDTV and SDTV (uses different coefficients). So if you decode assuming the wrong coefficients, your colors will be slightly off.

    Using an intermediate lagarith step does not change the coefficients used by the video, so I don’t see how it would make any difference. Hell, even if (for some unthinkable reason) you went… original -> lags -> huffyuv -> XviD -> DivX -> VC1 -> RV10 -> h.264… it still wouldn’t make any difference.

    Your video encodes in exactly the same way, with or without the flag. All it does is tell the decoder what coefficients you used so it can decode it correctly, it does not change how the encoder encodes the video, at all.

    Monday, October 26, 2009 at 01:08 #
  4. astrange wrote:

    The options for color primaries and transfer have the same values as colormatrix, I think it should be safe to set them all to the same value. And I think QuickTime 10 does attempt to correct for this.

    This works pretty badly since applying gamma correction to compressed video just adds a lot of visible blocking to the dark areas, so maybe ‘unspecified’ is best after all.

    (technical note: Perian sets all three entirely dependent on the video resolution; I’m going to add VUI reading but not sure if I should trust or ignore it being set to ‘unspecified’ there)

    Monday, February 15, 2010 at 00:15 #
  5. Anonymouse wrote:

    Hi, Fluffy,
    May I ask, what resolution is ffdshow/haali’s autodetection based on?
    I mean, if I have an anamorphic video, should I set the –colormatrix according to the encoded resolution, or the display resolution?

    Thursday, June 17, 2010 at 01:00 #
  6. toph wrote:

    thank you as always for the rare tips

    btw, in case someone needs to alter an existing video’s H.264 video track bitfields there is a Direct264 tool at http://forum.doom9.org/showthread.php?t=152419

    Saturday, July 31, 2010 at 07:11 #