Post
by Mister Hatt » Tue Jan 03, 2012 5:47 am
Ok so time for my new year Grain is Not a Defect talk, but before we get there, stop using JPEG screenshots as they mask any actual problems. You've run into a few issues here and they stem from two things. The first is your oversmoothing and producing subtle gradients. The second is DCT quantization accuracy at a given bit depth. This post is going to be a bit wordy but please read all of it as I'll explain what your problem was and what caused it, as well as how to fix and avoid this in future. It's something a lot of amv encoders seem to mess up.
Let's start with the first thing you complain about, detail going to shit in x264 but being fine in Huffy. You seem to be blaming H.264 itself for some reason but that doesn't really make sense. Huffy is a lossless codec, it encodes everything exactly as it was input, providing 100% accuracy to your original content when decoding. H.264 (AVC) is lossy and only estimates the original content. Depending on the encoder, various tricks or algorithms are used to make this as visually accurate as possible as well as to provide better compression, x264's RDO is a good example of this. The way AVC works is to break your picture into large blocks called macroblocks. These are then broken down into their component chunks of data and quantized, essentially creating a new set of data that closely resembles the original, but in less detail. Think of it as averaging curves and colour variations in a way. Lossy codecs however are not very good at compressing detailed content without either being relatively large, or by dropping more detail than you'd prefer. Depending on the number of bits of accuracy the codec has, you will get more or less detail in your content for a given bitrate. This is where 8bit or 10bit image accuracy comes in. 8bit gives you 256 values per channel per pixel while 10bit gives you 1024; 4 times as much for little change in filesize. Grain itself is complex but can be handled relatively well by various patches depending on the type of grain so I don't think it's much of a problem to leave it in. Things like static grain maps and p/b frames and RDO make a huge difference here.
You've sort of gone the opposite and smoothed everything. This might have seemed like a good idea except for two things. Firstly, grain while sometimes excessive, also tricks the eye into thinking there is more detail than there really is. It also hides large imperfections. Secondly, large chunks of grain are far easier to compress than very subtle changes in gradient which is what happens when you smooth things badly. Now, depending on the amount of grain, you may indeed want to reduce it, but the way you've done it here at least is not the right way. Besides for massively distorting your image (seriously, those lanters in your above screenshot, what the hell), you've also gone and created micro gradients that you probably can't even see that well. x264 however notices them, and in 8bit quantizing, it tries to average them too much, which is where the banding is coming from. You can use 10bit here for greater accuracy and most likely eradicate this banding altogether, or you can use light grain which is what I would recommend.
I've said it a lot before but grain is not a defect. It has many uses and should generally be left alone. That said, some companies abuse the fuck out of it. You've obviously gone full bore in removing it as much as possible with pretty much the worst filters possible. You've also then gone and given bad filters much smaller sample sizes to work with, making the effect even more noticeable. What you SHOULD be doing is filtering at full resolution and then dropping down to your export res with spline36resize. I would recommend either ttempsmooth if you're in a rush and/or your grain isn't REALLY that bad (post an unedited screenshot for advice here) or otherwise hit it up with a well tuned MCTD followed by a slight AA sharpen of some sort, probably EE() would be a good choice here. You need to be really careful to only remove grain and not detail, which is what these filters are good at. Otherwise you'll introduce gradients which as I mentioned above will cause x264's quantizer to shit all over it. Depending on how much grain you have, I would also recommend GrainFactory or something after resizing so as to trick the quantizer a bit more, but it really depends on individual levels of grain here as well as the type.
Summary: you're overfiltering and this is requiring MORE bitrate than just leaving it alone. Black gradients are hard to see on many monitors so it looks to you like there is no detail to die on. Leave some grain in or remove it properly and then add it later. Fix your filter order.
Let me know if you have any questions.