Archive

Archive for December, 2011

avs4x264mod v0.6.2 ( Add “–seek-mode” )

29th December 2011, Thursday Leave a comment

Download links :

GitHub: avs4x264mod-0.6.2-git-r43(7e0284f).7z
MediaFire: avs4x264mod-0.6.2-git-r43(7e0284f).7z
NMM-Mirror: avs4x264mod-0.6.2-git-r43(7e0284f).7z

v0.1:
– Fix capability with high bit depth avs.
– Add parameter to customize x264 binary.
– Print full command line used by x264.

v0.2:
– Delete ugly blank space in command line, which was generated in v0.1 if “–x264-binary” specified.

v0.3:
– Fix invalid x264 binary path when both avs4x264 and x264 binary is given by full path
– Add switch -L as a short name of –x264-binary, e.g., -L C:\x264.exe is equal to –x264-binary C:\x264.exe.

v0.4:
– Directly output i422/i444 with AviSynth 2.6 new csp YV16/YV24. No forced ConvertToYV12 for these two csp any more. Thanks to SAPikachu.
– Display version and help info when run with no options.

v0.5:
– Improve capability with more styles of parameters in x264.
E.g., –tcfile-in=”timecode.txt”, –input-depth=16, –x264-binary=”x264″, -L=x264 and -Lx264.

v0.5.1:
– Fix pipe error with YV12. This bug was introduced in v0.4.

v0.6:
– Do not add –input-res/–fps/–frames/–input-csp if already defined.
– Correct number of frames to be handled when –frames is defined. Now you can specify frame numbers to be encoded without getting errors.

v0.6.1:
— Correct –frames when used with –seek, faster –seek if no timecodes defined.
— Show more script information when opening avs.

v0.6.2:
— Add “–seek-mode” paramters, and some minor bug fixes.

Explanation on “–seek-mode” ( string, “fast”/”safe”, only affects how to deal with “–seek” )

— “fast” mode is similar to x264’s internal method of avs demuxer: skip frames until the frame –seek indicates. However, when used with –qpfile/–tcfile-in, it won’t skip but add a “FreezeFrame(0, seek, seek)” to avs script to actually skip the process of those frames. I have to play this trick because x264 regards qpfile/tcfile-in as qpfiles/timecodes of input files, not output files, so the frame numbers of input piped raw ( can be only linearly seeked in x264 ) has to be left untouched.
— “safe” mode uses a safer but slower method: delivering every untouched frame to x264. When the process of frames before “–seek” frame is heavy, the “useless” running time of processing those frames will be much longer than “fast” mode, but the result is safer for scripts like TDecimate(mode=3), which can only be seeked in a linear way.

Categories: avs4x26x, encode, x264

x264 rev2135+631 tMod ( Merry Xmas and happy new year! )

28th December 2011, Wednesday Leave a comment

Opened a repo on GitHub: https://github.com/astrataro/x264_tMod.

Use win32thread and fprofiled:

tMod/tMod-10bit/tMod+MixAQ/tMod+OreAQ:
Downloadx264_rev2135+631_tMod_v2.7zMultiUpload-Mirrornmm-hd-Mirror

Latest updates:
– Update x264 to git-devel rev2135, as it seems to be even more stable than rev2120;
– Change version head ( I don’t want to stash revision number of my local changes, but revision will be largely different from lsmash’s repo if I don’t hack on lsmash’s original version check, just like vfr_maniac’s Dangerous build );
Add support for unicode characters in file names, using the method LoRd_MuldeR provided. Personally I don’t like this method as all parameters are wrapped in a strange behavior, but anyway, it works; ( 29 Dec 2011, Removed, as it broke non-ASCII charaters in AviSynth script. )
– Remove 422/444 builds. No one cares about 422/444. Use *-all builds if you do;
– Add back “Color matrix detection with ffms and lavf input” patch.

—– How to load subtitles internally –—

These builds ported subtitles video-filter from direct264, and are able to render subtitles internally.
To render subtitles in this way, demuxer must be libav/ffms. Raw/avs demuxer is not supported for now. And VSFilter.dll(for 32-bit builds) or VSFilter64.dll(for 64-bit builds) must be placed together with the x264 binary or in the system path. Do NOT change filenames of them.
Rendering multiple subtitles is possible. Now you can render at most 16 subtitles simultaneously by calling –sub for each subtitles. Note that the later calling of –sub overlays the subtitles after previous calling of –sub, so if several subtitles are in the same position, the last one should overlays on all those rendered before.
The VSFilters are modified to add custom csri API. Patches can be found in svn of direct264 project. Therefore do NOT replace them with other builds. BT.709/BT.601 is auto detected according to video resolution. For HD videos BT.709 is used and for SD videos BT.601 is used.
Both 8-bit and 10-bit builds can use subtitles rendering, but rendering always takes place in 8-bit colorspaces.
The commandline example to render subtitles:
x264_32_tMod-8bit-420.exe –sub “subtitles-1.ass” –sub “subtitles-2.ass” –sub “subtitles-3.ass” –video-filter subtitles[/Other video filters] [Other x264 options] –output “output.mp4″ “input.mkv”

——– Attention on high bit depth usage ——–

Several days before madshi pointed out that x264′s internal low bit depth to high bit depth conversion and high bit depth to low bit depth dithering is incorrect. The present converting and dithering algorithms seem to work correctly on full range sources of RGB and Y plane of YUV. Till now neither x264 nor swscale has a correct chroma upscaling algorithm for full range sources, and I didn’t make one. Anyway, the converting algorithm of x264 is absolutely wrong on limited range sources, according to ITU-R 601/709′s standard, in which the conversion should be done by simply appending LSBs of zero to the source. The dithering algorithm may also causes overflow on limited range source. So I hacked on my tMod builds to solve this issue.

When the input and output bit depth are the same, no conversion or dithering will be apllied. Otherwise when used with “–fullrange on”, the conversion and dithering will be exactly the same with x264′s original algorithm, which is still not reliable for YUV sources, so it is recommanded to use your own reliable approaches to convert those sources by yourself before passing them to my x264 builds; if used without “–fullrange on”, the conversion will be done by shifting according to the standard, and the dithering algorithm is fixed as well, so that the result should be absolutely right.

astrataro@gmail.com

——————–

Patches:

–—––—–Download(L-Smash not included): patches-rev2135.7znmm-hd-Mirror

00:L-Smash(including audio encoding);
01:More detailed version head (new!!)
02:Add back “touhou” tune
03:Film Grain Optimization
04:Fade Compensation
05:Remove stats before renaming
06:Add a parameter to set level of writing options in User Data Unregistered SEI
07:Utilize internal threading in lavf/ffms
08:Auto VBV settings
09:AVI output
10:filters hqdn3d pad vflip yadif patch
11:Log file
12:Encoding time
13:Level force
14:Profile force
16:New experimental AQ mode (modification of Auto-variance AQ)
17:Correct the limited-range 8->10 conversion algorithm Now
18:Print video info with lavf/ffms demuxer
19:lto & Ofast
20:AviSynth 16bit hack
21:Skip bit depth filter when possible
22:Video filter: subtitles
23:Detect color matrix with lavf/ffms demuxer (updated!!)
24:Fix AviSynth color space converting matrix (new!!)
25:Win32 utf-8 hack (new!!) (Removed, as it broke non-ASCII charaters in AviSynth script.)
26-1:MixAQ-core;(only in tMod+MixAQ version, remove AQDebug)
26-2:OreAQ-core.(only in tMod+OreAQ version, remove AQDebug)

Compiler:
mingw-gcc 4.6.2

Libpack info:
libav r31466 git-723c35f
ffms svn-r597
VSFilter svn-r3698 (MPC-HC)
lame 3.99-3
libvorbis-aotuv_b6.03 (libvorbis-1.3.2)
opencore-amr-0.1.2 r178 git-08d6986
vo-aacenc v0.1.1-7-git-07931e3
vo-amrwbenc v0.1.1-6-git-dfc1623
aacplus 2.0.2
faac 1.28
qtsdk 7.3

—————————————————-
My other tools list ( including some of the x264 builds ) : MediaFirenmm-hd

Categories: encode, x264, x264-bin

x264 rev2120+630 tMod ( Support rendering subtitles internally )

12th December 2011, Monday Leave a comment

These builds are not officially merged by x264_L-SMASH project. I merged in my local git repo so don’t be surprised if you cannot find commits in x264_L-SMASH’s repo.

Latest updates:
– Update x264 to git rev2120;
– Update “skips depth filter when possible” patch, now raw input also skips depth filter when input and output depth are same;
– Add internal subtitles renderer ( ported from direct264 ), read below for more details;
– Add “–profile-force” to avoid x264 automatically reduce user defined profile when it is not needed according to other parameters, so –profile high –no-8x8dct –profile-force makes output profile high but not main, and –profile high422 –profile-force with 8bit 420 output won’t be reduced to high. I made this patch only to show that hi10p is not equal to 10-bit, so stop adding “hi10p” into filename when you actually means the content is 10-bit avc;
– Temporarily remove “Color matrix/range detection with ffms and lavf input” patch, as the new range structure in latest x264 changed the range logic and I need to update many codes of this patch, while I personally have no interest in it.

— How to load subtitles internally

These builds ported subtitles video-filter from direct264, and are able to render subtitles internally.
To render subtitles in this way, demuxer must be libav/ffms. Raw/avs demuxer is not supported for now. And VSFilter.dll(for 32-bit builds) or VSFilter64.dll(for 64-bit builds) must be placed together with the x264 binary or in the system path. Do NOT change filenames of them.
Rendering multiple subtitles is possible. Now you can render at most 16 subtitles simultaneously by calling –sub for each subtitles. Note that the later calling of –sub overlays the subtitles after previous calling of –sub, so if several subtitles are in the same position, the last one should overlays on all those rendered before.
The VSFilters are modified to add custom csri API. Patches can be found in svn of direct264 project. Therefore do NOT replace them with other builds. BT.709/BT.601 is auto detected according to video resolution. For HD videos BT.709 is used and for SD videos BT.601 is used.
Both 8-bit and 10-bit builds can use subtitles rendering, but rendering always takes place in 8-bit colorspaces.
The commandline example to render subtitles:
x264_32_tMod-8bit-420.exe –sub “subtitles-1.ass” –sub “subtitles-2.ass” –sub “subtitles-3.ass” –video-filter subtitles[/Other video filters] [Other x264 options] –output “output.mp4” “input.mkv”

——– Attention on high bit depth usage ——–

Several days before madshi pointed out that x264′s internal low bit depth to high bit depth conversion and high bit depth to low bit depth dithering is incorrect. The present converting and dithering algorithms seem to work correctly on full range sources of RGB and Y plane of YUV. Till now neither x264 nor swscale has a correct chroma upscaling algorithm for full range sources, and I didn’t make one. Anyway, the converting algorithm of x264 is absolutely wrong on limited range sources, according to ITU-R 601/709′s standard, in which the conversion should be done by simply appending LSBs of zero to the source. The dithering algorithm may also causes overflow on limited range source. So I hacked on my tMod builds to solve this issue.

When the input and output bit depth are the same, no conversion or dithering will be apllied. Otherwise when used with “–fullrange on”, the conversion and dithering will be exactly the same with x264′s original algorithm, which is still not reliable for YUV sources, so it is recommanded to use your own reliable approaches to convert those sources by yourself before passing them to my x264 builds; if used without “–fullrange on”, the conversion will be done by shifting according to the standard, and the dithering algorithm is fixed as well, so that the result should be absolutely right.

astrataro@gmail.com

——————–

Use win32thread and fprofiled:

tMod/tMod-10bit/tMod+MixAQ/tMod+OreAQ/tMod+OreAQ-10bit:
Downloadx264_rev2120+630_tMod.7zMultiUpload-Mirrornmm-hd-Mirror

My other tools list ( including some of the x264 builds ) : MediaFirenmm-hd

Patches:
–—––—–Download(L-Smash not included): patches-rev2120.7znmm-hd-Mirror
00-L-Smash(including qtaac in x86 build);
01-Re-enable –tune “touhou” in fullhelp;
02-Film Grain Optimization;
03-Fade Compensation;
04-Remove stats before renaming;
05-Set level of writing options in SEI;(new!!)
06-Demuxer Thread;
07-Auto VBV Settings;
08-Avi output;
09-Filter: hqdn3d/pad/vflip/yadif;
10-Logger;
11-Encode Time;
12-Force level;
13-Force profile;(new!!)
14-Cosmetic;
15-BugMaster’s new aq-mode 3;
16-Fix bit depth conversion and dithering;
17-Print video info lavf ffms;
18-Enable lto Ofast;
19-AviSynth 16bit hack;
20-Skip bit depth filter;(updated!!)
21-Filter: subtitles;(new!!)
22-1-MixAQ-core(only in tMod+MixAQ version, remove AQDebug)
22-2-OreAQ-core(only in tMod+OreAQ version, remove AQDebug)

Compiler:
mingw-gcc 4.6.2

Libpack info:
libav r31266 git-5695ae4
ffms svn-r587
VSFilter svn-r3698 (MPC-HC)
lame 3.99-3
libvorbis-aotuv_b6.03 (libvorbis-1.3.2)
opencore-amr-0.1.2 r178 git-08d6986
vo-aacenc v0.1.1-7-git-07931e3
vo-amrwbenc v0.1.1-5-git-05114fa
aacplus 2.0.2
faac 1.28
qtsdk 7.3

MixAQ’s diff(AQDebug removed, also included in the patch package above):
MixAQ-core-Taro.diff
OreAQ’s diff(AQDebug removed, also included in the patch package above):
OreAQ-core-Taro.diff

Categories: encode, x264, x264-bin

avs4x264mod with –seek and –frames?

2nd December 2011, Friday Leave a comment

When I solved the high bit depth input issue with original avs4x264 and published the first build, I had never thought that I would keep maintaining this mod as I only modified avs4x264 for my own use, and v0.1 already met all my own demands. So I didn’t test much. Recently I found another modified avs4x264 – Yet Another Avs4X264 Mod ( yaa4xm ). I was impressed by the fact that this piping tool claims not supporting x264’s –seek and –frames. I looked into avs4x264’s codes and comfirmed that –frames was not supported, by both the parameter parser and piping. –seek? Avs4x264 can handle it, although in a weird way.

There is nearly no reason for me to fix these two issues, if they had ever bothered anyone. Setting the start and the end frame for avs input is so easy — just add a Trim in avs. But if avs4x264mod has already finished many things I almost never used before and may never use in the future, why not making it more powerful?

There was two things preventing –frames from working correctly. The first stuff was in the parser that avs4x264 added –frames before user defined parameters without checking if it was already given in commandline, and x264 would get two –frames parameter. Though x264 actually use the later one, it is still not an option to leave the arbitrary problem to x264. The second stuff was that x264 finishes its work as soon as it has encoded the number of frames specified by –frames. But avs4x264 kept delivering until the end of the clip of avs. Then the frames exceeding –frames range could not be handle and avs4x264 threw error. Actually it did not matter too much as the encoding process could be finished without any error. Just leave avs4x264 away. But no one likes even error message, do we?

To solve –frames issue was quite easy and v0.6 of my mod version already support user defined –frames. However when I did it I made a mistake, and now I find out there is a bug of calculating frames delivered to x264 if both –seek and –frames are given in commandline. I prevented the parser from adding –frames if it is already defined, and made avs4x264mod only deliver the number –frames tells. This works fine if –seek is not given. But x264’s approach is to count from the first frame –seek gives until –frames gives. So if a clip has only 500 frames, for –seek 200 –frames 300, x264 will encode from the 201st frame to the 500th frame if get input from avs file directly, but will only encode from the 201st frame to the 300th frame if working with avs4x264mod.

This is still a simple thing to deal with. There are several ways to do. The simplest one is parsing –seek parameter, deleting this parameter when generating new commandline for x264 binary, and delivering from the seek+1 frame to the seek+1+frames frame. Another similar one is just adding a Trim in the end of avs after parsing and deleting –seek option. The third one is that do not touch seek parameter after parsing, but let avs4x264mod deliver from the first frame to the seek+1+frames frame, just like what it did in the past. All these three method shall give the same result in most cases, but wait, in most cases?

Yes, there is still some cases the results are different — with timecodes. x264 takes timecodes and thinks it is for the source file, not the encoded file. Say, if it takes the following timecodes for a 500 frames input file:
# timecode format v1
Assume 29.970030
0,299,23.976024
300,499,29.970030
, and if it gets parameters like –seek 200 –frames 300, then after encoding, the timecodes of output video will be like this:
# timecode format v1
Assume 29.970030
0,99,23.976024
100,299,29.970030
Now if we fix the way of dealing with –seek in the first or the second mothed above, as the first 200 frames will be omitted, the timecodes of output video will turn to be:
# timecode format v1
Assume 29.970030
0,299,23.976024
See, the result is not correct any more. This will not bother one too much as he or she can still use mkvmerge for matroska files and timelineeditor for mp4 files to fix the timecodes ( What? tc2mp4? What is that buggy stuff? -_,- ). However, begined from rev1943, x264 introduced a so-called VFR MB-tree that varies the mb-tree and therefore the quality of every frame to be encoded according to the actual frame rate of it. What if a 59.940fps frame is treated as 23.976 and a 23.976fps frame is treated as 59.970? The result file might be strangely large or the content might be somehow ugly or, at least it affects the efficiency of x264.

That is not all the issues of the first two methods. There are many things annoying avs’s seek, e.g., decoding a m2ts with ffms, or use TIVTC’s one-pass mode ( TDecimate(mode=3) ) to inverse telecine. But avs4x264mod cannot know if the avs script can be safely seeked or not. So it cannot even switch the method of doing this trimming automatically by the seekability. ( Actually avs4x264mod works in a way that it gets a frame from avs server, send it to x264, release the current frame and get the next frame. Every time it gets a frame, it calls the avs_h.func.avs_get_frame function from Avisynth library, requires a frame by its frame number. I am not familiar with how Avisynth acts in this case. Does it output frames linearly, or just seek to one frame, send and release the frame, then seek to another? If it is the first one, avs4x264mod is safe with all the avs scripts, and I personally believed it is how it works, as x264 works in nearly the same way with avs input, and should have the same problem with non-seekable avs input. That’s why I treat this breaking issue to be the second place of these two method. But if not, avs4x264mod should break all the avs that can only be linear seeked, not only with these two methods to deal with –seek. )

Now let us look at the third method again. It seems that nothing is broken in this time. Nothing. However, I called it weird before. If x264 reads input avs file directly, it acts just like the first two methods that I prepared for solving –seek issue. Although it may break non-seekbale avs, it skips all the frames until the frame number –seek indicates. The third method avs4x264mod may use to deal with –seek is somehow different. It still delivers all the frames that x264 might be interested in ( for timecodes parsing ). Now x264 acts as if with other raw piping tools ( excluding avs2pipemod if you change –seek into avs2pipemod’s -trim, in which case it does exactly in the second method ) or lavf demuxer. See both the two situation is that the source is not seekable. However the avs script is seekable, although may not be safe. It reads all the frames from the beginning, does nothing for the first to-be-discard frames except for skipping the same numbers of frame in timcodes, and starts to encode from the seek+1 frame. However, if you send a very heavy avs script to avs4x264mod working in this way, the processing time of the first *garbage* frames is just a waste of life.

Now what can I get as a conclusion? The only thing I know is that all the methods are not good enough for me. Two buggy, one wasteful. Then I got the fourth approach, i.e., a mixture the first and the third method. As x264 already breaks seekable avs with –seek, given that I aimed my avs4x264mod to be a replacement of x264.exe in the commandline, just like what I guessed the original author might do from the simple usage of avs4x264, I can just ignore that problem. Then the only problem is timecodes issue. I can make a detectection to fined if x264 gets a timecodes file or not. Actually it was already done for preventing adding –fps when –tcfile-in being used. If no tcfile is used, use the first method for fast seek, and otherwise use the third method.

This can still be improved more. If timecodes file is used, the only thing x264 cares is that the first *seek* frames being stashed. It is possible, though a little bit complicated, to replace those frames by simple BlankClip, or even easier freezed frame of the first frame after seek. Even without timecodes input we can still on this with only very litter time to be waste for the dummy frames, as long as we do not mention non-seekable avs any more. This fifth method seems to meet most demands we need. At lease I have no interest in adding a timecode parser and output a new trimmed timecodes to deal with timecodes.

And another choise is to add a –seek-mode option, to let user to decide which method to be used. “fast” uses method 1/2 if no timecodes input and 5 with timecodes, which should gives exactly the same result as inputing avs directly to x264 as well as the issue of non-seekable avs, and “safe” always uses 3 for safety of all avs scripts. This is the way I would like to accomplish in the future. Any comments are welcomed on this but for now, I will firstly fix the existing –seek & –frames bug and always use 3 as before in the next git commitment. For now, just keep tracking the git for the first solved revision, or stop using useless –seek and –frames!

Categories: avs4x26x, encode, x264

avs4x264mod v0.6 laa test build ( exceed 2GB memory limit )

1st December 2011, Thursday Leave a comment

laa version:
Still v0.6. Compiled with LARGEADDRESSAWARE flag to allow exceeding 2GB memory limit, should be faster on heavy scripts with SetMemoryMax(2048) or even larger. The memory limit now is 3GB on x86 Windows system when configured correctly, and 4GB on x64 Windows system with no special config required.

Download:
MediaFire: avs4x264mod_v0.6.7z
GitHub: avs4x264mod_v0.6(laa).7z
NMM-HD: avs4x264mod_v0.6(laa).7z

Categories: avs4x26x, encode, x264