[FFmpeg-trac] #3143(avcodec:new): H.261 encoding crashes with trellis

[FFmpeg]

#3143: H.261 encoding crashes with trellis
---------------------------------------+-----------------------------------
             Reporter:  maikmerten     |                    Owner:
                 Type:  defect         |                   Status:  new
             Priority:  normal         |                Component:  avcodec
              Version:  git-master     |               Resolution:
             Keywords:  crash SIGSEGV  |               Blocked By:
             Blocking:                 |  Reproduced by developer:  0
Analyzed by developer:  0              |
---------------------------------------+-----------------------------------
Changes (by cehoyos):

 * keywords:  crasher => crash SIGSEGV
 * component:  undetermined => avcodec


Old description:

> Summary of the bug:
> How to reproduce:
> {{{
> % ffmpeg -i /tmp/test.y4m -f h261 -vb 256k -trellis 2 /tmp/test.h261
> ffmpeg version N-58112-g5592d1b Copyright (c) 2000-2013 the FFmpeg
> developers
>   built on Nov 15 2013 18:16:18 with gcc 4.8 (Ubuntu/Linaro
> 4.8.1-10ubuntu8)
> }}}
>
> There's some revived interest in H.261 as possible guaranteed-patent-free
> baseline codec for videocommunication. While ffmpeg's H.261 usually works
> fine, it does not work with advanced techniques such as trellis
> quantization (or most rate/distortion options, for that matter). This is
> about trellis.
>
> It would be awesome if some of the cool advanced encoder features that
> are available for MPEG-1 (e.g., trellis) would also work for H.261. Think
> "pig with rockets attached". If this is not possible the encoder at least
> should not crash.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0,
> block=0x1626d60,
>     n=1, qscale=4, overflow=<optimized out>) at
> libavcodec/mpegvideo_enc.c:3619
> 3619                        int score= distortion +
> length[UNI_AC_ENC_INDEX(run, level)]*lambda;
> (gdb) bt
> #0  0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0,
>     block=0x1626d60, n=1, qscale=4, overflow=<optimized out>)
>     at libavcodec/mpegvideo_enc.c:3619
> #1  0x0000000000842d13 in encode_mb_internal (mb_block_count=6,
>     mb_block_width=8, mb_block_height=8, motion_y=0, motion_x=0,
> s=0x16342a0)
>     at libavcodec/mpegvideo_enc.c:2060
> #2  encode_mb (motion_y=0, motion_x=0, s=0x16342a0)
>     at libavcodec/mpegvideo_enc.c:2168
> #3  encode_thread (c=<optimized out>, arg=<optimized out>)
>     at libavcodec/mpegvideo_enc.c:3042
> #4  0x000000000090d8bf in avcodec_default_execute (c=0x1633b20,
>     func=0x83f8b0 <encode_thread>, arg=<optimized out>, ret=<optimized
> out>,
>     count=1, size=8) at libavcodec/utils.c:1016
> #5  0x000000000083a2c7 in encode_picture (picture_number=0, s=0x16342a0)
>     at libavcodec/mpegvideo_enc.c:3435
> #6  ff_MPV_encode_picture (avctx=0x1633b20, pkt=0x7fffffffd910,
>     pic_arg=<optimized out>, got_packet=0x7fffffffd90c)
>     at libavcodec/mpegvideo_enc.c:1494
> #7  0x000000000090e6a5 in avcodec_encode_video2
> (avctx=avctx at entry=0x1633b20,
>     avpkt=avpkt at entry=0x7fffffffd910, frame=frame at entry=0x1628700,
>     got_packet_ptr=got_packet_ptr at entry=0x7fffffffd90c)
>     at libavcodec/utils.c:1861
> #8  0x000000000046da85 in do_video_out (in_picture=0x1628700,
> ost=0x1633f80,
> ---Type <return> to continue, or q <return> to quit---
>     s=0x1633200) at ffmpeg.c:953
> #9  reap_filters () at ffmpeg.c:1098
> #10 0x000000000045d1df in transcode_step () at ffmpeg.c:3223
> #11 transcode () at ffmpeg.c:3266
> #12 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3444

New description:

 Summary of the bug:
 How to reproduce:
 {{{
 % ffmpeg -i /tmp/test.y4m -f h261 -vb 256k -trellis 2 /tmp/test.h261
 ffmpeg version N-58112-g5592d1b Copyright (c) 2000-2013 the FFmpeg
 developers
   built on Nov 15 2013 18:16:18 with gcc 4.8 (Ubuntu/Linaro
 4.8.1-10ubuntu8)
 }}}

 There's some revived interest in H.261 as possible guaranteed-patent-free
 baseline codec for videocommunication. While ffmpeg's H.261 usually works
 fine, it does not work with advanced techniques such as trellis
 quantization (or most rate/distortion options, for that matter). This is
 about trellis.

 It would be awesome if some of the cool advanced encoder features that are
 available for MPEG-1 (e.g., trellis) would also work for H.261. Think "pig
 with rockets attached". If this is not possible the encoder at least
 should not crash.
 {{{
 Program received signal SIGSEGV, Segmentation fault.
 0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0,
 block=0x1626d60,
     n=1, qscale=4, overflow=<optimized out>) at
 libavcodec/mpegvideo_enc.c:3619
 3619                        int score= distortion +
 length[UNI_AC_ENC_INDEX(run, level)]*lambda;
 (gdb) bt
 #0  0x000000000083536a in dct_quantize_trellis_c (s=0x16342a0,
     block=0x1626d60, n=1, qscale=4, overflow=<optimized out>)
     at libavcodec/mpegvideo_enc.c:3619
 #1  0x0000000000842d13 in encode_mb_internal (mb_block_count=6,
     mb_block_width=8, mb_block_height=8, motion_y=0, motion_x=0,
 s=0x16342a0)
     at libavcodec/mpegvideo_enc.c:2060
 #2  encode_mb (motion_y=0, motion_x=0, s=0x16342a0)
     at libavcodec/mpegvideo_enc.c:2168
 #3  encode_thread (c=<optimized out>, arg=<optimized out>)
     at libavcodec/mpegvideo_enc.c:3042
 #4  0x000000000090d8bf in avcodec_default_execute (c=0x1633b20,
     func=0x83f8b0 <encode_thread>, arg=<optimized out>, ret=<optimized
 out>,
     count=1, size=8) at libavcodec/utils.c:1016
 #5  0x000000000083a2c7 in encode_picture (picture_number=0, s=0x16342a0)
     at libavcodec/mpegvideo_enc.c:3435
 #6  ff_MPV_encode_picture (avctx=0x1633b20, pkt=0x7fffffffd910,
     pic_arg=<optimized out>, got_packet=0x7fffffffd90c)
     at libavcodec/mpegvideo_enc.c:1494
 #7  0x000000000090e6a5 in avcodec_encode_video2
 (avctx=avctx at entry=0x1633b20,
     avpkt=avpkt at entry=0x7fffffffd910, frame=frame at entry=0x1628700,
     got_packet_ptr=got_packet_ptr at entry=0x7fffffffd90c)
     at libavcodec/utils.c:1861
 #8  0x000000000046da85 in do_video_out (in_picture=0x1628700,
 ost=0x1633f80,
 ---Type <return> to continue, or q <return> to quit---
     s=0x1633200) at ffmpeg.c:953
 #9  reap_filters () at ffmpeg.c:1098
 #10 0x000000000045d1df in transcode_step () at ffmpeg.c:3223
 #11 transcode () at ffmpeg.c:3266
 #12 main (argc=<optimized out>, argv=<optimized out>) at ffmpeg.c:3444
 }}}

--

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/3143#comment:2>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker