[FFmpeg-trac] #2443(avfilter:new): Segmentation fault on deshake filter

[FFmpeg]

#2443: Segmentation fault on deshake filter
-------------------------------------+-------------------------------------
             Reporter:  jbvsmo       |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  avfilter     |                  Version:  git-
             Keywords:  deshake,     |  master
  segfault, filter                   |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 When you change the values of 'rx' or 'ry' in the deshake filter,
 ffmpeg/ffplay will segfault

 {{{
 $ ffplay -i Sam.flv -vf deshake="rx=20"

 ffplay version git-2013-03-11-73b40e2 Copyright (c) 2003-2013 the FFmpeg
 developers
   built on Mar 31 2013 15:19:05 with gcc 4.7 (Ubuntu/Linaro
 4.7.2-2ubuntu1)
   configuration: --enable-gpl --enable-libass --enable-libfaac --enable-
 libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-
 libopencore-amrwb --enable-libspeex --enable-librtmp --enable-libtheora
 --enable-libvorbis --enable-libvpx --enable-x11grab --enable-libx264
 --enable-nonfree --enable-version3 --enable-libfreetype
   libavutil      52. 19.100 / 52. 19.100
   libavcodec     54. 92.100 / 54. 92.100
   libavformat    54. 63.104 / 54. 63.104
   libavdevice    54.  3.103 / 54.  3.103
   libavfilter     3. 44.101 /  3. 44.101
   libswscale      2.  2.100 /  2.  2.100
   libswresample   0. 17.102 /  0. 17.102
   libpostproc    52.  2.100 / 52.  2.100
 Input #0, flv, from 'Sam.flv':=    0KB vq=    0KB sq=    0B f=0/0
   Metadata:
     starttime       : 0
     totalduration   : 21
     totaldatarate   : 1244
     bytelength      : 3322214
     canseekontime   : true
     sourcedata      : BC9111F49HH1365228066618561
     purl            :
     pmsg            :
   Duration: 00:00:20.80, start: 0.000000, bitrate: 1277 kb/s
     Stream #0:0: Video: h264 (Main), yuv420p, 720x480, 1163 kb/s, 30 tbr,
 1k tbn, 60 tbc
     Stream #0:1: Audio: aac, 44100 Hz, stereo, fltp, 132 kb/s
 Segmentation fault (core dumped)
 }}}

 Using ffplay_g, sometimes it will print some unrelated messages (probably
 invalid accesses made by the filter on other parts of the program), but
 most of the time it is just segfault. Some of the messages:

 {{{
 X Error of failed request:  BadGC (invalid GC parameter)
   Major opcode of failed request:  60 (X_FreeGC)
   Resource id in failed request:  0x6000000
   Serial number of failed request:  82
   Current serial number in output stream:  82
 }}}

 {{{
 [xcb] Unknown sequence number while processing queue
 [xcb] Most likely this is a multi-threaded client and XInitThreads has not
 been called
 [xcb] Aborting, sorry about that.
 ffplay_g: ../../src/xcb_io.c:273: poll_for_event: Assertion
 `!xcb_xlib_threads_sequence_lost' failed.
 Aborted (core dumped)
 }}}

 {{{
 XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server "�
 �"
       after 82 requests (82 known processed) with 0 events remaining.
 }}}


 ----

 Using the default value 16 (or 32 or 48) for "rx" or "ry", the filter
 seems to work fine. With any other value, it will segfault, but sometimes
 will produce 1 or 2 frames.

 With value 64, it will segfault faster -- without producing any frames.
 Looking at the source, there's an off-by-one writing in some 128x128 array
 (diff attached). I didn't dig very far for the the other segfault problem
 (possibly something in the "find_motion" function).

 Also attatched the valgrind output.

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/2443>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker