[FFmpeg-trac] #1752(avfilter:new): hqdn3d crash (assembly)

[FFmpeg]

#1752: hqdn3d crash (assembly)
-------------------------------------+-------------------------------------
               Reporter:  Cigaes     |                  Owner:
                   Type:  defect     |                 Status:  new
               Priority:  normal     |              Component:  avfilter
                Version:  git-       |               Keywords:  hqdn3d asm
  master                             |  crash segv
             Blocked By:             |               Blocking:
Reproduced by developer:  0          |  Analyzed by developer:  0
-------------------------------------+-------------------------------------
 A particular combination of pixels cause hqdn3d to crash.

 How to reproduce:
 {{{
 $ ./ffmpeg_g -loglevel debug -s 2x4 -pix_fmt yuv420p -i /tmp/t.raw -vf
 hqdn3d -f null -
 ffmpeg version N-44586-gb90210e Copyright (c) 2000-2012 the FFmpeg
 developers
   built on Sep 19 2012 12:24:19 with gcc 4.7 (Debian 4.7.1-7)
   configuration: --enable-shared --disable-static --enable-gpl --enable-
 libx264 --enable-libass --enable-libfreetype --assert-level=1
   libavutil      51. 73.101 / 51. 73.101
   libavcodec     54. 56.100 / 54. 56.100
   libavformat    54. 27.101 / 54. 27.101
   libavdevice    54.  2.100 / 54.  2.100
   libavfilter     3. 16.104 /  3. 16.104
   libswscale      2.  1.101 /  2.  1.101
   libswresample   0. 15.100 /  0. 15.100
   libpostproc    52.  0.100 / 52.  0.100
 [AVIOContext @ 0x1a8caa0] Statistics: 12 bytes read, 0 seeks
 Input #0, image2, from '/tmp/t.raw':
   Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
     Stream #0:0, 1, 1/25: Video: rawvideo (I420 / 0x30323449), yuv420p,
 2x4, 1/25, 25 tbr, 25 tbn, 25 tbc
 [Parsed_hqdn3d_0 @ 0x1a8cd40] ls:4.000000 cs:3.000000 lt:6.000000
 ct:4.500000
 [buffer @ 0x1a8ea00] Setting entry with key 'video_size' to value '2x4'
 [buffer @ 0x1a8ea00] Setting entry with key 'pix_fmt' to value '0'
 [buffer @ 0x1a8ea00] Setting entry with key 'time_base' to value '1/25'
 [buffer @ 0x1a8ea00] Setting entry with key 'pixel_aspect' to value '0/1'
 [buffer @ 0x1a8ea00] Setting entry with key 'sws_param' to value 'flags=2'
 [buffer @ 0x1a8ea00] Setting entry with key 'frame_rate' to value '25/1'
 [graph 0 input from stream 0:0 @ 0x1a8ce40] w:2 h:4 pixfmt:yuv420p tb:1/25
 fr:25/1 sar:0/1 sws_param:flags=2
 Output #0, null, to 'pipe:':
   Metadata:
     encoder         : Lavf54.27.101
     Stream #0:0, 0, 1/90000: Video: rawvideo (I420 / 0x30323449), yuv420p,
 2x4, 1/25, q=2-31, 200 kb/s, 90k tbn, 25 tbc
 Stream mapping:
   Stream #0:0 -> #0:0 (rawvideo -> rawvideo)
 Press [q] to stop, [?] for help
 zsh: segmentation fault
 }}}

 The sample file contains:

 {{{
 0000000: b586 1c00 0000 3c8f 7f7f 7f7f
 }}}

 valgrind says:

 {{{
 ==25957== Invalid read of size 2
 ==25957==    at 0x50B965E: ??? (hqdn3d.asm:103)
 ==25957==    by 0xE5877C7: ???
 ==25957==    by 0x50A2724: end_frame (vf_hqdn3d.c:115)
 ==25957==    by 0x50B1BC0: ff_end_frame (video.c:342)
 ==25957==    by 0x506759A: request_frame (buffersrc.c:379)
 ==25957==    by 0x5067785: av_buffersrc_add_ref (buffersrc.c:152)
 ==25957==    by 0x5067967: av_buffersrc_add_frame (buffersrc.c:91)
 ==25957==    by 0x416BF6: decode_video (ffmpeg.c:1646)
 ==25957==    by 0x4093E8: main (ffmpeg.c:1761)
 ==25957==  Address 0xffffffffee57aee0 is not stack'd, malloc'd or
 (recently) free'd
 }}}

 gdb says:

 {{{
 Program received signal SIGSEGV, Segmentation fault.
 ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
 103     HQDN3D_ROW 8
 (gdb) where
 #0  ff_hqdn3d_row_8_x86.loop2 () at libavfilter/x86/hqdn3d.asm:103
 #1  0x00000000006329c8 in ?? ()
 #2  0x00002aaaaaf41725 in denoise_spatial (temporal=0x645480,
 spatial=0x641420, depth=8, dstride=32, sstride=<optimized out>,
     h=4, w=2, frame_ant=0xffffffff, line_ant=0x635080, dst=<optimized
 out>, src=<optimized out>, hqdn3d=0x632bc0)
     at libavfilter/vf_hqdn3d.c:115
 #3  denoise_depth (depth=8, temporal=0x643480, spatial=<optimized out>,
 dstride=32, sstride=<optimized out>,
     h=<optimized out>, w=<optimized out>, frame_ant_ptr=<optimized out>,
 line_ant=0x635080, dst=<optimized out>,
     src=<optimized out>, hqdn3d=0x632bc0) at libavfilter/vf_hqdn3d.c:153
 #4  end_frame (inlink=<optimized out>) at libavfilter/vf_hqdn3d.c:338

 rax            0x645480 6575232
 rbx            0xffffffff       4294967295
 rcx            0x6329ca 6498762
 rdx            0x635082 6508674
 rsi            0x636581 6514049
 rdi            0x636581 6514049
 rbp            0x1      0x1
 rsp            0x7fffffffc940   0x7fffffffc940
 r8             0x0      0
 r9             0x641420 6558752
 r10            0x7      7
 r11            0xfffffffff0000000       -268435456
 r12            0x1      1
 r13            0x635080 6508672
 r14            0x641420 6558752
 r15            0x645480 6575232
 rip            0x2aaaaaf5865e   0x2aaaaaf5865e
 <ff_hqdn3d_row_8_x86.loop2+52>
 eflags         0x10296  [ PF AF SF IF RF ]
 cs             0x33     51
 ss             0x2b     43
 ds             0x0      0
 es             0x0      0
 fs             0x0      0
 gs             0x0      0
 }}}

 The crash does not happen if assembly is disabled. The arch setting is
 ARCH_X86_64.

 (The crash also happens with a real-world image, I just cropped very
 tightly.)

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1752>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker