[FFmpeg-trac] #6626(undetermined:new): ff_vdpau_common_init writes past the end of av_alloc_vdpaucontext memory

[FFmpeg]

#6626: ff_vdpau_common_init writes past the end of av_alloc_vdpaucontext memory
-------------------------------------+-------------------------------------
             Reporter:  aaronp24     |                    Owner:
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:
              Version:  unspecified  |  undetermined
             Keywords:               |               Resolution:
             Blocking:               |               Blocked By:
Analyzed by developer:  0            |  Reproduced by developer:  0
-------------------------------------+-------------------------------------

Comment (by peterbennett):

 MythTV is getting a segmentation fault due to memory corruption caused by
 this:

 AVCodecContext.hwaccel_context is supposed to point to an AVVDPAUContext
 as per the documentation of AVVDPAUContext. The user application allocates
 this using av_vdpau_alloc_context according to that documentation.
 However - in FFmpeg/libavcodec/vdpau.c, in ff_vdpau_common_init it assumes
 that AVCodecContext.hwaccel_context contains a pointer to VDPAUHWContext.
 VDPAUHWContext is a structure than contains an AVVDPAUContext plus other
 stuff and is therefore longer than AVVDPAUContext. ff_vdpau_common_init
 then proceeds to set the field VDPAUHWContext::reset to zero. This field
 is beyond the allocated data of AVVDPAUContext and therefore overwrites
 other storage in the caller, causing a segmentation fault. Note that
 av_vdpau_alloc_context only allocates enough memory for AVVDPAUContext and
 therefore not enough for VDPAUHWContext.

 My workaround in MythTV - use structure VDPAUHWContext to determine the
 size of memory to allocate and do not use av_vdpau_alloc_context.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6626#comment:4>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker