[FFmpeg-trac] #588(avcodec:open): FFmpeg crashes when transcoding a wmv video on windows 7 (64 bit) machine

[FFmpeg]

#588: FFmpeg crashes when transcoding a wmv video on windows 7 (64 bit) machine
-------------------------------------+-----------------------------------
             Reporter:  eiljoe       |                    Owner:  michael
                 Type:  defect       |                   Status:  open
             Priority:  important    |                Component:  avcodec
              Version:  unspecified  |               Resolution:
             Keywords:  win64        |               Blocked By:
             Blocking:               |  Reproduced by developer:  1
Analyzed by developer:  0            |
-------------------------------------+-----------------------------------

Comment (by DonMoir):

 I've been able to determine it's not an alignment issue.

 Looks like a bad count or similiar and results in access violation.

 All I have is the raw assembly starting from avcodec_decode_video2 and
 can't tell you much more than this about it at the moment.

 Registers at time of crash:

 EAX = 0536F96C EBX = 00000053 ECX = 00000008 EDX = 0BC7116D ESI = 0000000A
 EDI = 000000A6 EIP = 0A3D8B84 ESP = 0536F8F0 EBP = 00000053 EFL = 00000297

 EDX is bad in the 10th loop of the following: ESI contains the count

 0A3D8A20 55                push        ebp
 0A3D8A21 57                push        edi
 0A3D8A22 56                push        esi
 0A3D8A23 53                push        ebx
 0A3D8A24 8B 5C 24 18       mov         ebx,dword ptr [esp+18h]
 0A3D8A28 85 DB             test        ebx,ebx
 0A3D8A2A 0F 8E 5A 01 00 00 jle         0A3D8B8A
 0A3D8A30 31 F6             xor         esi,esi

 LOOP:

 0A3D8A32 0F B6 3A          movzx       edi,byte ptr [edx] ; crash right
 here
 0A3D8A35 0F B6 5A 01       movzx       ebx,byte ptr [edx+1]
 0A3D8A39 0F B6 6A FF       movzx       ebp,byte ptr [edx-1]
 0A3D8A3D 01 FB             add         ebx,edi
 0A3D8A3F 0F B6 7A 02       movzx       edi,byte ptr [edx+2]
 0A3D8A43 8D 1C DB          lea         ebx,[ebx+ebx*8]
 0A3D8A46 01 EF             add         edi,ebp
 0A3D8A48 29 FB             sub         ebx,edi
 ...................
 ...................
 ...................
 0A3D8B4E 88 58 06          mov         byte ptr [eax+6],bl
 0A3D8B51 0F B6 7A 07       movzx       edi,byte ptr [edx+7]
 0A3D8B55 0F B6 5A 08       movzx       ebx,byte ptr [edx+8]
 0A3D8B59 0F B6 6A 06       movzx       ebp,byte ptr [edx+6]
 0A3D8B5D 01 FB             add         ebx,edi
 0A3D8B5F 0F B6 7A 09       movzx       edi,byte ptr [edx+9]
 0A3D8B63 8D 1C DB          lea         ebx,[ebx+ebx*8]
 0A3D8B66 03 54 24 14       add         edx,dword ptr [esp+14h]
 0A3D8B6A 01 EF             add         edi,ebp
 0A3D8B6C 29 FB             sub         ebx,edi
 0A3D8B6E 83 C3 08          add         ebx,8
 0A3D8B71 C1 FB 04          sar         ebx,4
 0A3D8B74 0F B6 9B 80 13 02 0B movzx    ebx,byte ptr [ebx+0B021380h]
 0A3D8B7B 88 58 07          mov         byte ptr [eax+7],bl
 0A3D8B7E 01 C8             add         eax,ecx
 0A3D8B80 3B 74 24 18       cmp         esi,dword ptr [esp+18h]
 0A3D8B84 0F 85 A8 FE FF FF jne         0A3D8A32
 0A3D8B8A 5B                pop         ebx
 .....................
 .....................

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/588#comment:4>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker