jasper: memory allocation failure in jas_malloc (jas_malloc.c)

Description:
jasper is an open-source initiative to provide a free software-based reference implementation of the codec specified in the JPEG-2000 Part-1 standard.

Another round of fuzzing on an updated version (1.900.5) revealed a memory allocation failure.

The complete ASan output:

# imginfo -f $FILE
THE BMP FORMAT IS NOT FULLY SUPPORTED!                                                                                                                                                                                                                                         
THAT IS, THE JASPER SOFTWARE CANNOT DECODE ALL TYPES OF BMP DATA.                                                                                                                                                                                                              
IF YOU HAVE ANY PROBLEMS, PLEASE TRY CONVERTING YOUR IMAGE DATA                                                                                                                                                                                                                
TO THE PNM FORMAT, AND USING THIS FORMAT INSTEAD.                                                                                                                                                                                                                              
==18943==ERROR: AddressSanitizer failed to allocate 0x1000002000 (68719484928) bytes of LargeMmapAllocator (error code: 12)                                                                                                                                                    
==18943==Process memory map follows:                                                                                                                                                                                                                                           
        0x000000400000-0x000000520000   /usr/bin/imginfo                                                                                                                                                                                                                       
        0x00000071f000-0x000000720000   /usr/bin/imginfo                                                                                                                                                                                                                       
        0x000000720000-0x000000724000   /usr/bin/imginfo                                                                                                                                                                                                                       
        0x000000724000-0x0000013a6000                                                                                                                                                                                                                                          
        0x00007fff7000-0x00008fff7000                                                                                                                                                                                                                                          
        0x00008fff7000-0x02008fff7000                                                                                                                                                                                                                                          
        0x02008fff7000-0x10007fff8000                                                                                                                                                                                                                                          
        0x600000000000-0x602000000000                                                                                                                                                                                                                                          
        0x602000000000-0x602000010000                                                                                                                                                                                                                                          
        0x602000010000-0x603000000000                                                                                                                                                                                                                                          
        0x603000000000-0x603000010000                                                                                                                                                                                                                                          
        0x603000010000-0x604000000000                                                                                                                                                                                                                                          
        0x604000000000-0x604000010000                                                                                                                                                                                                                                          
        0x604000010000-0x606000000000                                                                                                                                                                                                                                          
        0x606000000000-0x606000010000                                                                                                                                                                                                                                          
        0x606000010000-0x60b000000000                                                                                                                                                                                                                                          
        0x60b000000000-0x60b000010000                                                                                                                                                                                                                                          
        0x60b000010000-0x619000000000                                                                                                                                                                                                                                          
        0x619000000000-0x619000020000                                                                                                                                                                                                                                          
        0x619000020000-0x625000000000                                                                                                                                                                                                                                          
        0x625000000000-0x625000020000                                                                                                                                                                                                                                          
        0x625000020000-0x640000000000                                                                                                                                                                                                                                          
        0x640000000000-0x640000003000                                                                                                                                                                                                                                          
        0x7f4f00738000-0x7f4f03593000                                                                                                                                                                                                                                          
        0x7f4f03593000-0x7f4f035fc000   /usr/lib64/libjpeg.so.62.2.0                                                                                                                                                                                                           
        0x7f4f035fc000-0x7f4f037fb000   /usr/lib64/libjpeg.so.62.2.0                                                                                                                                                                                                           
        0x7f4f037fb000-0x7f4f037fc000   /usr/lib64/libjpeg.so.62.2.0                                                                                                                                                                                                           
        0x7f4f037fc000-0x7f4f037fd000   /usr/lib64/libjpeg.so.62.2.0                                                                                                                                                                                                           
        0x7f4f037fd000-0x7f4f03990000   /lib64/libc-2.22.so                                                                                                                                                                                                                    
        0x7f4f03990000-0x7f4f03b90000   /lib64/libc-2.22.so                                                                                                                                                                                                                    
        0x7f4f03b90000-0x7f4f03b94000   /lib64/libc-2.22.so                                                                                                                                                                                                                    
        0x7f4f03b94000-0x7f4f03b96000   /lib64/libc-2.22.so                                                                                                                                                                                                                    
        0x7f4f03b96000-0x7f4f03b9a000                                                                                                                                                                                                                                          
        0x7f4f03b9a000-0x7f4f03bb0000   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1                                                                                                                                                                                 
        0x7f4f03bb0000-0x7f4f03daf000   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1                                                                                                                                                                                 
        0x7f4f03daf000-0x7f4f03db0000   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1                                                                                                                                                                                 
        0x7f4f03db0000-0x7f4f03db1000   /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1                                                                                                                                                                                 
        0x7f4f03db1000-0x7f4f03db3000   /lib64/libdl-2.22.so
        0x7f4f03db3000-0x7f4f03fb3000   /lib64/libdl-2.22.so
        0x7f4f03fb3000-0x7f4f03fb4000   /lib64/libdl-2.22.so
        0x7f4f03fb4000-0x7f4f03fb5000   /lib64/libdl-2.22.so
        0x7f4f03fb5000-0x7f4f03fbb000   /lib64/librt-2.22.so
        0x7f4f03fbb000-0x7f4f041bb000   /lib64/librt-2.22.so
        0x7f4f041bb000-0x7f4f041bc000   /lib64/librt-2.22.so
        0x7f4f041bc000-0x7f4f041bd000   /lib64/librt-2.22.so
        0x7f4f041bd000-0x7f4f041d4000   /lib64/libpthread-2.22.so
        0x7f4f041d4000-0x7f4f043d3000   /lib64/libpthread-2.22.so
        0x7f4f043d3000-0x7f4f043d4000   /lib64/libpthread-2.22.so
        0x7f4f043d4000-0x7f4f043d5000   /lib64/libpthread-2.22.so
        0x7f4f043d5000-0x7f4f043d9000
        0x7f4f043d9000-0x7f4f044d6000   /lib64/libm-2.22.so
        0x7f4f044d6000-0x7f4f046d5000   /lib64/libm-2.22.so
        0x7f4f046d5000-0x7f4f046d6000   /lib64/libm-2.22.so
        0x7f4f046d6000-0x7f4f046d7000   /lib64/libm-2.22.so
        0x7f4f046d7000-0x7f4f04891000   /usr/lib64/libjasper.so.1.0.0
        0x7f4f04891000-0x7f4f04a90000   /usr/lib64/libjasper.so.1.0.0
        0x7f4f04a90000-0x7f4f04a94000   /usr/lib64/libjasper.so.1.0.0
        0x7f4f04a94000-0x7f4f04aa3000   /usr/lib64/libjasper.so.1.0.0
        0x7f4f04aa3000-0x7f4f04aac000
        0x7f4f04aac000-0x7f4f04ace000   /lib64/ld-2.22.so
        0x7f4f04c67000-0x7f4f04cc2000
        0x7f4f04cc2000-0x7f4f04ccd000
        0x7f4f04ccd000-0x7f4f04cce000   /lib64/ld-2.22.so
        0x7f4f04cce000-0x7f4f04ccf000   /lib64/ld-2.22.so
        0x7f4f04ccf000-0x7f4f04cd0000
        0x7ffeaeaca000-0x7ffeaeaeb000   [stack]
        0x7ffeaeb8a000-0x7ffeaeb8c000   [vvar]
        0x7ffeaeb8c000-0x7ffeaeb8e000   [vdso]
        0xffffffffff600000-0xffffffffff601000   [vsyscall]
==18943==End of process memory map.
==18943==AddressSanitizer CHECK failed: /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:183 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x4c9ccd in AsanCheckFailed /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_rtl.cc:67
    #1 0x4d0803 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159
    #2 0x4d09f1 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char const*, char const*, int, bool) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:183
    #3 0x4d9a2a in __sanitizer::MmapOrDie(unsigned long, char const*, bool) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc:122
    #4 0x421dbf in __sanitizer::LargeMmapAllocator::Allocate(__sanitizer::AllocatorStats*, unsigned long, unsigned long) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1033
    #5 0x421dbf in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback> >, __sanitizer::LargeMmapAllocator >::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback> >*, unsigned long, unsigned long, bool, bool) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1302
    #6 0x421dbf in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:368
    #7 0x421dbf in __asan::asan_malloc(unsigned long, __sanitizer::BufferedStackTrace*) /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:718
    #8 0x4c0391 in malloc /var/tmp/portage/sys-devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:53
    #9 0x7f4f0474e170 in jas_malloc /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/libjasper/base/jas_malloc.c:117:9
    #10 0x7f4f0474e170 in jas_alloc2 /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/libjasper/base/jas_malloc.c:141
    #11 0x7f4f04764b4f in bmp_getinfo /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/libjasper/bmp/bmp_dec.c:297:25
    #12 0x7f4f04764b4f in bmp_decode /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/libjasper/bmp/bmp_dec.c:132
    #13 0x7f4f0470ef39 in jas_image_decode /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/libjasper/base/jas_image.c:380:16
    #14 0x4f1686 in main /tmp/portage/media-libs/jasper-1.900.5/work/jasper-1.900.5/src/appl/imginfo.c:188:16
    #15 0x7f4f0381d61f in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.22-r4/work/glibc-2.22/csu/libc-start.c:289
    #16 0x418e68 in _init (/usr/bin/imginfo+0x418e68)

Affected version:
1.900.5

Fixed version:
1.900.11

Commit fix:
https://github.com/mdadams/jasper/commit/65536647d380571d1a9a6c91fa03775fb5bbd256

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2016-8886

Timeline:
2016-10-17: bug discovered
2016-10-17: bug reported to upstream
2016-10-18: blog post about the issue
2016-10-22: upstream released a patch and 1.900.11
2016-10-23: CVE assigned

Note:
This bug was found with American Fuzzy Lop.

Permalink:

jasper: memory allocation failure in jas_malloc (jas_malloc.c)

This entry was posted in advisories, security. Bookmark the permalink.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.