har0ke/FFmpeg
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

avcodec/mv30: Fix several integer overflows in idct_1d()

Browse Source 
Fixes: signed integer overflow: -1846510390 + -361755993 cannot be represented in type 'int'
Fixes: 23941/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-5654696631730176

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Michael Niedermayer 2020-07-25 23:35:03 +02:00
parent 165d6b876b
commit ddf2ba5497
1 changed files with 17 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
libavcodec/mv30.c
View File

@ -104,23 +104,23 @@ static void get_qtable(int16_t *table, int quant, const uint8_t *quant_tab)
static inline void idct_1d(int *blk, int step)
{
const int t0 = blk[0 * step] + blk[4 * step];
const int t1 = blk[0 * step] - blk[4 * step];
const int t2 = blk[2 * step] + blk[6 * step];
const int t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2;
const int t4 = t0 + t2;
const int t5 = t0 - t2;
const int t6 = t1 + t3;
const int t7 = t1 - t3;
const int t8 = blk[5 * step] + blk[3 * step];
const int t9 = blk[5 * step] - blk[3 * step];
const int tA = blk[1 * step] + blk[7 * step];
const int tB = blk[1 * step] - blk[7 * step];
const int tC = t8 + tA;
const int tD = (int)((tB + t9) * 473U) >> 8;
const int tE = (((int)(t9 * -669U) >> 8) - tC) + tD;
const int tF = ((int)((tA - t8) * 362U) >> 8) - tE;
const int t10 = (((int)(tB * 277U) >> 8) - tD) + tF;
const unsigned t0 = blk[0 * step] + blk[4 * step];
const unsigned t1 = blk[0 * step] - blk[4 * step];
const unsigned t2 = blk[2 * step] + blk[6 * step];
const unsigned t3 = ((int)((blk[2 * step] - blk[6 * step]) * 362U) >> 8) - t2;
const unsigned t4 = t0 + t2;
const unsigned t5 = t0 - t2;
const unsigned t6 = t1 + t3;
const unsigned t7 = t1 - t3;
const unsigned t8 = blk[5 * step] + blk[3 * step];
const unsigned t9 = blk[5 * step] - blk[3 * step];
const unsigned tA = blk[1 * step] + blk[7 * step];
const unsigned tB = blk[1 * step] - blk[7 * step];
const unsigned tC = t8 + tA;
const unsigned tD = (int)((tB + t9) * 473U) >> 8;
const unsigned tE = (((int)(t9 * -669U) >> 8) - tC) + tD;
const unsigned tF = ((int)((tA - t8) * 362U) >> 8) - tE;
const unsigned t10 = (((int)(tB * 277U) >> 8) - tD) + tF;
blk[0 * step] = t4 + tC;
blk[1 * step] = t6 + tE;