Mozilla Foundation Security Advisory 2009-67
Title: Integer overflow, crash in libtheora video library
Announced: December 15, 2009
Reporter: Dan Kaminsky, David Keeler
Products: Firefox 3.5, SeaMonkey 2.0, Thunderbird 3.0
Fixed in: Firefox 3.5.6
Security researcher Dan Kaminsky reported an integer overflow in the Theora video library. A video's dimensions were being multiplied together and used in particular memory allocations. When the video dimensions were sufficiently large, the multiplication could overflow a 32-bit integer resulting in too small a memory buffer being allocated for the video. An attacker could use a specially crafted video to write data past the bounds of this buffer, causing a crash and potentially running arbitrary code on a victim's computer.
Mozilla intern David Keeler also independently reported this issue as well as an additional crash which was determined to be a denial-of-service.
Video capabilities were added to the Mozilla browser engine in Firefox 3.5, SeaMonkey 2.0, and Thunderbird 3.0; prior releases of these products were not affected.
These bugs were fixed upstream in Theora version 1.1 ("Thusnelda") but the older version used in Firefox 3.5 needed this patch.