A little-explained material that is capable of creating singularities (black holes) by some undefined process. It is refined from an ore found rarely in Romulan space using a method only known to the Vulcan Science Institute.
Red matter appears to create singularities of the same size regardless of the amount used, and these singularities have varying effects (some create time warps, others apparently are only destructive in nature). Also, the singularities appear to be unstable, as they collapse on themselves, or are destroyed in some other fashion
For a little more information, see the Star Trek: Countdown comics. It tries its hardest to explain the movie.