DJ-MVP
A music video (MV) is a videotaped performance of a recorded popular song, usually accompanied by dancing and visual images. In this paper, we outline the design of DJ-MVP-a generative music video system, which automatically generates an audio-video mash-up for a given target audio track. The system performs segmentation for the given target song based on the beat detection. Next, according to audio similarity analysis and color heuristic selection methods, we obtain generated video segments. Then, these video segments are truncated to match the length of audio segments and are concatenated as the final music video. An evaluation of our system has shown that users are receptive to this novel presentation of music videos and are interested in future developments.
Figure 1 shows the design of DJ-MVP. For building the corpus, audio tracks and their corresponding videos are segmented based on the beat detection. Given a target song, the system segments it in the same way as above. It then conducts an audio similarity analysis between the target audio segment and segments in a corpus and returns a list of candidate audio segments. This maintains a strong connection between audio and video because they originally came from the same music videos. We take advantage of the color features of video segments and build video sequences that have color contrast or color coherence. We provide heuristic selection methods to enable users to control the number of repetition a particular segment appears so that the system can keep the consistency of the visual content toward the same audio segment and maintain the diversity of video segments. This is explained in the section of “Heuristic Selection Module”. The duration of video segments and the duration of the target audio segment are compared in order to truncate the longer segments. After truncation and concatenation of video segments, the system outputs the generated music video.
Go to Vimeo profile to watch more videos
Papers & Posters
Fan, J., Li, W., Bizzocchi, Jim., Bizzocchi, Justine., Pasquier, P.: DJ-MVP : An Automatic Music Video Producer, November 2016 ACE '16: Proceedings of the Advances in Computer Entertainment Technology Conference (2016).