Making Video Glitch Art: How to Datamosh, in Plain English
Learn how to make video glitch art by datamoshing with a variety of tools. This is a great introduction to datamoshing for beginners!
What’s datamoshing?
This is datamoshing:
Jim Carrey
Miley Cyrus Datamosh Parody (NSFW video)
Exploding people
http://i.imgur.com/mKMrpie.gifv
In plain English, datamoshing is an umbrella term for an array of cool effects you get by glitching, altering, or otherwise breaking the fundamental structure of video files’ data. We datamosh because we can, and because it looks really cool. And you can, too! In this tutorial, I’ll go over in detail a couple different ways to datamosh a video to achieve different effects.
We’ll be going over three primary types of datamoshing that are currently known:
- File destruction
- I-frame destruction
- P-frame duplication (also sometimes referred to as the “bloom” effect)
Let’s run over the fundamental concepts of all of them before we go too deep.
File destruction, I-Frame destruction, and P-frame duplication
With file destruction, we take any old video file and mess with its data by simply opening it and replacing random bytes with other random bytes. This will make it glitch out in crazy, unpredictable ways and can produce some really cool effects (or break your file - always keep the original copy!). Here’s an example of a gif that was made by destroying the file:
don't look at the moon when the night is dark
We typically do this with a hex editor, a low-level editing tool for looking at the contents of files, but I wrote a tool that can do this in a more automated way which I provide below. Different video file formats glitch in different ways depending on the video codec that was used to compress the video file. I’ll show you a couple different file formats and what they look like when their data is destroyed.
“Flip and Swtich”, /r/glitch_art
To understand I-Frame destruction (and the next section on P-frame duplication), we need to briefly cover how video compression works. Video files layer multiple images in sequence in order to make the appearance of motion. However, storing every single image in a video takes up a lot of space! So these images are compressed as “frames”, with three types - I-frames, P-frames, and B-frames.
I-frames are “inter” frames, also called keyframes, which contain the entirety of the frame’s image data. You can think of them as regular images.
P-frames are predictive frames. They don’t actually contain much image data themselves but represent the abstract motion of pixels within chunks of the frame between frames. Essentially you can think of them as tracking where exactly pixels move.
B-frames are bidirectional predictive frames. They’re basically like P frames, except they’re a little more efficient. We don’t use them for glitch as much and you can pretty much ignore them.
source: https://en.wikipedia.org/wiki/Video_compression_picture_types
(If you want a more technical overview of how I, P, and B frames function, take a look at this Programmer Stack Exchange answer and this post.)
So what happens if you destroy an I-Frame, leaving nothing but P-frames on both sides? Well, you’ve removed the actual important image data that distinguishes the previous frame from the next frame. This means that when a video player plays your I-frame-less video, it’ll force the image from the P frame that was before an I-frame onto the next P-frame. This will result in the image of your previous clip sticking on top of the motion of the next clip, because the video player thinks that it’s just continuously predicting the motion of clip without realizing that the actual image content has changed! This is where that “sticking” effect comes from in datamosh - a video file whose I-frames have been stripped, being misinterpreted by an innocent video player.
P-frame duplication: “Andy explodes”, /r/brokengifs
P-frame duplication is the flip side of I-frame destruction, where you duplicate a P-frame multiple times in the same location to get a trippy “blooming” effect where the image distorts and gets very colorful. You’ll notice that the motion in a P-frame duplication datamosh tends to “follow itself”, i.e. it goes in a given direction and keeps going. This happens because when you take the same P-frame and duplicate it over and over, you’re taking the P-frame’s motion prediction and understanding of the next frame’s pixels and repeating it, causing recursive effects where the P-frames in turn emphasize the effect of the last P-frame. Another thing to note is that P-frame duplication tends to make the image get very colorful. I believe that’s because the P-frame’s color information degrades each time the P-frame repeats, because each individual P-frame only has the exact same color information as the one before it. When the motion prediction is added, the colors get simplified and thus really trippy.
OK! Let’s get to learning how to actually make this stuff happen.
Tools for Datamoshing
You’ll need some stuff for both of these. Here’s what you’ll need:
-
Some raw video files. A lot of datamosh involves putting disparate clips together for interesting effects, but finding and managing all those video files ends up being the most annoying part of the process. For this tutorial I went ahead and cut up some short clips from two of my old school video projects that you can use to practice. Consider these individual clips in the public domain and feel free to use them for whatever.
-
avidemux 2.5.6. Avidemux is a free, open source video editor used for cutting, filtering, and encoding with support for a number of different file types. We’ll be using it to convert between file types, and if you want to try the I-Frame destruction method, destroy those pesky I-Farmes (more on what that means below). Here’s the catch: you must use version 2.5.6, as this older version is just buggy enough to accomplish the effects we need it to. Other versions may or may not work, but we know 2.5.6 to work well.
Downloads for Windows or Linux: http://sourceforge.net/projects/avidemux/files/avidemux/2.5.6/
Mac OS X is a little bit trickier. Check out this short post to learn how to install avidemux on Mac. (I fibbed about the 2.5.6 thing a bit - 2.5.4 seems to work fine on the Mac.)
-
VLC Media Player. VLC is an extremely capable video player that can play almost any type of video file. When it comes to glitching, you’ll generally want to use something like VLC. (NOTE: Sometimes other obscure video players like Windows Media Player or Quicktime Pro can play videos that you’ve totally broken.)
File destruction
As discussed above, file destruction is very simple. You destroy the file, open it, and then see what it looks like. Let’s try it with a hex editor first, then we’ll go over a way to do it in an automated fashion. I’m going to be using my film_zoom.mp4 sample, but feel free to use whatever sample of your own that you like. Note that some formats are more fragile/volatile than others, so if you’re having trouble, use one of my sample files or skip to the “Prepping and Rescuing Your Files” section.
Hex editing
You’ll need a hex editor, which is sort of like a text editor that lets us look at the raw data of very large files. My favorite for Mac is HexFiend and my favorite for PC is HxD. (Can someone recommend a Linux option?) In the screenshot above I’m on a Mac, so I’m using HexFiend.
Once you have your hex editor of choice, open up your video file and look at its guts. What see here is hexadecimal, the base 16 representation of the data in our file. Great! We’re ready to destroy it! However, there are a couple important general rules to follow when doing this:
- Don't ruin the header of the file. A header is a chunk of data that’s at the front of a file (its “head”) that tells programs what type of file that file is. The size of a header varies from file type to file type - for instance, a BMP has a header that’s 54 bytes large, and a video file I peaked at has a 10.7 kb large header. If you ruin the header of your file, it will not open because your video player won’t know how to interpret it. Don’t touch it!
- Don’t change the total byte size of your file. Most video players / programs will interpret this as a change in the dimensions or content of a single frame, call bogus, and then refuse to open your file. When destroying a file, don’t delete or add - replace.
- Don’t destroy your file too much! There’s a critical mass point where your file becomes so broken that the effects are just soupy and not interesting. I’ve found that this happens when I break more than 5% of the total number of bytes in a file.
Just go in there, change some values, do whatever. I like to cut out a large chunk of the data and paste it back in somewhere else, or just copy and then paste over some other data. Once you’re happy with your results, save it as a new file (never overwrite your original file!) and then open it up in a video player (VLC).
Here’s what I got after turning my results into a .gif and removing some unnecessary frames:
My original result had a lot of freezing and stuttering that didn’t look very good, so don’t fret if yours is the same! The upside to file destruction is that it can have unpredictable and good results - the downside is that it can have unpredictable and bad results. If you saw the above image and wondered how you can turn yours into a .gif, don’t fret - I’ll go over that in this tutorial, too.
Automated destruction with byebyte
So, cool! You did it! Or maybe you didn’t, found the process annoying, and wish there were a quicker way. So did I! I wrote a script to solve that problem. If you’re familiar with the command line or not afraid to learn it, you can use byebyte, which automatically replaces random bytes in the file with random values. byebyte ensures a random distribution of destroyed bytes, so you can get some pretty effects, and if you don’t like the results, just run the command again. Follow the instructions in the documentation to give it a spin.
Here’s what I got after running film_zoom.mp4 through byebyte, replacing 3000 bytes.
Here’s another runthrough byebyte, while replacing 1000 bytes:
Let’s give I-frame destruction a shot now.
I-frame Destruction
As described above, I-frame destruction involves removing important frames that contain full pixel information from a video file, leaving behind only frames that predict motion (and some pretty interesting effects). There’s two easy ways to do this that I know of: with avidemux, and with an automated tool called aviglitch.
avidemux
Fair warning with the older version of avidemux - it’s really glitchy. It crashes frequently and you can’t undo edits, so always store a backup of your files. Such is the life of a glitch artist.
Choose what video file you want to edit. I recommend choosing a video file that has lots of cuts in it. I’m using fly_quick-cuts.mp4. Open up avidemux and then open your video file in it. You can do this either by choosing File > Open or just drag your file onto the application window. It will probably ask you if you want to use another mode - hit No.
It may also tell you that your index is up to date and that you should not update it. Also hit No.
Basically, any time avidemux tries to help you, tell it no. You should now have your video file in avidemux.
Next, we’re going to drastically reduce the number of iframes in this video file and save it to a format that produces glitches pretty reliably. Click the dropdown on the left under the Video section that says “Copy”, and select MPEG-4 ASV (Xvid). This puts avidemux into a mode where if you press Save, it’ll export the video file into this format.
Now, hit the Configure button right below that. An “Xvid Configuration” window will pop up. Select the “Frame” category, then in the “Maximum I-frame Interval” put some really large number. This will make it so that when you save your video file, the resulting file will have significantly fewer I-frames than it ordinarily would (typically, I’ve found that every 24th frame tends to be an I-frame if you don’t do this).
Hit OK. Choose “AVI” from the dropdown underneath the Format section below the Video and Audio sections, then save your file to a new filename. I usually append _max-interval.avi to the filename.
Once that’s done, close your current video (File > Close) and then open up the file you just saved. As before, if avidemux tries to help you, tell it no thank you.
See the little tools at the bottom here?
Try and “scrub” your video: click on the slider widget and drag it back and forth. You’ll notice that it skips around. That’s because avidemux will only let you scrub directly to I-frames, so for every time the slider moves, that’s an I-frame. An I-frame will pretty much always be placed anywhere there’s a cut in the video from one image to a next. We’re going to destroy those I-frames.
Use the double left and right arrows to jump to the first I-frame you can find after frame 0 (the first frame is always an I-frame but we don’t want to destroy our first frame). Hit the “A” button. You’ll see a blue line highlight from where your first I-frame is to the rest of the video. This indicates your selection. We want to select only the I-frame, so move forward one frame by pressing the right arrow key or the single right arrow button, then hit “B”. This will set the blue highlight to now only focus on your I-frame. Hitting the A and B are respectively setting the “in” and “out” points of your selection.
Then, press delete. The frame should disappear. Repeat this process until you’ve deleted all the I-frames. Some shortcuts to make this faster: up and down arrow keys go to each I-frame, left and right arrow keys go back and forward one frame, and left and right bracket ([ and ]) set the A/in and B/out points, respectively. The delete key, of course, deletes. You’ll know you’ve deleted all of the I-frames when you can no longer scrub the video and trying to jump to the next I-frame does nothing.
Done? Great! Change your Video setting to “Copy” and then save the file under a new name. It might ask you if you want to enable smart copy - just say no.
Open that file up in VLC and check out your results!
Here’s what I got by deleting all my I-frames from fly_quick-cuts.mp4:
Combining clips in avidemux
The I-frame destruction we did above is fun but also fairly limited. A common datamoshing technique is to turn one thing into another by composing shots next to one another. You probably want to do this, and doing so is easy: export each of your clips with the max interval settings, and then drag them one at a time into avidemux in the order you want them to be in. When you add another file to avidemux it automatically appends it to the current video clip you’re working with. Then, delete the iframes between your clips and save your video. Here’s what I got by combining two clips:
Give it a shot! Next, we’re going to learn how to remove I-frames automatically using some command line tools.
Automated I-frame destruction with node-aviglitch
Depending on the clip you chose, you might have ended up with a lot of I-frames no matter how hard you tried. Removing them one-by-one can be a laborious process, but if you’re comfortable with the command line and have an .AVI file to play with, you can remove all of them at once!
If you have Node.js, you can install node-aviglitch and then simply run the datamosh script like so: datamosh /path/to/file.avi -o /path/to/output.avi
This will automatically remove the I-frames from an .AVI file, and output a new one, which you can use on your own or throw it back into avidemux to combine it with other aviglitched videos.
P-frame Duplication
avidemux
After having done I-frame duplication, P-frame duplication is relatively easy to do in avidemux. Just select a single P-frame the same way you select an I-frame (setting the in point at the start of the frame and the out point at the end of the frame with the A/B buttons). Then, copy the frame, and paste it repeatedly. NOTE that avidemux will choke once you paste too many times, especially if you do it quickly, so you should save a copy of your progress every now and then. I’ve found it tends to break around 20 - 30 sequential pastes, but behaves fairly well if you wait about half a second between pasting.
Here’s a result I got out of using P-frame duplication:
Automated P-frame duplication with AviGlitch
If you have basic programming skills and a knowledge of Ruby, you can use the original AviGlitch that node-aviglitch is based off of to achieve some really interesting visual results without having to fiddle around with glitchy avidemux. Check out the site for more details.
Rescuing Your Files
So you messed up your files real good, and you love how they look - but you don’t know how to get them on YouTube properly, or in GIF format? This is where some more tools come into play!
”Bake” your datamosh to upload it online
Datamoshing involves breaking files. This means that inherently, your file is in an unstable state and may be read differently by different programs that use it, especially services like YouTube, Vimeo, and Giphy. What you want to upload is a video of your datamosh that treats all the broken frames as if that’s how they’re supposed to be. I call this “baking the mosh”, which I totally just made up. We’re basically gonna resave your video file.
There’s a couple ways I like to do this:
- Export your video from avidemux in MP4 format, which is a commonly accepted format for many video services. The only issue I’ve found is that avidemux will export an insanely jerky video in MP4 format if you don’t turn off the motion estimation setting for some reason. Use the settings you see in this screenshot:
Specifically, that’s the Video MPEG-4 ASP (Xvid) Video option, the MP4 Format, and in the Xvid Configuration, Motion Estimation and Chroma Motion Estimation unchecked. This will export a video that is nothing -but- I-frames, so it ends up being much larger than your datamoshed video, but it “bakes your mosh” into the new video that you can then recompress and upload to YouTube without getting any unintended artifacts or breaking errors.
- VLC can actually convert AVI files into MP4 (.m4v) files, which is a pretty good all-around video format for uploading to video services. Check out the “Convert / Stream…” option in VLC’s menu.
- If you have access to Adobe products, you can use Adobe Media Encoder to convert almost any video format into any other video format, and it tends to handle the glitched AVI files that avidemux / aviglitch export well.
Turning them into gifs
The quickest way that I know to turn your datamosh into a GIF is to upload your video to Giphy, which does it for you: http://giphy.com/upload
If you have Photoshop, you can also import them directly into Photoshop with the Import > Video Frames to Layers option. This works best with smaller video clips (less than 200 frames) because Photoshop renders them all as individual layers, which can be very taxing on your computer. From there, though, you can adjust the speed of individual frames, remove extraneous frames, and export the whole thing using the Save as Web option to be optimized for different sites’ filesize requirements (for instance, Tumblr demands that GIFs uploaded to it be 2MB or less).
If you have Adobe Media Encoder or Adobe Premiere Pro, both of these programs can also export videos directly to GIF.
You can upload your GIFs to a variety of places: Imgur, Giphy, and Tumblr are all pretty good options.
What’s next?
If you followed along, congratulations! You’ve been reading this post on the Glitchet Forum, a fledgling community dedicated to futuristic news and all kinds of cool art made with technology, especially glitch art. I’d love to see anything you moshed! If you're interested in glitch art in general, you should check out the Glitchet Art Resources page which is positively overflowing with more tutorials, resources, and tools for making glitch art.
Glitchet also has a weekly technology and digital art newsletter, a Facebook page and a pretty kickass Tumblr.
If you have any questions, tips, or advice, feel free to post them here!
Sources
Thanks to all of these great tutorials for having the information I needed to put this together.
