Converting "unsupported" Video files for Apple TV 2

Let's explore the least painful way to convert incompatible files like mkv and avi to Apple TV compatible mp4 videos. Also, the solution I'm using is a Mac only application, so if you're on another platform, I'm afraid you'll need to keep searching.

Hello again and welcome back to my nearly abandoned blog!  A couple posts per year ain't bad, right?

For my last post of the year, I wanted to share some of what I learned about beating various video formats (.mkv, .avi, etc) into something that both imported to iTunes and would play on the Apple TV 2. The main thing we want to do is avoid re-encoding the video if it's already stored in the original file with the correct codec.

I want to try and break this post up into sections to make it a little easier to digest since I tend to be a bit verbose sometimes.  I'll break this up into sections for an overview of my experience, technical details on codecs, transcoding and conversion tools and finally audio formats (which is the main problem I had with conversion).  If you don't really want the back story or the details on codecs and tools, jump to the summary section at the end of the post that explains the process.

My Foray into Apple TV
This Christmas, I found that Santa left me a shiny new Apple TV under the tree.  Interestingly, I found it does stuff that my TiVos already also do, but the Apple TV is much more elegant, and streaming content located on my Mac super fast and much more user friendly.  That is to say once I was able to actually get content loaded into iTunes and playable on my new device.

Since I have been using PyTivo  for ages to get various video copied over to the TiVo boxes in the house, I've amassed a fair collection of content in a number of different formats.  Of course all of this content is perfectly legitimate, I would never condone any sort of piracy or circumvention of the DMCA, that is just wrong, mmmkay?

At first, I just started dropping my files into Handbrake using either the built-in presets (high profile was the recommended one on the forums) or by downloading and using specific presets for new Apple devices.  That worked great, the converted files played just fine on the Apple TV, save for two problems.  First was that since I was re-encoding stuff that was already compressed, there was a loss in video quality (though it was slight) and second was the process took an eternity.  With my dual core iMac it would take a few hours to convert a full-length 720P video.

Being an impatient person, I decided to school myself in the world of video conversion, so with some quality time with Google and some trial and error I was able to come up with a solution that will save countless hours of encoding, I hope.  Now is a good time to point out that none of this stuff is really original though, just things I found online. I will try my best to provide links wherever possible, but if I missed something, please let me know.  So let's move on to some of the technical details.

Tech Overview (or really basic things I should have known but didn't)
To understand the most efficient way to get video compatible with the Apple TV, I found it was important to learn about some basics like the ATV's capabilities, the difference between codecs and containers and some of the more common formats for (totally legit) files out there.

Supported Video Formats for the Apple TV 2:  There are basically a few highlights to take away from the spec details.  The only videos ATV can play must be MP4 files (.mp4, m4v, or mov), have to be encoded as h.264 and have an AAC soundtrack with a bit rate no higher than 160kbps.  Alternately, the files can have an AC3 soundtrack ONLY IF your Apple TV is connected to a receiver or TV that can decode that format.  That last part is where I ran into most of my trouble, more on that later.

Ok, let's dissect the supported format details a little bit more to try and get a feel for what we need to look out for.  The first part of the specs are what I tend to refer to as the container, which is probably not technically correct, but it's my blog dammit and I can call it whatever I want.  So by container I'm referring to the actual file type that stores the video and audio information that makes up a video file.  There are a number of containers running around out there in the wild, there are of course the ones directly supported by Apple like mp4, then there are the more popular and unsupported ones that we are mostly interested in like mkv and avi.

As mentioned, these container files contain video and audio streams, in many cases, the video stream included in a container unsupported by Apple TV is actually compatible with the Apple TV.  If a video file includes a stream encoded with h.264 (you may also see it called x264) then assuming it's not encoded at an outrageous bit or frame rate, it will be playable without transcoding into a different format.  If your file uses a different codec like XviD or DivX, you'll have no choice but to re-encode the entire file.

The same thing applies to the audio codec stored in the file, if it's compatible, it won't need to be converted.  The bad news is that I haven't seen a lot of mkv files with stereo AAC.  This particular point is where I ran into issues, especially finding a utility that could do video pass through while re-encoding the audio.  The good news is it takes WAY less time to just transcode audio when converting a video to an mp4.

ProTip: Most video players (like VLC or Quicktime) have an option to bring up information on a particular video which should tell you exactly what codecs are being used as well as bit rates and frame rates.

Get info in Quicktime X (using Perian). We would need to transcode the video in this file since it's DivX.  The audio is compatible, however.

Ok, that's great but...
So let's cover some of the applications and techniques required to actually make this thing work.  There are a bunch of different things you can use, depending on what you need to do and your individual preferences.   Tools that I personally use a lot are Handbrake (for full on encoding), Quicktime X (with Perian installed to allow me to load most types of videos) and finally, MKVtools.  Of the three applications listed here, MKVtools can actually do everything and will be the focus of the rest of this post.

Using the information I had up to this point, I was using QuickTime to simply convert the container to MOV for existing h.264 video by using save as and selecting the movie format.  This resulted in a file that imported into iTunes and played just fine, hooray!  Then I tried to open it up on my Apple TV, it attempted to stream it, but quickly produced the dreaded "An Error Occurred Loading this Content. Try Again Later" message.  Researching the error message, I found this blog post that indicated the ATV might have trouble detecting the AC3 audio stream.  I switched the AC3 setting to on, but alas, my super cheap-o house brand TV couldn't handle an AC3 signal apparently as I continued to get the error message when attempting to play the video.

Create Apple TV magic with MKVtools
Ok, so I need to find a tool to allow me to pass through the video stream while transcoding the audio stream to a compatible AAC format.  I keep a number of tools around, but I found none of them really offered the ability to pass through video, most could pass through audio, but not video.  Some searching revealed that the old version of QuickTime Pro would let you do that when saving as, but that wasn't available to me.  I also have a copy of VisualHub laying around, which lets you use the pass through flags for both video and audio.  For some reason the resulting video files were unplayable in anything.  Since VisualHub has been abandoned for years, I decided to keep looking.

I stumbled across this thread on the Mac Rumors forum that was discussing methods of using a bunch of different command line open source tools to manually extract/convert the video and audio streams and then re-combine (mux) them back into a playable mp4 file.  Too much work for me :)  However, the thread was 18 pages long, and in the latter pages there was mention of a graphical front end for these tools called MKVtools.  

MKVtools is available with an unlimited trial that limits a few minor features (like queuing) but otherwise is fully functional.  The author wants to make sure the application works for you before you pony up the princely sum of... 5 bucks!

MKVtools can take a variety of formats as input and produce avi and most importantly mp4 files as the results.  It can do full encoding of video and audio or selectively let you select either or none for conversion to an mp4 file.  As an additional bonus, you can also keep (or convert to) an AC3 sound track and just add your AAC stereo stream as a secondary stream.  This solution worked for me and allows me 5.1 compatibility with my TiVo that is connected to a proper AV receiver.  As an added bonus you can also work with subtitles, though if you want to hard code a subtitle stream from an mkv file to your mp4, you will have to re-encode your video file.

The main MKVtools window with our sample file loaded

Once your video file is loaded, the main window will show you the specifics of the streams included in your video file.  Note the track info for the video stream.  It is listed as mpeg4.  This is NOT compatible with the ATV, if it were it would be listed as h.264.  Therefore, in this particular example we would need to do a full re-encode, this tool would work, or you could always use your favorite tool to re-encode.

The MP4 section

Here's the mp4 section where we'll be creating our compatible mp4.  We'll ignore the incompatible format for the moment and pretend we have a real h.264 stream.  To convert, check the boxes next to the audio and video streams in the top window.  Then simply click on the pass thru button in the video section, this video already has an aac stream, so we'll also select pass thru for audio and allow the application to just repackage the file as an mp4 file.  If the packaged audio were AC3, you could also leave pass-thru selected and simply check the add 2 Ch. track button, it will leave the existing audio alone and add an AAC secondary track which could be read by the Apple TV.  Finally if you have some other type of 5.1 soundtrack, like DTS, you could select either AAC 2ch or AC3 (5.1) and then check the add 2 Ch. track button.

Click convert and the resulting file will by default be saved to the same location as the original (along with the log file and intermediate audio and video files).  You can change the save location and the retention of the extra files in the preferences.

Thanks for sticking with me this far.  The summary of all this is below if you need a quick reference (or didn't want to read this massive article).

Summary
Ok, so here's the basic steps for conversion happiness in numbered list form.  The moral of the story is if you have a video already encoded in h.264, you want to avoid re-encoding the video if at all possible.

1. Download MKVTools
2. Open your video file in MKVTools and click the MP4 tab
3. Select the video and audio streams check boxes, note the codecs of each
4. If video is listed as h.264, select pass thru in the video section.  Otherwise, select one of the options to re-encode the video (or use your favorite other video encoding tool, you can't pass through the video for this file).
5. If the audio is listed as aac 2ch. no futher steps are required, select pass thru for audio then click convert.
6. If the audio is AC3, click the add 2 Ch. track button then click convert
7. If the audio is something else, select AAC 2ch and click convert, you may be able to convert 5.1 sound to ac3 5.1 but make sure to click the add 2 ch. track option, click convert.
8. import the resulting mp4 file into itunes when it's done.
9. View the file on the Apple TV.

Play any Shoutcast stream through TiVo (OS X and iTunes)

I discovered this feature by total accident the other day.  This may be well known, but who knows, I figured it might be worth sharing to anyone that stumbles across this.

Those of us with TiVos and Macs are kind of left out in the cold when it comes to the built-in functionality of the TiVo supplied TiVo Desktop for Macs.  You get only photo and music sharing (and some basic video publishing if you hack it).  If you want anything more advanced you have to spring for commercial software from Roxio (or go with one of the unsupported, but excellent community apps like PyTiVoX).  Anyway, there's one thing the stock software offers that as far as I know the Windows version doesn't: sharing your iTunes library complete with playlists to your TiVo units.

The fact you can share iTunes playlists with the Mac version of the default desktop is nice to begin with.  But, it turns out if you have a ShoutCast (or similar streaming audio) location saved within iTunes, you can select it from your library and the stream will play. I noticed this simply because a streaming station ended up in a smart playlist and just started playing along with my regular MP3 files.

It's pretty simple to setup, here's the details:

The first step is to download and install the TiVo Desktop then configure iTunes Sharing.

Next, add the address for a stream in iTunes, which can be found under the Advanced menu.  As an example, I'm using the audio feed from TWiT, which steams live and recorded audio from the podcasts being recorded by Leo Laporte's TWiT network.

Enter the URL:

When you click OK, a link to the stream will show up in your music library like any other audio file.  To make it easier to find on the TiVo side you can add it to or create a new playlist so it's listed on the top level of your iTunes listing on your DVR.

Once you have the file in iTunes, go over to your DVR and select your shared iTunes library from the Music, Photos and Showcases menu.  From there you can navigate to the location of your newly added stream and just hit play to begin streaming.

Keep in mind this is just simple streaming, as far as I can tell it won't send across any metadata such as current track or play times, but it's a nice little undocumented feature to have.

I almost want an iPad (Or NO FARMVILLE FOR YOU!)

Quick math quiz - solve for X: Old Dell Laptop + X > iPad - X

The solution in just a minute, but first some background. I, like most of the free world spent a great deal of time following the live coverage of the new Jesus Pad powered by Unicorn Tears presented by Steve Jobs himself (instead of being a productive contributor to my employer). At first I was wondering what was wrong with me. I just wasn't beside myself with crazy fervor over the iPad. I must be running low on Apple Kool-Aid, surely they couldn't have totally whiffed what was supposed to be their most revolutionary gizmo ever. It was just a big ass iPod Touch, no wild hand flailing like a futuristic Tom Cruise in Minority Report, no wireless HDMI, and not even a single camera (let alone two of them).

At least the price was a bit of a surprise. Sure, it still out costs most serviceable netbooks and even some full notebooks, but for a first gen Apple product to be under 7 bones, that was pretty impressive. Still I kept wondering what was wrong with me, why wasn't I trying to figure out the best way to make this thing a part of my life? I simply decided I must be in the process of losing my cool, I am a parent after all, and by the time the kids hit middle school the cool will be completely gone anyway.

So I did something odd, I simply went on with my life and got back to work. Then my thoughts wandered off into the realm of how my wife and I use our computers at home. We each have our own laptops (I inherit her old ones, she gets the new ones.) My wife's MacBook is getting to be of a certain age, so soon it shall be mine, and my old Dell can find a new life on a farm where it can frolic and play with other decrepit computers.

Then thoughts of the iPad and the $499 entry price sunk in. Hey, you know you could buy TWO iPads for the price of a MacBook and each have your own shiny new Unicorn Tear powered device. No more stupid arguments over who's the most downtrodden second-class citizen in the family. Being at work, my mind shifted to use case and workflow analysis.

My wife and I pretty much do the same stuff on our laptops; we just sort of fart around with them after the kids go to bed. You know the same thing everyone does, e-mail, Facebook, web surfing and so forth. Hmm... the iPad does that, and it offers a seemingly better form factor (and no wobbly lid and loose hard drive cage like my Dell). Holy cow, now I get it... NOW I'M COOL AGAIN!

I was ready to start pitching my wife on the idea when we got home from work. And I started thinking a little bit more about the specific sites we visit on our laptops. Then I had a horrible thought: No Tiki Farm! No Farmville! CRAP! My wife more than me is addicted to one of these silly Facebook games, and then I realized, it's Flash based. Early reviews of the iPad confirm there's no Flash. After a quick inventory of the sites I visited frequently, I realized there was a lot of Flash involved. It's kind of a big deal

Sure we've all lived with iPhones without Flash for a long time now, but that's a different animal in my mind. I use the phone differently, sure it's sometimes a bummer to not have Flash support, but I still prefer to do a lot of things on a larger screen. Not having it on the iPad on the other hand is a serious problem. Basically, I would need to have the iPad AND still hang on to my rickety old Dell so I can whip it out when I feel the need to watch Hulu or any number of other tasks. The old laptop still can surf the web, and check e-mail and do everything else the iPad can do, except I can't turn it upside down, the screen would probably fall off or something.

Yeah that seems kinda stupid to me too. So the answer to our math quiz is flash: Old Dell + Flash > iPad - Flash. It can do all the web surfing, watching videos e-mail fairly competently as well as play stupid little games where I have to try and annoy enough friends into joining my quest to plant virtual pineapples.

Ok, well it's early in the game the iPad isn't in anyone's hands yet, maybe Flash can make it into the iPad at or shortly after shipping time. Also, perhaps pigs will fly.

But then again, Steve's reasoning was that the resource hog nature of Flash would melt the poor little iPhone into a quivering mass of molten mobile phone processor bits. The iPhone just couldn't give Flash the necessary resources. Ok, understandable I've seen how some apps simply make my poor phone whimper in pain. And of course being a dedicated fanboi, I'd never question His Jobbiness. Well, except there's one thing I don't quite understand Mr. Jobs - unlike the iPhone, by all accounts, the iPad runs like a stripe-assed ape. Imagine I said that last part like detective Columbo.

So if the iPad is a speed demon, is there actually a reason (other than it is kinda crappy) Apple will put forward as to why Flash won't work on it this time?

While I wouldn't mind cashing in my old Dell for my wife's gently used MacBook, I'd much rather have other options. I burn my incense and swing the dead chicken over my head at my little shrine of the great Apple (while facing Cupertino of course) and pray that there may be a lasting peace made between the houses of Adobe and Apple and the iPad could indeed become a respectable citizen of my leisure time.

I promise you Oh Great Jobs, I won't be greedy, I won't ask for Flash on the iPhone. I will trust in your wisdom in such magics, but please please let me live a peaceful existence of technical equity with my significant other.

Until then, you can find me at the store buying duct tape for my trusty old Dell.

Static Routes on Snow Leopard

Snow Leopard seems to be a mixed bag of core changes to underlying infrastructure. Adding static routes appears to be one of those changes. See the end of this post if you just want to see the new syntax.

Welcome to another sporadic and completely random tech update. This will be a short one as I just have a little bit of info to share that I couldn't quickly find on the interwebs with the google. Turns out there are some slight changes to the terminal command to add a static route.

I am currently running OS X Snow Leopard (10.6) on my work MacBook. The update was not pretty and I essentially had to completely wipe my OS X partition and start over. Along with that comes a few incompatibilities, most notably for me, MS Live Mesh doesn't work any more, which is how I had intended to get my data back onto my MacBook after I had to clean install, oops!

But here's a slightly obscure change that probably affects very few people:
The basic command to add a static route has changed slightly. I use a static route to connect to a split tunnel VPN (bad practice I know, but it's a long story).

The old command I used to use:
sudo route add -net 192.168.x.0 192.168.y.z
Appeard to work as it used to, but no traffic would actually route through the VPN tunnel. I searched around and saw no references other than a blog comment from today (Sep 2) saying the command in that how-to wouldn't work for them.

So it seemed to me that somehow the way static routes were added changed a little bit. I broke down and read the man page for route (man route) to see if I could figure anything out. Granted I never read it before so I have no idea if there was a difference, but I noticed the following optional parameters that could be used in the route command like static (indicate a static route) and interface (which tells it the interface is directly connected already). I was then able to successfully create a static route using the following command:

sudo route add -net 192.168.x.0 192.168.y.z -static -interface

A call to Mac users: stop with the malware huberis already!

Ok, first of all I am currently a Mac user and have been off and on for years. I say this as a friend, not some sort of anti-mac PC geek: Stop being so damned smug. I am talking about the stance on the lack of malware. This stance is pretty much echoed by everyone on the pro-Mac side of the fence from users all the way to Apple itself (just watch a random I'm a Mac, I'm a PC ad).

I was listening to the latest installment of Macbreak Weekly, and a couple of panelists were extolling the virtue of the malware free paradise that is OS X. Well here's the deal folks. This sort of thing is absolutely a dangerous attitude. How many Mac users use any sort of anti-virus at all? Pretty few. To that point, there actually ARE very few AV programs to begin with. Edit, there is the open source (and keeping to my roots, free)ClamXav.

Sure, the Mac OS is actually reasonably secure out of the box (we'll ignore the fact that the firewall is OFF by default) but some of the other protections work very well and are what Windows Vista wanted to be *cough* UAC *cough*

What we have here is the calm before the storm or so to speak. These days, malware, botnets viruses (virii?) are all about dollars and sense. Gone are the days of the pimply faced kid writing a little code to erase hard drives. The bad guys are writing stuff to make money by stealing personal data, sending spam, DDOS blackmail, you name it. Since this is a business, it's simply much more profitable to target the PC world.

Therefore, many Mac users sit around sneering at the PC world because the Mac folk live some sort of mythical land of rainbows and unicorn tears that no virus dare enter. I'm pretty sure the people in Pompeii kinda thought the same thing about volcanoes too.

That will always be the case, a target rich environment will always get the attention, but think about the Mac side of things for a while. There's a 10% or so slice of the market that's almost completely unprotected. Once somebody really takes advantage of this situation (and I do mean when, not if) it will be an unmitigated crisis.

Think I'm being a little bit alarmist? You might have missed this, but there's a currently exploitable (as in drive by scripting) vulnerability in the Java implementation of OS X. Despite 10.5.7 being put out mere days ago, this vulnerability has not been patched even though it's been known for at least six months. For more, visit this site which even features a proof of concept that will execute arbitrary (but harmless) code on your very own magic, walled unicorn powered Garden of Eden.

So my Mac friends, enjoy your beautifuly designed hardware, (mostly) reliable OS and fantastic UI. But please, please, stop thinking it's a freakin' suit of armor and go install some sort of anti-malware. Tin foil hat optional.

iStat Menus

I'm not sure why I haven't written about iStat Menus before. I think it's because it's about the first thing I grab on any Mac I'll be using for more than 10 minutes. Some people do the same with Quicksilver (I have yet to figure out why it would make my life that much better, call me backwards). iStat Menus is the menu bar version of iStat, which is a utility for viewing a number of different meters representing system performance.

While I generally learned pretty quickly that menu bar apps add a whole lot of clutter very quickly, iStat has earned a spot on the second most coveted real estate on my screen. The first is the desktop and nothing goes there 'cept the little hard drive icons.

iStat gives you the ability to view CPU, Memory, drives, network, temp, fans, power, and Bluetooth status. You can throw all of these into your menu, or pick and chose which ones grace your screen. I personally like the CPU, Memory, Network, and oddly enough, date and time.

Date and time? Not exactly a performance measure, is it? Well no, but I happen to like it better than the default option (I hide that one in system prefs) It is more compact when you are displaying the date, and if you click the date it gives you a neat-o pop up calendar.


One thing I miss about my life in the PC world is my PC machines generally had some sort of hard disk indicator light somewhere on them. So when the PC was acting odd, I could always glance down at the hardware and see that the disk was thrashing around like mad. I was slightly clueless when my Mac did the same thing (oh yes Macs are just as capable at sucking as PCs). Now I have a cool blue icon that does the same thing. It's hard to tell from any screen shots, but it even has indicators on the icon to show if the disk is reading or writing.

The other three indicators are pretty much common stuff (you Vista users are familiar with the little desktop gadget that looks like a car instrument cluster and shows this stuff off, everyone has that one.) I have at a glance my physical memory in use, cpu usage and network activity. One cool thing about it is that you can click on one of the indicators and it will give you the top processes using the resource. This was great for finding out that our stupid printer driver goes nuts on occasion and ties up an entire core for no good reason whatsoever.


Sure, most of the stuff the app does isn't exactly revolutionary, there's tons of other apps out there that do the same thing. My favorite thing about this one is that it's a simple heads up at-a-glance indicator that takes up little screen real estate. I can easily click to get more in depth details on what's happening and even launch OS X's native task management software to kill misbehaving processes. The only thing I really wish I could do with this would be to kill an app directly from the menu rather than having to jump over to Activity Monitor.

The price is right at zero dollars, you can grab it at the iSlayer website.

PyTivo

Welcome to my random musings of stuff I find useful, and blissfully cost free Remember, a lot of this stuff involves a lot of time and sweat from real people. You should support these efforts if you find them very useful by whatever method requested by the authors.

I’ll start off the festivities with a little writeup on an app that has transformed my TiVO from a simple DVR to a whole house video jukebox system.

PyTivo is an open source effort that uses a number of different open source tools to convert just about any video format you can think of and serving it up to any Series 2, 3, HD, etc tivos on your local network. When configured, your PC, or Mac, or Linux box will show up on your TiVO in a similar manner to other DVRs. Just select the video you want to transfer, and depending on your network speed and quality settings, either go find something else to do, or start watching right away.

PyTivo can be configured to serve up both Standard Def and HD content, and can even down convert HD content to your old caveman era SD TiVOs eliminating the need for multiple copies of the same content. Most video formats are supported including VOB, MKV, AVI, MP4.

I run PyTivo on my iMac with a USB external HDD. Oddly enough, the original version of PyTivo is actually pretty difficult on the Mac. You have to install the actual PyTivo as well as compiling a version of FFMEPG to run on your Mac (which involves XCode tools and downloading the components from an online repository). It’s pretty intimidating at first, but once it’s up and running its very stable. By contrast, the PC version is a simple single installer and you’re up and running. All versions use a web interface to tweak the settings.

Fortunately, someone has finally stepped in and built a GUI version of PyTivo called PyTivoX. The GUI version is very easy to configure, but does lack the ability to control some of the advanced settings from within the GUI (you can still modify the configuration files to change the advanced settings as you would in the regular version of PyTivo). I ran the graphical version briefly, but as I had my older version up and running with the advanced settings configured, I stuck with that version.

Once everything was up and running, I pretty much only have to fire up the DVD Player when I don’t want to take the time to rip a DVD. The kids wants to watch Finding Nemo for the 43rd time on any TV in the house? No problem, it’s on TiVO.

Useful links:

PyTiVO project: http://pytivo.armooo.net/

PyTiVO discussion forums: http://pytivo.krkeegan.com/

The steps I followed to install the original PyTivo: http://www.tivoblog.com/archives/2008/12/01/how-to-install-pytivo-on-a-mac/

PyTiVOX: http://code.google.com/p/pytivox/