Ok, it took me quite a while of messing around with, and more than my fair share of random blog reading, but I have found a simple setup that allows the on the fly transcoding of HD (I tested only 720p videos) MKV's to your PS3.

It's actually fairly simple, and there are only a few things that you need. For reference the machine this was done on was running Ubuntu 9.04 64 bit. Run all the commands like this in Terminal.

1. sudo apt-get update

2. sudo apt-get install mencoder mediatomb

3. mediatomb let it run for about 5 seconds, then "ctrl c" to close it

4. sudo gedit /home/username/.mediatomb/config.xml paste in the mediatomb configuration below, but remember to change the USERNAME throughout the whole of the file. Save the file and exit.

5. sudo gedit /home/USERNAME/.mediatomb/mencoder-tr paste in the mencoder script below, save and exit.

6. sudo chmod 775 /home/USERNAME/.mediatomb/mencoder-tr

7. mediatomb

Sample Mediatomb Config
Mencoder Script

You should be done. All you have to do now is open up the supplied link and start adding those HD videos to your library. For the linux n00bs a brief explanation of what each step does it below.

1. Makes sure everything is up to date
2. Installs both mencoder (which will be doing the transcoding) and mediatomb, which will be serving up the transcoded files
3. Starts mediatomb, and lets it generate it's configuration file, then closes it
4. Opens up said configuration file for editing, and allows you to paste in the sample configuration
5. Creates a file for the transcoding script, and lets you paste in the sample mencoder script.
6. Changes the permissions on the script we just created so that mediatomb can execute it
7. Restarts mediatomb with the new working configuration file and script

Feel free to comment or send me an email.