Hello. I am also a Mac user with a PS3. I have files that play perfectly on the PS3 via internal HDD, but serviio sends as "Unsupported Data" which can not be played: "The data type is not supported."
Note that I have turned off the transcoding options in serviio... these files are natively supported by the ps3.
For instance,
./ffmpeg -i 1408.m4v
- Code:
FFmpeg version UNKNOWN, Copyright (c) 2000-2011 the FFmpeg developers
built on Jan 12 2011 19:51:48 with gcc 4.0.1 (Apple Inc. build 5493)
configuration: --enable-static --disable-shared --bindir=/tmp/ffmpeg --disable-ffplay --disable-ffserver --enable-libmp3lame --enable-pthreads --disable-mmx --extra-ldflags=-L/tmp/static/lib --extra-cflags=-I/tmp/static/include
libavutil 50.36. 0 / 50.36. 0
libavcore 0.16. 0 / 0.16. 0
libavcodec 52.108. 0 / 52.108. 0
libavformat 52.92. 0 / 52.92. 0
libavdevice 52. 2. 3 / 52. 2. 3
libavfilter 1.72. 0 / 1.72. 0
libswscale 0.12. 0 / 0.12. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '1408.m4v':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42isomavc1
creation_time : 2011-05-10 14:53:16
encoder : HandBrake 3975svn 2011050901
title : 1408
artist : John Cusack, Samuel L. Jackson, Tony Shalhoub, Mary McCormack, Jasmine Jessica Anthony, David Nicholson, Alexandra Silber, Emily Harvey, Noah Lee Margetts
date : 2007-06-22T00:00:00Z
genre : Horror
description : Based on a short story by Stephen King, a man who specializes in
debunking the paranormal checks into the infamous room 1408 in the
Dolphin Hotel, only to discover; the terror is real.
synopsis : The cynical and skeptical writer Mike Enslin writes books evaluating supernatural phenomena in hotels, graveyards and other haunted places, usually debunking the mystery. While writing his last book, he travels from Los Angeles to New York to spend one night in the evil room 1408 of the Dolphin Hotel, which is permanently unavailable for guests. The reluctant manager Mr. Gerald Olin objects to his request and offers an upgrade, expensive booze and finally the reports relating the death of more than fifty guests along decades in the cursed room. However, Mike threatens Mr. Oiln, promising to sue the hotel, and finally checks in the room. Along the night, he finds that guests of room 1408 can check out when they like, but they can never leave the room alive.
Duration: 01:52:25.81, start: 0.000000, bitrate: 5131 kb/s
Stream #0.0(und): Video: h264, yuv420p, 1920x798 [PAR 1:1 DAR 320:133], 4321 kb/s, 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
Metadata:
creation_time : 2011-05-10 14:53:16
Stream #0.1(eng): Audio: ac3, 48000 Hz, 6 channels, s16, 640 kb/s
Metadata:
creation_time : 2011-05-10 14:53:16
Stream #0.2(eng): Audio: aac, 48000 Hz, stereo, s16, 164 kb/s
Metadata:
creation_time : 2011-05-10 14:53:16
At least one output file must be specified
MediaInfo:
- Code:
General
Complete name : /Users/Shared/Movies/1408.m4v
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42
File size : 4.03 GiB
Duration : 1h 52mn
Overall bit rate : 5 131 Kbps
Movie name : 1408
Performer : John Cusack, Samuel L. Jackson, Tony Shalhoub, Mary McCormack, Jasmine Jessica Anthony, David Nicholson, Alexandra Silber, Emily Harvey, Noah Lee Margetts
Genre : Horror
Recorded date : UTC 2007-06-22 00:00:00
Encoded date : UTC 2011-05-10 14:53:16
Tagged date : UTC 2011-05-10 21:07:10
Writing application : HandBrake 3975svn 2011050901
Cover : Yes
iTunEXTC : mpaa|PG-13|300|
desc : Based on a short story by Stephen King, a man who specializes in / debunking the paranormal checks into the infamous room 1408 in the / Dolphin Hotel, only to discover; the terror is real.
ldes : The cynical and skeptical writer Mike Enslin writes books evaluating supernatural phenomena in hotels, graveyards and other haunted places, usually debunking the mystery. While writing his last book, he travels from Los Angeles to New York to spend one night in the evil room 1408 of the Dolphin Hotel, which is permanently unavailable for guests. The reluctant manager Mr. Gerald Olin objects to his request and offers an upgrade, expensive booze and finally the reports relating the death of more than fifty guests along decades in the cursed room. However, Mike threatens Mr. Oiln, promising to sue the hotel, and finally checks in the room. Along the night, he finds that guests of room 1408 can check out when they like, but they can never leave the room alive.
stik : 0
iTunMOVI : <?xml version="1.0" encoding="UTF-8"?> / <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> / <plist version="1.0"> / <dict> / <key>cast</key> / <array> / <dict> / <key>name</key> / <string>John Cusack</string> / </dict> / <dict> / <key>name</key> / <string>Samuel L. Jackson</string> / </dict> / <dict> / <key>name</key> / <string>Tony Shalhoub</string> / </dict> / <dict> / <key>name</key> / <string>Mary McCormack</string> / </dict> / <dict> / <key>name</key> / <string>Jasmine Jessica Anthony</string> / </dict> / <dict> / <key>name</key> / <string>David Nicholson</string> / </dict> / <dict> / <key>name</key> / <string>Alexandra Silber</string> / </dict> / <dict> / <key>name</key> / <string>Emily Harvey</string> / </dict> / <dict> / <key>name</key> / <string>Noah Lee Margetts</string> / </dict> / </array> / <key>directors</key> / <array> / <dict> / <key>name</key> / <string>Mikael Håfström</string> / </dict> / </array> / <key>producers</key> / <array> / <dict> / <key>name</key> / <string>Lorenzo di Bonaventura</string> / </dict> / </array> / <key>screenwriters</key> / <array> / <dict> / <key>name</key> / <string>Matt Greenberg</string> / </dict> / <dict> / <key>name</key> / <string>Scott Alexander</string> / </dict> / <dict> / <key>name</key> / <string>Larry Karaszewski</string> / </dict> / </array> / </dict> / </plist>
rtng : 0
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 1h 52mn
Bit rate mode : Variable
Bit rate : 4 321 Kbps
Width : 1 920 pixels
Height : 798 pixels
Display aspect ratio : 2.40:1
Frame rate mode : Variable
Frame rate : 23.976 fps
Minimum frame rate : 23.974 fps
Maximum frame rate : 23.981 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.118
Stream size : 3.39 GiB (84%)
Writing library : x264 core 115
Encoding settings : cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=umh / subme=8 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=4 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=24 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=21.0 / qcomp=0.60 / qpmin=3 / qpmax=69 / qpstep=4 / vbv_maxrate=15000 / vbv_bufsize=15000 / crf_max=0.0 / nal_hrd=none / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2011-05-10 14:53:16
Tagged date : UTC 2011-05-10 19:44:24
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177
Audio #1
ID : 2
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Codec ID : ac-3
Duration : 1h 52mn
Bit rate mode : Constant
Bit rate : 640 Kbps
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 515 MiB (12%)
Language : English
Encoded date : UTC 2011-05-10 14:53:16
Tagged date : UTC 2011-05-10 19:44:24
Audio #2
ID : 3
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 1h 52mn
Bit rate mode : Variable
Bit rate : 164 Kbps
Maximum bit rate : 253 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Stream size : 132 MiB (3%)
Language : English
Encoded date : UTC 2011-05-10 14:53:16
Tagged date : UTC 2011-05-10 19:44:24
This is an mp4 (m4v) file encoded with HandBrake, then tagged with MetaX. It works perfectly on the Mac via Quicktime or VLC. It works perfectly on the PS3 if copied over to the internal drive. It also streams fine via DLNA if using TVMOBiLi (no transcoder) on the Mac or even WMP12 on Win7. Heck, the PS3 will even play it HTTP streamed via "python -m SimpleHTTPServer 3333"! But, sadly, my favorite application serviio hands out "Unsupported Data".

Interestingly enough, another file I encoded using the same process works just fine:
./ffmpeg -i "Tron Legacy.m4v"
- Code:
FFmpeg version UNKNOWN, Copyright (c) 2000-2011 the FFmpeg developers
built on Jan 12 2011 19:51:48 with gcc 4.0.1 (Apple Inc. build 5493)
configuration: --enable-static --disable-shared --bindir=/tmp/ffmpeg --disable-ffplay --disable-ffserver --enable-libmp3lame --enable-pthreads --disable-mmx --extra-ldflags=-L/tmp/static/lib --extra-cflags=-I/tmp/static/include
libavutil 50.36. 0 / 50.36. 0
libavcore 0.16. 0 / 0.16. 0
libavcodec 52.108. 0 / 52.108. 0
libavformat 52.92. 0 / 52.92. 0
libavdevice 52. 2. 3 / 52. 2. 3
libavfilter 1.72. 0 / 1.72. 0
libswscale 0.12. 0 / 0.12. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Tron Legacy.m4v':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42isomavc1
creation_time : 2011-05-09 04:48:40
encoder : HandBrake 3975svn 2011050901
title : Tron Legacy
artist : Jeff Bridges, Garrett Hedlund, Olivia Wilde, Bruce Boxleitner, Michael Sheen
date : 2010-05-09T00:00:00Z
genre : Sci-Fi & Fantasy
description : The son of a virtual world designer goes looking for his father and ends up inside the digital world that his father designed. He meets his father's creation turned bad and a unique ally who was born inside the digital domain of The Grid.
synopsis : Sam Flynn, the tech-savvy 27-year-old son of Kevin Flynn, looks into his father's disappearance and finds himself pulled into the same world of fierce programs and gladiatorial games where his father has been living for 20 years. Along with Kevin's loyal confidant, father and son embark on a life-and-death journey across a visually-stunning cyber universe that has become far more advanced and exceedingly dangerous. Written by ZootCatchy
Duration: 02:05:08.67, start: 0.000000, bitrate: 5466 kb/s
Stream #0.0(und): Video: h264, yuv420p, 1920x816 [PAR 1:1 DAR 40:17], 4828 kb/s, 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
Metadata:
creation_time : 2011-05-09 04:48:40
Stream #0.1(und): Audio: aac, 48000 Hz, 5.1, s16, 466 kb/s
Metadata:
creation_time : 2011-05-09 04:48:40
Stream #0.2(und): Audio: aac, 48000 Hz, stereo, s16, 164 kb/s
Metadata:
creation_time : 2011-05-09 04:48:40
At least one output file must be specified
MediaInfo:
- Code:
General
Complete name : /Users/Shared/Movies/Tron Legacy.m4v
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42
File size : 4.78 GiB
Duration : 2h 5mn
Overall bit rate : 5 467 Kbps
Movie name : Tron Legacy
Performer : Jeff Bridges, Garrett Hedlund, Olivia Wilde, Bruce Boxleitner, Michael Sheen
Genre : Sci-Fi & Fantasy
Recorded date : UTC 2010-05-09 00:00:00
Encoded date : UTC 2011-05-09 04:48:40
Tagged date : UTC 2011-05-09 14:07:42
Writing application : HandBrake 3975svn 2011050901
Cover : Yes
iTunEXTC : mpaa|PG|200|
desc : The son of a virtual world designer goes looking for his father and ends up inside the digital world that his father designed. He meets his father's creation turned bad and a unique ally who was born inside the digital domain of The Grid.
ldes : Sam Flynn, the tech-savvy 27-year-old son of Kevin Flynn, looks into his father's disappearance and finds himself pulled into the same world of fierce programs and gladiatorial games where his father has been living for 20 years. Along with Kevin's loyal confidant, father and son embark on a life-and-death journey across a visually-stunning cyber universe that has become far more advanced and exceedingly dangerous. Written by ZootCatchy
stik : 0
iTunMOVI : <?xml version="1.0" encoding="UTF-8"?> / <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> / <plist version="1.0"> / <dict> / <key>cast</key> / <array> / <dict> / <key>name</key> / <string>Jeff Bridges</string> / </dict> / <dict> / <key>name</key> / <string>Garrett Hedlund</string> / </dict> / <dict> / <key>name</key> / <string>Olivia Wilde</string> / </dict> / <dict> / <key>name</key> / <string>Bruce Boxleitner</string> / </dict> / <dict> / <key>name</key> / <string>Michael Sheen</string> / </dict> / </array> / <key>directors</key> / <array> / <dict> / <key>name</key> / <string>Joseph Kosinski</string> / </dict> / </array> / <key>producers</key> / <array> / <dict> / <key>name</key> / <string>Sean Bailey</string> / </dict> / <dict> / <key>name</key> / <string>Steven Lisberger</string> / </dict> / <dict> / <key>name</key> / <string>Jeffrey Silver</string> / </dict> / </array> / <key>screenwriters</key> / <array> / <dict> / <key>name</key> / <string>Edward Kitsis</string> / </dict> / <dict> / <key>name</key> / <string>Adam Horowitz</string> / </dict> / </array> / </dict> / </plist>
rtng : 0
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 2h 5mn
Bit rate mode : Variable
Bit rate : 4 829 Kbps
Width : 1 920 pixels
Height : 816 pixels
Display aspect ratio : 2.35:1
Frame rate mode : Variable
Frame rate : 23.976 fps
Minimum frame rate : 23.968 fps
Maximum frame rate : 23.981 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.129
Stream size : 4.22 GiB (88%)
Writing library : x264 core 115
Encoding settings : cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=umh / subme=8 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=4 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=24 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=21.0 / qcomp=0.60 / qpmin=3 / qpmax=69 / qpstep=4 / vbv_maxrate=15000 / vbv_bufsize=15000 / crf_max=0.0 / nal_hrd=none / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2011-05-09 04:48:40
Tagged date : UTC 2011-05-09 10:33:14
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177
Audio #1
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 2h 5mn
Bit rate mode : Variable
Bit rate : 466 Kbps
Maximum bit rate : 726 Kbps
Channel(s) : 6 channels
Channel positions : Front: L C R, Side: L R, LFE
Sampling rate : 48.0 KHz
Compression mode : Lossy
Stream size : 417 MiB (9%)
Encoded date : UTC 2011-05-09 04:48:40
Tagged date : UTC 2011-05-09 10:33:14
Audio #2
ID : 3
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 2h 5mn
Bit rate mode : Variable
Bit rate : 164 Kbps
Maximum bit rate : 275 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Stream size : 147 MiB (3%)
Encoded date : UTC 2011-05-09 04:48:40
Tagged date : UTC 2011-05-09 10:33:14
The only difference would be the 5.1 AC3 audio passthru in the "Unsupported Data" file, whereas the one the works contains 6 channel AAC.
EDIT: So I took the first file with AC3 track, demuxed it into elementary streams with mp4box. I then remuxed just the h264 stream and the AAC stream, and the file can now be served successfully. It appears as though serviio is choking when the mp4 contains an AC3 audio track.