Failed retreavin media information for AKAMAI applehttp

[ottomatic]

Hi,

I am writing a new WebResource plugin for svtplay, and I am finding that Serviio can not sniff out the media type for the applehttp stream.

Here is an error from the log file:

2012-11-16 00:35:39,074 DEBUG [FeedUpdaterThread] Retrieving file size from the URL connection
2012-11-16 00:35:39,567 DEBUG [FeedUpdaterThread] Retrieving information about the video stream 'Adrian'
2012-11-16 00:35:39,568 DEBUG [FFMPEGWrapper] Invoking FFMPEG to retrieve media information for file: http://svtplay9d-f.akamaihd.net/i/se/se ... K5kA%3d%3d
2012-11-16 00:35:39,568 DEBUG [ProcessExecutor] Starting C:\Program Files\Serviio\lib\ffmpeg.exe -i http://svtplay9d-f.akamaihd.net/i/se/se ... K5kA%3d%3d
2012-11-16 00:35:40,710 WARN [FeedUpdaterThread] Failed to retrieve online item information for http://svtplay9d-f.akamaihd.net/i/se/se ... K5kA%3d%3d. It might not play.
java.io.IOException: org.serviio.library.local.metadata.extractor.InvalidMediaFormatException: Unknown video file type.
at org.serviio.library.online.metadata.FeedUpdaterThread.retrieveMetadata(FeedUpdaterThread.java:291)
at org.serviio.library.online.metadata.FeedUpdaterThread.retrieveTechnicalMetadata(FeedUpdaterThread.java:209)
at org.serviio.library.online.metadata.FeedUpdaterThread.run(FeedUpdaterThread.java:102)
Caused by: org.serviio.library.local.metadata.extractor.InvalidMediaFormatException: Unknown video file type.
at org.serviio.library.metadata.FFmpegMetadataRetriever.validateCodecsFound(FFmpegMetadataRetriever.java:382)
at org.serviio.library.metadata.FFmpegMetadataRetriever.retrieveMetadata(FFmpegMetadataRetriever.java:86)
at org.serviio.library.metadata.FFmpegMetadataRetriever.retrieveOnlineMetadata(FFmpegMetadataRetriever.java:124)
at org.serviio.library.online.metadata.FeedUpdaterThread.retrieveMetadata(FeedUpdaterThread.java:268)
... 2 more

The url reported by the FFMPEGWrapper contains commas.
But the url reported by the ProcessExecutor contains escape sequences (url-encoding) for those commas.

If i try to run the unescaped url in VLC, the stream plays fine.

If I try to dump the unescaped url using ffmpeg (latest static build for Windows x64) I get some warnings, but it still parses and dumps the stream:

C:\Users\otto>"C:\Program Files\Serviio\lib\ffmpeg.exe" -i http://svtplay4h-f.ak
amaihd.net/i/world/open/20121115/1308916-064A_ABC/REGIONALA_NYHET-064A-abc-da733
6991ee4e4ce_,900,320,420,620,1660,2760,.mp4.csmil/index_5_av.m3u8?e=b2a7ace1bf66
1c32 d:\tmp\test.mp4
ffmpeg version N-46469-gc995644 Copyright (c) 2000-2012 the FFmpeg developers
built on Nov 5 2012 17:58:15 with gcc 4.7.2 (GCC)
configuration: --enable-gpl --enable-version3 --disable-pthreads --enable-runt
ime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass -
-enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enab
le-libgsm --enable-libmp3lame --enable-libnut --enable-libopenjpeg --enable-libo
pus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheo
ra --enable-libutvideo --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --ena
ble-zlib
libavutil 52. 5.100 / 52. 5.100
libavcodec 54. 71.100 / 54. 71.100
libavformat 54. 36.100 / 54. 36.100
libavdevice 54. 3.100 / 54. 3.100
libavfilter 3. 21.106 / 3. 21.106
libswscale 2. 1.102 / 2. 1.102
libswresample 0. 16.100 / 0. 16.100
libpostproc 52. 1.100 / 52. 1.100
[h264 @ 0000000004850340] non-existing SPS 0 referenced in buffering period
[h264 @ 0000000004850340] non-existing SPS 32 referenced in buffering period
[h264 @ 0000000004850340] non-existing SPS 0 referenced in buffering period
[h264 @ 0000000004850340] non-existing SPS 32 referenced in buffering period
[mpegts @ 0000000001f28380] Estimating duration from bitrate, this may be inaccu
rate
[h264 @ 0000000004d38280] non-existing SPS 0 referenced in buffering period
[h264 @ 0000000004d38280] non-existing SPS 32 referenced in buffering period
Input #0, hls,applehttp, from 'http://svtplay4h-f.akamaihd.net/i/world/open/2012
1115/1308916-064A_ABC/REGIONALA_NYHET-064A-abc-da7336991ee4e4ce_,900,320,420,620
,1660,2760,.mp4.csmil/index_5_av.m3u8?e=b2a7ace1bf661c32':
Duration: 00:10:03.00, start: 0.100667, bitrate: 0 kb/s
Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720
[SAR 1:1 DAR 16:9], 12.50 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:1: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, s16, 160
kb/s
[libx264 @ 0000000004d58c80] using SAR=1/1
[libx264 @ 0000000004d58c80] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShu
ffle SSE4.2
[libx264 @ 0000000004d58c80] profile High, level 3.1
[libx264 @ 0000000004d58c80] 264 - core 128 r2216 198a7ea - H.264/MPEG-4 AVC cod
ec - Copyleft 2003-2012 - http://www.videolan.org/x264.html - options: cabac=1 r
ef=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed
_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pski
p=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 dec
imate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b
_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=
25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.
60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[h264 @ 00000000048d3420] non-existing SPS 0 referenced in buffering period
[h264 @ 00000000048d3420] non-existing SPS 32 referenced in buffering period
Output #0, mp4, to 'd:\tmp\test.mp4':
Metadata:
encoder : Lavf54.36.100
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:
1 DAR 16:9], q=-1--1, 12800 tbn, 25 tbc
Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, s16, 128
kb/s
Stream mapping:
Stream #0:0 -> #0:0 (h264 -> libx264)
Stream #0:1 -> #0:1 (aac -> libvo_aacenc)
Press [q] to stop, [?] for help
frame= 66 fps=0.0 q=28.0 size= 137kB time=00:00:02.95 bitrate= 379.0kbits/
frame= 104 fps=103 q=28.0 size= 324kB time=00:00:04.46 bitrate= 594.3kbits/
frame= 122 fps= 81 q=28.0 size= 581kB time=00:00:05.19 bitrate= 915.9kbits/
frame= 144 fps= 70 q=28.0 size= 847kB time=00:00:06.06 bitrate=1143.1kbits/
frame= 162 fps= 62 q=28.0 size= 1130kB time=00:00:06.77 bitrate=1366.3kbits/
frame= 182 fps= 58 q=28.0 size= 1359kB time=00:00:07.58 bitrate=1468.7kbits/
frame= 208 fps= 56 q=28.0 size= 1711kB time=00:00:08.62 bitrate=1624.2kbits/
frame= 230 fps= 54 q=28.0 size= 1968kB time=00:00:09.52 bitrate=1693.2kbits/
frame= 242 fps= 47 q=28.0 size= 2112kB time=00:00:09.97 bitrate=1735.2kbits/
frame= 262 fps= 47 q=28.0 size= 2438kB time=00:00:10.80 bitrate=1848.3kbits/
frame= 262 fps= 33 q=28.0 size= 2438kB time=00:00:10.82 bitrate=1844.7kbits/
frame= 288 fps= 34 q=28.0 size= 2786kB time=00:00:11.80 bitrate=1933.0kbits/
frame= 304 fps= 31 q=-1.0 Lsize= 3372kB time=00:00:12.48 bitrate=2211.8kbits
/s dup=1 drop=0
video:3167kB audio:196kB subtitle:0 global headers:0kB muxing overhead 0.287847%

[libx264 @ 0000000004d58c80] frame I:5 Avg QP:18.57 size: 31036
[libx264 @ 0000000004d58c80] frame P:178 Avg QP:22.34 size: 14817
[libx264 @ 0000000004d58c80] frame B:121 Avg QP:24.99 size: 3714
[libx264 @ 0000000004d58c80] consecutive B-frames: 35.5% 31.6% 7.9% 25.0%
[libx264 @ 0000000004d58c80] mb I I16..4: 56.4% 24.2% 19.5%
[libx264 @ 0000000004d58c80] mb P I16..4: 28.0% 14.7% 3.1% P16..4: 27.6% 6.8
% 2.2% 0.0% 0.0% skip:17.6%
[libx264 @ 0000000004d58c80] mb B I16..4: 3.0% 2.2% 0.2% B16..8: 28.7% 2.0
% 0.3% direct: 3.7% skip:60.0% L0:40.5% L1:54.4% BI: 5.0%
[libx264 @ 0000000004d58c80] 8x8 transform intra:32.3% inter:68.9%
[libx264 @ 0000000004d58c80] coded y,uvDC,uvAC intra: 22.1% 57.9% 6.8% inter: 12
.3% 27.7% 1.0%
[libx264 @ 0000000004d58c80] i16 v,h,dc,p: 32% 28% 15% 25%
[libx264 @ 0000000004d58c80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 25% 33% 3% 2%
2% 2% 3% 4%
[libx264 @ 0000000004d58c80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 21% 28% 5% 4%
4% 3% 4% 4%
[libx264 @ 0000000004d58c80] i8c dc,h,v,p: 44% 27% 20% 10%
[libx264 @ 0000000004d58c80] Weighted P-Frames: Y:13.5% UV:12.9%
[libx264 @ 0000000004d58c80] ref P L0: 71.1% 13.7% 12.2% 2.8% 0.1%
[libx264 @ 0000000004d58c80] ref B L0: 86.9% 11.8% 1.3%
[libx264 @ 0000000004d58c80] ref B L1: 95.5% 4.5%
[libx264 @ 0000000004d58c80] kb/s:2132.91

If I try using the escaped url, ffmpeg fails (gets a http 404):

C:\Users\otto>"C:\Program Files\Serviio\lib\ffmpeg.exe" -i http://svtplay9d-f.ak
amaihd.net/i/se/secure/20121104/1128572-005A/ADRIAAN-005A-e2c103ba629791e0_%2c90
0%2c320%2c420%2c620%2c1660%2c2760%2c.mp4.csmil/index_5_av.m3u8?e=b2a7ace1bf661c3
2&id=AgBV5iXiXCEhlAh8pVD7zea+31zF8dplSYJPqHsPWBkpDQ8JIFGjKsMNAneJn%2f9bReMdvoGqn
bK5kA%3d%3d
ffmpeg version N-46469-gc995644 Copyright (c) 2000-2012 the FFmpeg developers
built on Nov 5 2012 17:58:15 with gcc 4.7.2 (GCC)
configuration: --enable-gpl --enable-version3 --disable-pthreads --enable-runt
ime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass -
-enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enab
le-libgsm --enable-libmp3lame --enable-libnut --enable-libopenjpeg --enable-libo
pus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheo
ra --enable-libutvideo --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --ena
ble-zlib
libavutil 52. 5.100 / 52. 5.100
libavcodec 54. 71.100 / 54. 71.100
libavformat 54. 36.100 / 54. 36.100
libavdevice 54. 3.100 / 54. 3.100
libavfilter 3. 21.106 / 3. 21.106
libswscale 2. 1.102 / 2. 1.102
libswresample 0. 16.100 / 0. 16.100
libpostproc 52. 1.100 / 52. 1.100
[http @ 00000000020dae80] HTTP error 404 Not Found
http://svtplay9d-f.akamaihd.net/i/se/se ... AAN-005A-e
2c103ba629791e0_%2c900%2c320%2c420%2c620%2c1660%2c2760%2c.mp4.csmil/index_5_av.m
3u8?e=b2a7ace1bf661c32: Input/output error
id är inte ett internt kommando, externt kommando,
program eller kommandofil.

So, my guess is that one of the components FFMPEGWrapper or ProcessExecutor erroneously url-encodes the url before passing it to FFMPeg.

[Illico]

Suggest this command line:

ffmpeg.exe" -i "http://svtplay4h-f.akamaihd.net/i/world/open/20121115/1308916-064A_ABC/REGIONALA_NYHET-064A-abc-da7336991ee4e4ce_,900,320,420,620,1660,2760,.mp4.csmil/index_5_av.m3u8?e=b2a7ace1bf661c32" -y -copyts -c:v copy -c:a copy -f mpegts C:\Windows\TEMP\Serviio\TEST.ts

[ottomatic]

Well, to clarify:

It's not the exact command line switches I'm after. I'm trying to report what I perceive to be a bug in Serviio, namely:

* My plugin delivers the correct url in the contaier.
* When serviio attempts to find out the media at the other end of that url, the url appears to have been mangled

Thus, Serviio does not show any items for the online feed, although they obviously exist and can be dumped using ffmpeg and the url which my plugin reported.

[zip]

I'll look into this. There is code to manage commas in URLs, will try it with your example.

[ottomatic]

I'll look into this. There is code to manage commas in URLs, will try it with your example.

Zip: great.
The report got a little confused by the forum board mangling the urls as well

Basically, the FFMPEGWrapper reports the url suggested by Illico, whereas the ProcessExecutor reports the url with all the commas replaced with %2c.

I can't be certain that the mangled url is used in the media retrieval; it might just be mangled in the log entry.

If it's just a log entry bug, I suppose Serviio can sniff the stream but does not recognise "hls,applehttp" as a media container type. Which if that is the case it needs to start supporting.

[zip]

fixed for 1.1

[ottomatic]

fixed for 1.1

Wonderful. Is there a scheduled release date?

[zip]

No Aiming for before Christmas.