FAQ  •  Register  •  Login

Serviio Web Stream Transcoding 100% CPU

<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Thu Feb 15, 2018 4:07 pm

Serviio Web Stream Transcoding 100% CPU

Hello to All,
I have spend quite a some time searching for solutions, but to no avail.
My problem(s):
Serviio v1.9.0 (did not try any earlier version or beta)
Tryingt to stream web stream segmented (HLS), HD, audio aac and video h264 (High) 1920x1080, and stream it to WDTV profile.
WDTV by itself supports ts and aac without any issues (no transcoding needed).
running command ffmpeg -i http://STREAM_TO_WATCH I am getting all this info, also duration of the stream: Duration: 00:MM:SS.00
No Internet problems.
I also tried to web stream locally on my 1Gb LAN with the same issues and results.
However when trying to watch the stream, since for web streams ffmpeg transcoding is build-in into the serviio code (according to ZIP)), my laptop CPU goes to 100% while transcoding, and shortly after the stream begins, it just stops or looses audio or entirely dies because of CPU usage.
There is no problem if I transmux the segmented stream into an entire .ts file (no more transcoding)
Under serviio I have transcoding disabled. I tried to disable it in WDTV profile, but since for web streams it is build into serviio code, this overtakes and forcefully transcodes the stream, forcing CPU into 100% and dying video stream.
I have also tried to run ffmpeg command but I never succeeded to send the stream to WDTV in a serviio style (or in any ways).
I was able to save it (the stream segments) to a file, but this is not what I do want to do.
Maybe anyone could give me a constructive piece of advice how could I stop serviio transcoding entirely, even web streams, or possibly what command to issue to ffmpeg to send this http:// stream to WDTV?
In detailed serviio logs I do see FFMPEG WRAPPER starting the stream, but there is no command part showing redirection to WDTV.
Also, is there any way to pause or froward stream segments since ffmpeg is aware of the stream length (yet serviio displays it incorrectly)?
Now I only see 00:18:18 as the length of any web stream.
Once again, any constructive advice will be greatly appreciated and welcomed.
Thanks.
P-B
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Sat Feb 17, 2018 4:56 pm

Re: Serviio Web Stream Transcoding 100% CPU

Bump :(
No one? No hints or suggestions? :?:
P-B
<<

DenyAll

DLNA master

Posts: 2257

Joined: Fri Mar 08, 2013 11:16 pm

Location: Adelaide, Australia

Post Sun Feb 18, 2018 5:29 am

Re: Serviio Web Stream Transcoding 100% CPU

If you set debug logging on you will see the ffmpeg command that is being executed. You can then run that from a command line and see what ffmpeg says is happening. May provide some hints.

You can also try playing with the profile:

  • Edit config/profiles.xml using your favourite text editor (Notepad++ recommended).
  • Find the following within the "WDTV Live" profile:
      Code:
    <OnlineTranscoding>
       <Video targetContainer="mpeg" targetVCodec="mpeg2video" targetACodec="ac3" aBitrate="384">
          <Matches container="mp4" />
          <Matches container="asf" /> <!-- for mms:// -->
       </Video>
    </OnlineTranscoding>
  • Change the first line to:
    <Video targetContainer="mpeg" targetACodec="ac3" aBitrate="384">
  • Save the file and restart your PC
This will remux the video rather than transcoding it. No idea if this will work. Try targetContainer="mpegts" as well.
DenyAll
Panasonic Viera FX800A | Panasonic Viera CS610A | Sony PS4 | Sony PS3 | Panasonic DMP-BD79 | Yamaha RX-V500D | iPad | Windows 10 | Serviio 1.10.1 Pro
WinHelper | MediaInfo

Beta Tester, Moderator
Please do not PM me for support as any solution cannot be shared with others.
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Sat Feb 24, 2018 11:41 am

Re: Serviio Web Stream Transcoding 100% CPU

Hello again,
@DenyAll---Thanks for your advice, however this does not solve the problem.
Let me reiterate it again:
1. Serviio works fine in general, however,
2. when it invokes ffmpeg on m3u8 web live streams, ffmpeg during transcoding/remuxing uses 100% CPU of my machine and ultimately videos die (stutter, loose audio, etc, they are just simply not watchable)
3. Changing profile for WDTV (my renderer) as DenyAll suggested, does not make any difference since Serviio overrides profile settings while live web streaming.
4. I tried to stream simple .ts file and there is no transcoding/remuxing, but .m3u8 "playlist" files are always transcoded/remuxed.
5. When I try commandline ffmpeg: ffmpeg -i http://MY_STREAMED_URL.m3u8 -c copy -f h.264 udp://localhost:1234, I can stream it to vlc without any problems
6. when I try 5. to pipe: to any process (| vlc or java) ffmpeg sends it somewhere, but I cannot find it or play it. (I also tried to invoke ffserver, but this one is becoming deprecated, and has issues with mp4 files, among other issues).
7. I wish, I could know how to pipe/redirect ffmpeg stream to serviio's java process to override serviio's forceful transcoding/remuxing of web live m3u8 streams.
8. Or maybe there is a hidden door to completely disable serviio's forceful transcoding/remuxing of web live m3u8 streams?
I tried PM ZIP for any hints, however the board will not let me (will not allow me) do it, since I am too new member.
Any suggestions in this matter?
Sincerely,
P-B
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Sat Feb 24, 2018 3:10 pm

Re: Serviio Web Stream Transcoding 100% CPU

try to add this as another entry to the WDTV profile's Online Transcoding:

  Code:
<Video targetContainer="mpegts">
  <Matches container="applehttp"/>
</Video>
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Sun Feb 25, 2018 5:10 pm

Re: Serviio Web Stream Transcoding 100% CPU

Serviio_Logs_and_Info.zip
(142.13 KiB) Downloaded 244 times
Hello Again,
Thank you Zip for responding to my call for help.
Also, again thanks to DenyAll for some hints.
I did some research and attaching zipped serviio logs (in DEBUG mode) and additional info which might be useful for debugging.
But making long story short Zips hint:
zip wrote:try to add this as another entry to the WDTV profile's Online Transcoding:

  Code:
<Video targetContainer="mpegts">
  <Matches container="applehttp"/>
</Video>

or DenyAll hint:
DenyAll wrote:If you set debug logging on you will see the ffmpeg command that is being executed. You can then run that from a command line and see what ffmpeg says is happening. May provide some hints.

You can also try playing with the profile:

  • Edit config/profiles.xml using your favourite text editor (Notepad++ recommended).
  • Find the following within the "WDTV Live" profile:
      Code:
    <OnlineTranscoding>
       <Video targetContainer="mpeg" targetVCodec="mpeg2video" targetACodec="ac3" aBitrate="384">
          <Matches container="mp4" />
          <Matches container="asf" /> <!-- for mms:// -->
       </Video>
    </OnlineTranscoding>
  • Change the first line to:
    <Video targetContainer="mpeg" targetACodec="ac3" aBitrate="384">
  • Save the file and restart your PC
This will remux the video rather than transcoding it. No idea if this will work. Try targetContainer="mpegts" as well.

were only partially helpful.
applehttp container seems to be incompatible with WDTV: I could not open any live web streams.
I ended up with mpegts, which is not perfect but seems to cycle CPU more frequently, although still well above 90% usage.
I have tested streams with 1920x1080, 1280x720, and 960x540 video resolutions (should be included in serviio logs).
I also tested local (python http invoking command; I did not feel like involving Apache server into the puzzle), and only plain .ts file seems not to transcode/transmux. If m3u8 playlists are involved, ffmpeg "lays its hands" on them and CPU goes way up.
Well, anyway, please see attached zipped logs and other info, and if you would need any additional info, please, do not hesitate to contact me.
BTW streams urls in logs are geo-localized and will not play in other geo-locations.
Sincerely,
P-B
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Mon Feb 26, 2018 8:59 am

Re: Serviio Web Stream Transcoding 100% CPU

how does your wdtv profile look like after my suggestion? and did you restart serviio serivce / pc after you made that change
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Tue Feb 27, 2018 10:55 am

Re: Serviio Web Stream Transcoding 100% CPU

Hello Zip,
I just appended the selection you indicated:

<Video targetContainer="mpegts">
<Matches container="applehttp"/>
</Video>

to original profoles.xml file:
to the front of the WDTV Live section:

Sections BEFORE this part were left UNCHANGED (NOT MODIFIED)
</Transcoding>
<OnlineTranscoding>
<Video targetContainer="mpeg" targetVCodec="mpeg2video" targetACodec="ac3" aBitrate="384">
<Matches container="mp4" />
<Matches container="asf" /> <!-- for mms:// -->
</Video>
</OnlineTranscoding>
<AudioTrackRemux> Following Sections FURTHER Down were left UNCHANGED (NOT MODIFIED)

So ultimately it looked this way:

BEFORE UNCHANGED (NOT MODIFIED)
</Transcoding>
<OnlineTranscoding>
<Video targetContainer="mpegts">
<Matches container="applehttp"/>
</Video>
<Video targetContainer="mpeg" targetVCodec="mpeg2video" targetACodec="ac3" aBitrate="384">
<Matches container="mp4" />
<Matches container="asf" /> <!-- for mms:// -->
</Video>
</OnlineTranscoding>
<AudioTrackRemux> FURTHER UNCHANGED (NOT MODIFIED)

So, the profile looked as the original file with the insert in the WDTV Live OnlineTranscoding section. I did not remove (delete) anything, any other parts from the original profiles.xml file.

Also, before I made any changes to serviio configuration I did STOP the service:
/opt/serviio/bin/serviio.sh -stop (java process disappeared from processes list; serviio was not on media servers list)
and after configuration modifications I did restart serviio:
/opt/serviio/bin/serviio.sh -start (java process re-appeared on processes list; serviio appeared again on media servers list)

On occasion I would reboot the system entirely, but not every time I did made changes to serviio configurations.
FYI: This is Ubuntu 17.10 x64 (not Windows), so I did not think it was necessary to reboot ever time after configuration changes, just stopping and re-starting the service/process was sufficient enough (but maybe this was wrong?).
Sincerely,
P-B
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Tue Feb 27, 2018 1:30 pm

Re: Serviio Web Stream Transcoding 100% CPU

can you send the detailed log again, trying to play that stream with this updated profile?
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Tue Feb 27, 2018 2:45 pm

Re: Serviio Web Stream Transcoding 100% CPU

Hello Zip,
the attempt to use your suggested profile additions:
<Video targetContainer="mpegts">
<Matches container="applehttp"/>
</Video>
were logged and are in the attached files.
Beeing more specific:
Serviio.log.1 (3 hits)
Line 1383: [hls,applehttp @ 0x557294368c20] Opening 'https://sdt-plix35-12.tvp.pl/token/video/vod/35924191/20180225/3240244527/619e41f0-1299-4b5d-87bb-844c4af9fb06/video.ism/video-audio=173000-video=1991000-33.ts' for reading
Line 2715: [hls,applehttp @ 0x558432f6dc20] Opening 'https://sdt-plix35-12.tvp.pl/token/video/vod/35924191/20180225/3240244527/619e41f0-1299-4b5d-87bb-844c4af9fb06/video.ism/video-audio=173000-video=4785000-18.ts' for reading
Line 2879: [hls,applehttp @ 0x5588459fbc20] Opening 'https://sdt-plix35-12.tvp.pl/token/video/vod/35924191/20180225/3240244527/619e41f0-1299-4b5d-87bb-844c4af9fb06/video.ism/video-audio=173000-video=3280000-27.ts' for reading
Serviio.log.2 (2 hits)
Line 1528: [hls,applehttp @ 0x5604ed123c20] Opening 'https://sdt-plix35-12.tvp.pl/token/video/vod/35924191/20180225/3240244527/619e41f0-1299-4b5d-87bb-844c4af9fb06/video.ism/video-audio=173000-video=3280000-27.ts' for reading
Line 1694: [hls,applehttp @ 0x5588dba51c20] Opening 'https://sdt-plix35-12.tvp.pl/token/video/vod/35924191/20180225/3240244527/619e41f0-1299-4b5d-87bb-844c4af9fb06/video.ism/video-audio=173000-video=1991000-35.ts' for reading

But before these line I can see some kind of ffmpeg error:
2018-02-25 16:40:22,415 DEBUG [TimeoutStreamDelegator] Closing piped input stream and closing related feeder process
2018-02-25 16:40:22,415 DEBUG [ProcessExecutor] Stopping external process: Thread[Thread-102,5,main]
2018-02-25 16:40:22,415 DEBUG [ProcessUtils] Sending kill -9 to the Unix process: 23270
2018-02-25 16:40:22,415 DEBUG [ProcessExecutor] Starting kill -9 23270
2018-02-25 16:40:22,426 WARN [ProcessExecutor] Process ffmpeg has a return code of 137! This is a possible error. Detailed output follows.
frame= 3099 fps= 71 q=1.0 size= 57236kB time=00:02:03.98 bitrate=3781.6kbits/s dup=2 drop=0 speed=2.82x
frame= 3143 fps= 71 q=1.0 size= 57780kB time=00:02:05.69 bitrate=3765.7kbits/s dup=2 drop=0 speed=2.83x

In every listed above instance of applehttp container, such ffmpeg 137! error shows up.
Not sure if this is what you are looking for.
Please, advice what others steps should I undertake, to provide better logs and possible resolution to this puzzle.

As a side note, would it be possible to add me to serviio beta-testers group, if I could possibly contribute with something to this project, I will gladly do it.

Thank You. Very Sincerely,
P-B
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Wed Feb 28, 2018 9:32 am

Re: Serviio Web Stream Transcoding 100% CPU

There is something weird. First, it uses the wrong transcoding profile to what I'd expect, and second the WDTV sends a bye-bye message after a while and that kills the transcoding job (or more likely the job is killed after 20 seconds of inactivity on the device part).

Did you turn off the device during streaming or something?
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Wed Feb 28, 2018 4:57 pm

Re: Serviio Web Stream Transcoding 100% CPU

Hello Zip,
please see attached more serviio logs.
Just to clarify previous logs contained entire profile without any deletions, just suggested additions and modifications.
And after the video started streaming (or not) I would terminate serviio to proceed to another test option, that is why they were terminating after 20 secs.
This time I did as follows:
I edited profiles.xml file in such a way, that I left only default DLNA ID=1 profile and WDTV (since without profile 1 serviio would not start).
OK, so, now 1st log it shows original profile (WDTV only, but no changes to profile), and video stream starts. I ran it about 20-30 secs, and proceeded to the next option (after stopping serviio, emptying log file, making modifications to profiles file, and re-starting serviio).
Profile and log 2 contained only your suggested modifications: applehttp (exactly as indicated in previous posts, replacing original entries of WDTV section). It was not possible to play video with WDTV error that it was unknown (non-compatible) file format.
Profile and log 3 I replaced applehttp with "*" (asterisk). Results as with applehttp-No video play.
And finally I did modified original WDTV profile, it would be:
<OnlineTranscoding>
<Video targetContainer="mpegts">
<Matches container="mp4" />
<Matches container="asf" /> <!-- for mms:// -->
</Video>
</OnlineTranscoding>
<AudioTrackRemux>.......
and video stream played again, of course transcoded.
If You need more logs and info, please do not hesitate to contact me.
Sincerely,
P-B
More_Serviio_logs.zip
(61.61 KiB) Downloaded 245 times
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Wed Feb 28, 2018 8:01 pm

Re: Serviio Web Stream Transcoding 100% CPU

so it looks like with log2, it's what I was trying to do - transcode HLS to mpegts (byt with the audio and video streams left untouched, thus using very little CPU).
But it looks like the device doesn't like this format.

Serviio doesn't support passing through HLS streams, without touching them somehow.
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Wed Feb 28, 2018 10:07 pm

Re: Serviio Web Stream Transcoding 100% CPU

Hi Zip,
thanks a lot for your assistance regarding this issue.
So, I guess I will have to live with the current situation and just hope for future developments.
(Unfortunately WDTV is out of support; EOL=End_of_Life, so this side of equation will not change). Maybe some other future developments will bring a viable solution to the issue, lets hope.
Or maybe there will be a way to pipe ffmpeg i/o to serviio any other way (eg from cli?) to pass it to WDTV with ffmpeg directives and without serviio's ffmpeg transcoding attempts.
But, once again, truly thank you for your time and assistance.
Very Sincerely,
P-B
<<

freaknik

User avatar

DLNA master

Posts: 345

Joined: Thu Mar 27, 2014 2:05 pm

Location: Endor

Post Thu Mar 01, 2018 12:52 pm

Re: Serviio Web Stream Transcoding 100% CPU

Dude it might be worth it to state what your usual bandwidth is since what you're watching is online? That's why it doesn't work for me.
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Sat Mar 03, 2018 9:54 am

Re: Serviio Web Stream Transcoding 100% CPU

Hello again,
Just to let you know I think I have found a temp solution to my problem. :D
I have scanned all available serviio profiles for different devices and assigned one-by-one all of them to my WDTV SMP box.
I have found that 3 profiles would play web live streams of my desire with minimal ffmpeg involvement (almost none; 0-1% CPU).
Roku 4K TV, Roku TV (non 4K), and Sony BDP-Sx80/SMP-N200 (2011) would produce similar (not exact: -map 0:X, X would vary) ffmpeg command execution:
ffmpeg -fflags +genpts -threads 0 -analyzeduration 10000000 -i http://MY_WEB_LIVE_STREAM.m3u8 -y -threads 0 -c:v copy -c:a:0 copy -map 0:1 -map 0:0 -sn -f mpegts pipe:
The web streams startup takes a longer while up to 1-2 mins to start playing, and sometimes they abort to play, but at some point they do play.
I have noticed that Roku devices are referencing some other Roku profiles. Similar with Sony.
So, now I have to sift through Roku profiles to figure out what parts are relevant to succeed with such ffmpeg behaviour.
Debug logs reffer to something like this: AVC_TS_MP_HD_AAC_MULT5_ISO as successful format.
This item is included in "Generic DLNA profile", however WDTV and most of other profiles seem to ignore it.
It must be something additional in Roku 4K profile that produces such an output. Even Roku Media Player, which profile is referenced in Roku 4K does not produce proper output.
Since I am not good with codecs and containers, it might take me a while to sift through and sort it out, but as soon as I figure it out (I hope so), I will report it here for others to make use of it whenever needed.
Sincerely,
P-B
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Mon Mar 05, 2018 4:16 pm

Re: Serviio Web Stream Transcoding 100% CPU

The same media profile is used when you added the applehttp transocidng to wdtv profile (log 2 from your last attachment).

  Code:
2018-02-28 17:01:08,901 DEBUG [VideoDeliveryEngine] Found Format profile for transcoded file "https://sdt-plix43-43.tvp.pl/token/video/vod/36027238/20180228/3240244527/149e597b-5a99-4bae-a54d-3b7b2ccd879a/video.ism/video-audio%3D175000-video%3D4766000.m3u8": AVC_TS_MP_HD_AAC_MULT5_ISO


Only there is no ffmpeg invoked then, and you stop the server. So maybe it was the case when it sometimes starts, and sometimes it does not.
<<

Pandora-Box

Serviio newbie

Posts: 14

Joined: Tue Feb 06, 2018 3:33 pm

Post Wed Mar 07, 2018 6:15 pm

Re: Serviio Web Stream Transcoding 100% CPU

Hello Zip,
Not sure what to say.
When I used the profile attached to this log 2 file, I got response from my WDTV that it was unsupported file, and it would not want to play it. It gives the only option to dismiss this info screen and try again.
Why nothing showed up in the log file, I do not know. Logging was set up to DEBUG.
After 4-5 attempts to play the file and the same response from WDTV: Unsupported file and no way to get it to play, I just stopped serviio server and moved on to the new configuration with the new empty log file to log other profile debug info.
I have also noticed some other strange behavior while trying to play web live streams playlists and .ts files, which I will try to describe as another topic in this forum, but, this is not only related to WDTV, but also to other renderers, like Android VLC player.
Sincerely,
P-B

Return to Serviio Support & Help

Who is online

Users browsing this forum: No registered users and 53 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.