FAQ  •  Register  •  Login

Transcoding Load Balancer

<<

y3ti

Serviio newbie

Posts: 7

Joined: Fri Jul 06, 2018 10:57 am

Post Mon Jun 21, 2021 3:39 pm

Transcoding Load Balancer

So ive been having some issues with my little old pc running a number of streams for my network to all my different TV's

So for example im finding that if i have 5 TV's in the house all streaming i sometimes get lag before the FFMPEG will start, and becuase of the number of exe's running the PC just gets completely locked up running too much.

With that in mind, introducing my own work around.

FFMPEG Load Balancer
Image

This basically has a small client running on the machine you wish to use for Transcoding
And a Small FFMPEG replacement file for the Serviio Installation, that will forward the FFMPEG request into the Load Balancer

Things Still Left To Finish
  • Add function for Hardware Accelerated transcode for GPUs that are supported (CUDA based cards - so NVIDIA)
  • Tidy Up Code
  • Re-Write Timeout function for auto reconnect if link drops out
  • Make sure the FFMPEG forwarder will select the least used Node to start Transcode on
  • Documentation
Last edited by y3ti on Tue Jun 22, 2021 11:42 am, edited 2 times in total.
Clifford

Main Serviio Server: Intel i5 4590T - 8GB Ram (Windows 10 Pro)
Node1: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node2: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node3: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
<<

atc98092

User avatar

DLNA master

Posts: 4509

Joined: Fri Aug 17, 2012 10:22 pm

Location: Washington (the state)

Post Mon Jun 21, 2021 4:04 pm

Re: Transcoding Load Balancer

I think I understand what your intent is, but I'm not completely following how you're doing this. Are you installing this program on multiple computers, and dispersing the FFMPEG task to them as the load increases? I can see a benefit of reducing the load on the Serviio computer, but I also see the potential of saturating your network with these streams across multiple computers. You better be running Gigabit!
Dan

LG NANO85 4K TV, Samsung JU7100 4K TV, Sony BDP-S3500, Sharp 4K Roku TV, Insignia Roku TV, Roku Ultra, Premiere and Stick, Nvidia Shield, Yamaha RX-V583 AVR.
Primary server: Intel i5-6400, 8 gig ram, Windows 10 Pro, 22 TB hard drive space | Test server Windows 10 Pro, AMD Phenom II X4 965, 8 gig ram

HOWTO: Enable debug logging HOWTO: Identify media file contents
<<

y3ti

Serviio newbie

Posts: 7

Joined: Fri Jul 06, 2018 10:57 am

Post Mon Jun 21, 2021 4:10 pm

Re: Transcoding Load Balancer

So i have a small setup of 3 PCs on a internal switch separate from my main network and over a second network card on the PCs (all gbit)

I then have a Shared Folder with a Simple Client and a Copy of the FFMPEG that Serviio uses. Simply run the Client on each "End Node", which in turn connects to the server app on the Serviio Server
As soon as a request for FFMPEG to transcode, the command and everything is forwarded into the Load Balancer and then passed onto a End Node

End Node has a Symlinked Folder to the Serviio Transcode Folder that Serviio has in its settings, thus making Serviio think that the transcode is being done locally.
This allows me to spread the load of the work of number of streams that are running across my network
Clifford

Main Serviio Server: Intel i5 4590T - 8GB Ram (Windows 10 Pro)
Node1: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node2: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node3: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
<<

atc98092

User avatar

DLNA master

Posts: 4509

Joined: Fri Aug 17, 2012 10:22 pm

Location: Washington (the state)

Post Mon Jun 21, 2021 6:48 pm

Re: Transcoding Load Balancer

OK, I'm following you now. Interesting setup. It's beyond the abilities of most standard computer users, but I can get my geekiness behind it. For myself, most of my local playback is done using an Nvidia Shield, which can play everything I have without transcoding. And for my Roku devices, it's mostly just audio if something needs transcoding, and that's a far lighter load on the CPU. But I applaud your ingenuity! :D

Are you going to make this available once you've tidied up the code? If so, I also suggest some detailed discussion on the networking configuration you're doing between the nodes.
Dan

LG NANO85 4K TV, Samsung JU7100 4K TV, Sony BDP-S3500, Sharp 4K Roku TV, Insignia Roku TV, Roku Ultra, Premiere and Stick, Nvidia Shield, Yamaha RX-V583 AVR.
Primary server: Intel i5-6400, 8 gig ram, Windows 10 Pro, 22 TB hard drive space | Test server Windows 10 Pro, AMD Phenom II X4 965, 8 gig ram

HOWTO: Enable debug logging HOWTO: Identify media file contents
<<

y3ti

Serviio newbie

Posts: 7

Joined: Fri Jul 06, 2018 10:57 am

Post Tue Jun 22, 2021 11:25 am

Re: Transcoding Load Balancer

Im certainly thinking of releasing the code once completed, especially for anyone who wants to run alot of TV's/Screens in their home all streaming at the same time

Setup as follows
Main Serviio Server: Intel i5 4590T - 8GB Ram (Windows 10 Pro)
Node1: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node2: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node3: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)

All Nodes run a simple Console Application that receives the TCPIP Packet (Websocket within C#) and initiates the FFMPEG to transcode on the given node. The Node transcodes directly into a shared folder that matches the folder on the Main Serviio Server

D:\Serviio\Transcode = This is the folder setup in Serviio for transcoding
Node1/2/3 all have a Symlink for D:\Serviio\Transcode on their machines, that goes over a Internal 1Gbit network separate from the main network

Once the transcode is finished, a TCPIP Packet (Websocket) gets sent back to the Load Balancer to tell it that the transcode has been completed

Image

In the image you can see the client saying TC:0
TC stands for Transcode and 0 is the number running, this is reported back to the server on a 10 second loop (near instant)
Image

After adding some extra tweaks for allowing Hardware Accelerated Graphics Transcoding (Non CUDA Cards). Below are some screenshots of it running and how its performing
Image
Image
Clifford

Main Serviio Server: Intel i5 4590T - 8GB Ram (Windows 10 Pro)
Node1: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node2: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
Node3: Intel i3 3220T - 8GB Ram - Dual network card (Windows 10 Pro)
<<

atc98092

User avatar

DLNA master

Posts: 4509

Joined: Fri Aug 17, 2012 10:22 pm

Location: Washington (the state)

Post Tue Jun 22, 2021 3:01 pm

Re: Transcoding Load Balancer

Nice work!
Dan

LG NANO85 4K TV, Samsung JU7100 4K TV, Sony BDP-S3500, Sharp 4K Roku TV, Insignia Roku TV, Roku Ultra, Premiere and Stick, Nvidia Shield, Yamaha RX-V583 AVR.
Primary server: Intel i5-6400, 8 gig ram, Windows 10 Pro, 22 TB hard drive space | Test server Windows 10 Pro, AMD Phenom II X4 965, 8 gig ram

HOWTO: Enable debug logging HOWTO: Identify media file contents

Return to Third-party tools integration

Who is online

Users browsing this forum: No registered users and 4 guests

cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.