Proposed changes in Managing On-line Resources

[jhb50]

Proposed changes in Managing On-line Resources
=========================================
My understanding is that in order to play on-line resources via DLNA, for those feeds declared as On-line Streams, RSS-Atoms, and Web Resources in the console, Serviio must determine the attributes of and optionally, the thumbnail for, each individual feed item (stream) in order to be displayed in the DLNA menu on the client device.

Hence, when a feed is added or enabled, Serviio must first build a list of the feed item (stream) url's. For on-line streams this is the single url declared in the console. For RSS-atoms and Web Resources, Serviio first matches the declared feed url against the users installed plug-ins. If an RSS-atom feed url matches a plug-in then the Serviio accesses the feed url to obtain the map of links that must be provided by the feed url, and Serviio then executes the plug-in to extract the feed item (stream) url's from those links. If an RSS-atom does not match a plug-in, Serviio accesses the feed url and extracts the list of feed item (stream) url's contained in the RSS feed url. If a Web Resource feed url matches a plug-in, the plug-in is executed to first build a map of feed item links and then executed to extract the feed item (stream) url's from those links.

Once Serviio has obtained a list of the feed item (stream) url's, it then checks to see if the attributes of those url's have been previously stored in its cache (database). If not, then ffmpeg is executed to obtain the feed item attributes and store them in the cache.

Since the feed items (streams) provided from many sources frequently change, Serviio establishes an expiry date for each feed, being the lesser of the console expiry interval or the feed items expiry defined by the plug-in. When that expiry interval is reached, Serviio will refresh the feed, by repeating the above sequence. Refreshes can also be individually forced for any enabled feed and each enabled feed will be automatically refreshed when Serviio starts.

Once a feed has been processed, its feed items extracted, and their attributes cached, Serviio has the information required to provide any client device with the names of the available feeds., a list of the feed items, and based on the client profile, the format that will be sent to the client for each item. The client user may then select a feed item and the client will request that feed item from Serviio if it is in an acceptable format for the client. Serviio will then send the item in native format or transcode it in real time to the format determined by the client profile. If Serviio is unable to access the online feed item and the item has been defined as "expires immediately" by a plugin, Serviio will again attempt to extract the feed item (stream) url for that link, and obtain the latest valid url in order to access and send the item to the client.

This is an excellent design for managing online resources, and for on-line streams with the immediate availability of the feed item url, or for valid rss links with the availability of all links with the single access of the rss url the building and refresh of feed item url's is near instantaneous.

A problem arises however with those feed items identified by plugins because the extract typically requires multiple url accesses and significant processing to determine each feed item url, which can require many seconds for each feed, and with many enabled feeds and periodic refreshes will utilize considerable resources and delay the availability of feed items during these extended duration refreshes.

The problem is that the current refresh logic involves not only the extract of any new feed item urls, but also the multisecond re-extract of former items to re-obtain their feed item urls, even though in most cases the feed item url is unchanged. The argument for doing this is that the former feed item url may contain a time sensitive key which will be updated by the reextract. The counter argument is that those urls will already have been defined as "expires immediately" and so the re-extract to obtain the new key will automatically occur if required when the original url is accessed, and will likely occur anyway even if a prior forced refresh has occured.

The trade off is obvious: the periodic multisecond reextract of all old urls in all enabled feeds regardless of whether they are ever played versus the single
reextract of a single feed item if the item is ever played. Consequentally the refresh of a stable feed with multiple of feed items will take less than 30 seconds to confirm the stable items rather than many minutes to also reextract the same feed items that Serviio already has. Similarly refreshes will become largely transparent to the user, regardless of the number of enabled feeds, whereas today the user must be cognizant of his feeds and selectively enable them as required in order to minimize the refresh load.

This request therefore is to only extract new feed items on the refresh of any given feed, just as we only invoke ffmpeg for the metadata on new feed items.

[zip]

That is a perfect description of the design. Might even make it t the wiki, if it's not there already

The counter argument is that those urls will already have been defined as "expires immediately" and so the re-extract to obtain the new key will automatically occur if required when the original url is accessed, and will likely occur anyway even if a prior forced refresh has occured.

Not always. In the example of iPlayer, it doesn't expire immediately, but there is expiry date. So the URL/feed item is valid for a certain period of time, rather than having to be extracted on every play.

Let's talk about this again after 1.1.

The trade off is obvious: the periodic multisecond reextract of all old urls in all enabled feeds regardless of whether they are ever played versus the single
reextract of a single feed item if the item is ever played. Consequentally the refresh of a stable feed with multiple of feed items will take less than 30 seconds to confirm the stable items rather than many minutes to also reextract the same feed items that Serviio already has.

and may be other option could be to avoid any caching at all and display online resources on fly when online resource folder is selected in menu. that will provide many benefits:

1. content will always be fresh, no refresh ever needed
2. support for huge online resources with multilevel sub-folders (fresh such resource would take forever in current design)
3. no computer resources spent to refresh feeds which were not viewed
4. feed folder can be shown quickly on TV or BD player because it only needs 10 or so first items be displayed when menu item is selected

maybe it's possible to even avoid ffmpeg call when displaying folder content and call ffmpeg only when online stream is selected. this way output format for items in resource folder can initially be assigned to some default format from profile.

[jhb50]

miksa: You missed the point that Serviio must provide a DLNA client with the attributes of the feed items when it displays a feed menu. The only way it can do that is to have pre-extracted the items and run ffmpeg to cache those values, hence that must happen at least once. My proposal is to use that initial set of data and only refresh it if the initial feed item data fails to work.

zip: Thanks for being open to change.

Regarding iplayer, the same is true of YouTube, however it seems to me that there would be no downside to always treating online resources as "expires immediately" by default, which would ensure that item refreshes would be done at playback time if required. This would allow all feeds including iplayer and youtube to be enabled, and periodically refreshed to include new items and drop old ones, without the need to re-extract previous but still active items.

This changes the focus of feed refreshes to one of maintaining a current set of items rather than on maintaining a current set of item url's and attributes, and has the added benefit of making it practical to refresh feeds that are constantly changing like sports and news on a more frequent basis since only new items will be extracted.

I'll put a pointer to this thread in bitbucket.

[zip]

always treating online resources as "expires immediately" by default, which would ensure that item refreshes would be done at playback time if required. This would allow all feeds including iplayer and youtube to be enabled, and periodically refreshed to include new items and drop old ones, without the need to re-extract previous but still active items.

expiresImmediately by default would mean every time you hit play it'd have to run the plugin, causing delay. Might be ok for some plugins, not so ok for others. The iPlayer example is doubly interesting, as not only the URL expires, but also the item itself might expire (when BBC take it down from iPlayer) - in this case the user would still see the programme, but would not be able to play it, which is bad experience.

miksa: You missed the point that Serviio must provide a DLNA client with the attributes of the feed items when it displays a feed menu. The only way it can do that is to have pre-extracted the items and run ffmpeg to cache those values, hence that must happen at least once. My proposal is to use that initial set of data and only refresh it if the initial feed item data fails to work.

hi jhb50, as you might notice i didn't miss it, but offered a workaround by sending some "default" metadata based on profile first. and then send actual ffmpeg metadata when a stream is selected to play:

maybe it's possible to even avoid ffmpeg call when displaying folder content and call ffmpeg only when online stream is selected. this way output format for items in resource folder can initially be assigned to some default format from profile.

This changes the focus of feed refreshes to one of maintaining a current set of items rather than on maintaining a current set of item url's and attributes, and has the added benefit of making it practical to refresh feeds that are constantly changing like sports and news on a more frequent basis since only new items will be extracted.

this looks to me like fixing symptoms instead of the root cause. consider this comparison with PlayOn:

1. PlayOn is DLNA server

2. PlayOn can display fresh content for dozens of online sources with dozens of sub-folders WITHOUT any need for initial scan and refresh.

3. PlayOn starts fast without all resource rescan and caching

4. PlayOn adds a new resource without initial scan and caching either.

5. PlayOn plugins in LUA harvest links on web sites similar to Serviio's plugins in Groovy but with sub-folders (take a look at PlayOn HGTV plugin http://www.playonscripts.com/?w=view&id=707 )

So I don't understand why PlayOn can do all this without a need for scan-cache-refresh but Serviio cannot do that now. and it looks like it will never be fixed.

[jhb50]

by sending some "default" metadata based on profile first. and then send actual ffmpeg metadata when a stream is selected to play:

I don't think that will work. The client needs to know exactly the format that will be sent and each profile/stream will have a different format.

I am going solely on zip's assertion that the DLNA client must have this. Perhaps he can comment on the difference with Playon.

@zip: I was of the understanding that "expires immediately" feeds would only be reparsed if the existing feed url returned an error. If not, perhaps that would be the solution.

I am going solely on zip's assertion that the DLNA client must have this. Perhaps he can comment on the difference with Playon.

I guess I know the answer - PlayOn knows upfront the target metadata from DLNA client profile. and when a stream is selected it just transcodes the stream to that format.

hence no need to run ffmpeg to collect, cache, and refresh stream metadata.

this way content of any online source is always up to date without a refresh and regardless of how many folders, sub-folders, and items can be found.

[jhb50]

Sounds like playon will then transcode everything to a predefined format and resolution regardless of the source quality and irregardless of if it would have played natively on the device.

I like the idea of only identifying the menu items when the feed is selected, which eliminates the need to refresh periodically. Unfortunately the downside is the time required to obtain the url of the feed item each time it is selected. In some of my plugins 2 or 3 page accesses are required before the url info can be obtained and this can take considerable time, and then there is no guarantee the resulting url is valid or available. The upside with Serviio is that all urls will have been tested for validity by ffmpeg when the feed item is initially defined, and will not be displayed in the menu if it is invalid or unavailable.

While I see pros and cons with both solutions, I think my proposed changes offer the best of both. Only get metadata once with ffmpeg; only extract feed url's when they are new, or found to be obsolete at play time, and on a refresh update the list of feed items but only extract the url if new. This results in rapid display of menus; transcodes only when the the stream cannot be played natively, and streams at the optimal resolution. A further improvement would be to verify that the menu items are still current by checking the current list of feed items at menu display time and not display those that are no longer current, however I see no way to add any new items at menu display time without excessive delays.

A further improvement would be to verify that the menu items are still current by checking the current list of feed items at menu display time and not display those that are no longer current however I see no way to add any new items at menu display time without excessive delays.

a few comments:

1. even after this "improvement" Serviio would still require scan of ALL online resources on start or periodically because it has no way to know when and where new items may appear. that verification may take very long time for a big collection of online resources. and it will be a complete waste of one's computer resources and the source web sites for any menu folders which were verified/updated by Serviio but not open between refreshes.

2. at menu display time only a few menu items have to be displayed initially because screen size is limited on TV. so it's not necessary to wait until complete folder is scanned. in PlayOn it works Ok even if plugin needs a few http requests to find url for each stream.

3. as noted previously, harvesting plugins in PlayOn are very similar to serviio plugins. but there are NO "excessive delays" while adding new and removing old items at menu display time. you can check for yourself http://www.playon.tv/download?rsrc=sas

[jhb50]

I have documented the above online streams logic in the wiki at http://wiki.serviio.org/doku.php?id=online_logic
and summarized my implementation of the proposed Quick Refresh logic at http://wiki.serviio.org/doku.php?id=quick_refresh

These are both in support of the bit bucket request https://bitbucket.org/xnejp03/serviio/i ... k-per-item to implement Quick Refresh within Serviio for Expires Immediately urls.