Generic ContentURLContainer plugin for dynamic web scripts

Hey,

I am looking for a plugin that is basically capable of receiving a set of ContentURLContainer objects from a dynamic web script. This way, a web script can be used to harvest the required data, which should be a lot faster (since it can be hosted on a fast web server) if many more HTTP requests are required to construct content URLs etc. In contrast to LiveFeeds, this approach allows all ContentURLContainer parameters to be specified by the web script instead of just some.

An RSS feed could look something like this:

  Code:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/">
   <channel>
      <title>Folder name here ...</title>
      <pubDate>...</pubDate>
      <item>
         <title>Here be dragons!</title>
         <expiresImmediately>1</expiresImmediately>
         <live>1</live>
         <contentUrl>rtmp://...</contentUrl>
         <cacheKey>27821ea227782eed16e5b7062874d944</cacheKey>
         <link>http://27821ea227782eed16e5b7062874d944</link>
      </item>
      <item>
         [...]
      </item>
   </channel>
</rss>

The extractorMatches method would either have to identify a magic string (like LiveFeeds does) or an URL prefix like

  Code:

        boolean extractorMatches(URL feedUrl) {
                if (URL.toString().startsWith(URL_PREFIX)) {
                        def rssDoc = new URL(feedUrl.toString()).getText().find('(?s)<\\?xml.*')
                        def rssNodes = new XmlParser().parseText(rssDoc)
                        itemNode = rssNodes.channel.item

                        return true;
                }

                return false;
        }

The extractUrl method would have to extract the parameters from the RSS feed and turn them into appropriate Java types and return the resulting ContentURLContainer object.

I would appreciate if

• someone could look into this, commenting about whether this makes sense or not
• implemented the extractUrl method of the groovy plugin

Thanks,
wae3

[jhb50]

And how would you trigger the webscript to run whenever you want to refresh the feed? Refreshing the RSS in Serviio will only refresh the data that the webscript last created, rather than the data current at that time, whereas the refresh of a WebResource updates the data current at that time, which is a requirement for dynamic sources.

And how would you trigger the webscript to run whenever you want to refresh the feed? Refreshing the RSS in Serviio will only refresh the data that the webscript last created, rather than the data current at that time, whereas the refresh of a WebResource updates the data current at that time, which is a requirement for dynamic sources.

Ideally, the feed fetcher would honor the HTTP expires header or the RSS equivalent, refetch the feed after it expires in the cache and add/remove items accordingly. It certainly doesn't have to be real time for me and it should not matter if there are items that are 10 minutes or so old and do not play. These are really corner cases most of the time (at least for me). In doubt, Serviio's minimal refresh time of an hour would suffice.

I guess what I would basically like to have is LiveFeeds (which can already be generated by dynamic web scripts), but with the added flexibility of all the other missing ContentURLContainer parameters.

[jhb50]

You miss my point..the serviio RSS feed fetcher will refresh the feed per the RSS contents. The point is there is no mechanism to invoke your webscript at any RSS refresh time to update your webscript generated RSS even if the Live_Feeds groovy were expanded to accept other parameters.

Thats the whole purpose of the Webresource, so you can either access multiple urls in the Webresource or just make a single call to an external url to run a script to obtain whatever you need.

[zip]

The point is there is no mechanism to invoke your webscript at any RSS refresh time to update your webscript generated RSS even if the Live_Feeds groovy were expanded to accept other parameters.

Not sure I get it. When the feed expires (either after the interval in the console, by force refreshing it or by setting expiresOn on ContentURLContainer). At that point Serviio would make a new request to the script, which would generate a new RSS feed including new/updated items.

[jhb50]

At that point Serviio would make a new request to the script, which would generate a new RSS feed including new/updated items

Now I don't get it. If I add an RSS, at expiry time all Serviio does is reaccess the RSS and process its then present feed items. If I add a WebResource at expiry time it will execute a groovy to generate a list of feed items and process them. Where is the mechanism to "make a new request to the script"?

[zip]

If I get it right, the OP will create a script that produces RSS (e.g. http://myscript.php). So if the PHP script URL is added as a RSS feed to the console it'll call the php on the feed expiry. The PHP script will generate a new set of items (new RSS feed) every time, I suppose.

[jhb50]

Now I get it..finally...the script "returns" an rss to serviio. I was misled by "generate" thinking it was just going to store the rss and then you would need to access it separately. Makes sense.

If I get it right, the OP will create a script that produces RSS (e.g. http://myscript.php). So if the PHP script URL is added as a RSS feed to the console it'll call the php on the feed expiry. The PHP script will generate a new set of items (new RSS feed) every time, I suppose.

Exactly.

[jhb50]

On second thought I do not get it. Sure you can generate an RSS that way but how do you invoke the Generic Plugin to process it?

[zip]

just match it on the script URL, the same as any other RSS that needs a plugin. The plugin would then just parse the RSS (similar to what you're doing in your live feeds plugin) , although the elements from the example above are not valid RSS elements. It might still parse though, possibly with a help of a custom XML namespace, like:

  Code:

 <item>
         <title>Here be dragons!</title>
         <serviio:expiresImmediately>1</serviio:expiresImmediately>
         <serviio:live>1</serviio:live>
         <contentUrl>rtmp://...</contentUrl>
         <serviio:cacheKey>27821ea227782eed16e5b7062874d944</serviio:cacheKey>
      </item>

[...], although the elements from the example above are not valid RSS elements. It might still parse though, possibly with a help of a custom XML namespace, like:[...]

Now this statement raises some questions.

1. Does Serviio fetch the specified URL of the feed before Plugin invocation? Considering that requests only show up in batches of two in my web server access log and the plugin I'm using right now (LiveFeeds) does fetch the content again in extractorMatches(), this seems to be the case.

Follow-up questions then are:
2. How can I get the content of the requested URL without refetching it myself from within the plugin. My understanding of a good plugin architecture suggests that this has to be possible.

3. Does Serviio do any parsing on the requested content before Plugin invokation?

If answer to 3. is yes:
4. Why does Serviio do any parsing on the requested content before Plugin invokation? My understanding of a good plugin architecture suggests that it should not care about it as long as the Plugin generates a valid ContentURLContainer object. It should not care about whether the transport is e.g. XML or JSON.

5. What exactly does Serviio do with the content? What does it check? What requirements have to be fulfilled in order to be accepted as a valid document?

Edit:
Trying to answer my own questions:
- Answers to {1, 3, 4} basically boil down to those plugins being "RSS/ATOM Feed Plugins" (so they are XML) and not generic plugins.
- Question 2 sounds like a useful improvement. Maybe a getContent() method can be added that returns the document as a String.

[zip]

1. Does Serviio fetch the specified URL of the feed before Plugin invocation? Considering that requests only show up in batches of two in my web server access log and the plugin I'm using right now (LiveFeeds) does fetch the content again in extractorMatches(), this seems to be the case.

For RSS feed plugins it does parse the RSS first and then calls possible plugin per feed item. For Web Resource plugins the plugin is responsible for reading and parsing the original content file.

2. How can I get the content of the requested URL without refetching it myself from within the plugin. My understanding of a good plugin architecture suggests that this has to be possible.

Use Feed plugin type.

3. Does Serviio do any parsing on the requested content before Plugin invokation?

Yes for feed plugins.

4. Why does Serviio do any parsing on the requested content before Plugin invokation? My understanding of a good plugin architecture suggests that it should not care about it as long as the Plugin generates a valid ContentURLContainer object. It should not care about whether the transport is e.g. XML or JSON.

As explained above, it only parses RSS/Atom feeds. This is so that plugin devs don't have to deal with parsing the standard RSS documents themselves. WebResource plugin has to do all that.

5. What exactly does Serviio do with the content? What does it check? What requirements have to be fulfilled in order to be accepted as a valid document?

For Feed plugin, it hs to be a vlid RSS/Atom document. For WebResource plugin it can be anything (incl. binary data )

2. How can I get the content of the requested URL without refetching it myself from within the plugin. My understanding of a good plugin architecture suggests that this has to be possible.

Use Feed plugin type.

According to the plugin development guide (viewtopic.php?f=22&t=3274), a feed plugin author only has access to "URL feedUrl" in extractorMatches() and "Map<String,URL> links, PreferredQuality requestedQuality" in extractUrl().

I fail to see how to get the previously downloaded document. All you can do is download the document again using feedUrl.getText(), which is however exactly what I do not want to do (as per my description, see above) and does in general not make sense as the document can change in the meantime.

Overwriting openUrl() would probably work, but it cannot be overwritten by a plugin because it is defined final.

[zip]

2. How can I get the content of the requested URL without refetching it myself from within the plugin. My understanding of a good plugin architecture suggests that this has to be possible.

Use Feed plugin type.

According to the plugin development guide (viewtopic.php?f=22&t=3274), a feed plugin author only has access to "URL feedUrl" in extractorMatches() and "Map<String,URL> links, PreferredQuality requestedQuality" in extractUrl().

I fail to see how to get the previously downloaded document. All you can do is download the document again using feedUrl.getText(), which is however exactly what I do not want to do (as per my description, see above) and does in general not make sense as the document can change in the meantime.

Overwriting openUrl() would probably work, but it cannot be overwritten by a plugin because it is defined final.

Ok, in the case of your example you'd have to write a WebResource plugin to get access to the other, non standard elements. Then you have to implement 2 methods, one to read the document and return a list of items and the other to returns the content URL for each item.

Ok, in the case of your example you'd have to write a WebResource plugin to get access to the other, non standard elements. Then you have to implement 2 methods, one to read the document and return a list of items and the other to returns the content URL for each item.

This sounds and looks like it would work. Nevertheless, I would consider the lack of access to the original document a bug in serviio's plugin architecture. Let me again point out that there is no guarantee the document does not change.

- Question 2 sounds like a useful improvement. Maybe a getContent() method can be added that returns the document as a String.