Page 1 of 2

YouTube plugin using Data API V3

PostPosted: Sat Apr 25, 2015 5:02 pm
by stefana888
Plugin type Webresource

One simple url:


https://www.googleapis.com/youtube/v3/playlistItems?playlistId=YOUR_PLAYLIST_ID_HERE

Note: The url covers channel uploads since everything is playlists.

A Google YouTube API key is needed in code below.

  Code:
import org.serviio.library.metadata.*
import org.serviio.library.online.*
import groovy.json.*

/**
 * YouTube.com content URL extractor plugin.
 *
 * API call playlistItems.list is implemented
 * Using youtube_dl with best (default) quality
 * 
 * @author Stefan Andersson
 *
 **/

class YouTube extends WebResourceUrlExtractor {

    final VALID_RESOURCE_URL = '^https?://www.googleapis.com/youtube/.*$'
       
    String getExtractorName() {
        return getClass().getName()
    }
   
    boolean extractorMatches(URL feedUrl)
    {
        return feedUrl ==~ VALID_RESOURCE_URL
    }
   
    WebResourceContainer extractItems(URL resourceUrl, int maxItemsToRetrieve)     
    {
      final user_agent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
      final api_key = "YOUR_API_KEY_HERE"

      if (maxItemsToRetrieve == -1) maxItemsToRetrieve = 50

      def apiUrl = new URL(resourceUrl.toString() + "&part=snippet" + "&maxResults=" + maxItemsToRetrieve.toString() + "&key=" + api_key)

      def json = new JsonSlurper().parseText(openURL(apiUrl, user_agent))

      def items = []

      json.items.each()
      {
         if (it.snippet.title != "Deleted video")
         {
            items.add(new WebResourceItem(title : it.snippet.title,
                                                               additionalInfo : ['videoId' : it.snippet.resourceId.videoId,
                                                               'thumb' : it.snippet.thumbnails.high.url]))

            log("title: ${it.snippet.title}")
            log("videoId: ${it.snippet.resourceId.videoId}")
            log("thumb: ${it.snippet.thumbnails.standard.url}")
         }
      }

      return new WebResourceContainer(title : "", items: items)
    }
     
    ContentURLContainer extractUrl(WebResourceItem item, PreferredQuality requestedQuality)
    {
      def linkUrl = "http://www.youtube.com/watch?v=${item.additionalInfo['videoId']}"
      def thumbnailUrl = item.additionalInfo['thumb']
   
      .
      .
      .
      .


 }

Re: YouTube plugin using Data API V3

PostPosted: Sun Apr 26, 2015 1:48 pm
by zip
Thanks, I've updated the main plugin based on your code. hope it's ok.

viewtopic.php?f=20&t=3276

Re: YouTube plugin using Data API V3

PostPosted: Wed Apr 29, 2015 7:00 pm
by stefana888
Yes. I see now that GET 'videos' can be used with the same code. I was mixing that up with GET 'search'.


Edit: No, it cannot be usd with same code now.

Re: YouTube plugin using Data API V3

PostPosted: Sat May 09, 2015 12:38 pm
by kerulen
Hi,

I have made few changes to support the latest uploads by referencing the channel name:
Now, with verision 2, you can supply a PLAYLIST_ID.
With this version 3 you can specify the Channel Name and it retrieves the Uploads Playlist for that channel.

  Code:
https://www.googleapis.com/youtube/v3/channels?forUsername=CHANNEL_NAME


To obtain the "real and exact" Channel Name, you can go to the channel an click on "videos" menu.
Then you can look for an URL like:

  Code:
https://www.youtube.com/user/CHANNEL_NAME/videos

Example:

  Code:
https://www.youtube.com/user/EpicTVadventure/videos

In Serviio you must create a Web Resource like this:

  Code:
https://www.googleapis.com/youtube/v3/channels?forUsername=CHANNEL_NAME

Examples:

  Code:
EpicTVadventure
https://www.googleapis.com/youtube/v3/channels?forUsername=EpicTVadventure
GoProCamera
https://www.googleapis.com/youtube/v3/channels?forUsername=GoProCamera


This is the piece of code I have changed...

  Code:
      ...

   @Override
   WebResourceContainer extractItems(URL resourceUrl, int maxItemsToRetrieve) {
      final user_agent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
      final api_key = "AIzaSyAPeRAlHMSa_WQMbarc_ffse7-t0-DOuVQ" // this is Serviio API key, don't use it for anything else please, they are easy to get

      if (maxItemsToRetrieve == -1) maxItemsToRetrieve = 50
    
     if (resourceUrl.toString().contains("channels")){
      def channelUrl = new URL(resourceUrl.toString() + "&part=contentDetails" + "&maxResults=" + maxItemsToRetrieve + "&key=" + api_key) 
      def channeljson = new JsonSlurper().parseText(openURL(channelUrl, user_agent))
      def uploadPlaylist = channeljson.items[0].contentDetails.relatedPlaylists.uploads
         
      resourceUrl = new URL("https://www.googleapis.com/youtube/v3/playlistItems?playlistId=$uploadPlaylist")
     }
          
      def apiUrl = new URL(resourceUrl.toString() + "&part=snippet" + "&maxResults=" + maxItemsToRetrieve + "&key=" + api_key)

       ...



I have attached "version 3" Youtube plugin to this post.

This works for me! ;) :D

Serviio: 1.5.2
O.S: Windows 8.1
Youtube Plugin: version 3


Youtube.groovy
Serviio Youtube Plugin version 3
(8.42 KiB) Downloaded 1593 times

Re: YouTube plugin using Data API V3

PostPosted: Sat May 09, 2015 1:08 pm
by stefana888
Well, it is always possible to make a suitable abstraction of the Youtube API v3, e.g. take channel names in playlist input. To support a specified number of result pages (more than 50 items) in the input would also be interesting.

Re: YouTube plugin using Data API V3

PostPosted: Sat May 09, 2015 9:49 pm
by tuareg64ve
For linux users, remind to set the correct youtube-dl path in the following line:

  Code:
final YOUTUBE_DL = "/usr/local/bin/youtube-dl"


Could you please tell me how to test a playlist?
The following lines:

  Code:
    static void main(args) {
        // this is just to test
        YouTubeDL extractor = new YouTubeDL()

        assert extractor.extractorMatches( new URL("https://www.youtube.com/watch?v=AJtDXIazrMo&list=PLDcnymzs18LVXfO_x0Ei0R24qDbVtyy66") )

        WebResourceContainer container = extractor.extractItems( new URL("https://www.youtube.com/watch?v=AJtDXIazrMo&list=PLDcnymzs18LVXfO_x0Ei0R24qDbVtyy66"), 10)
        println container
        ContentURLContainer result = extractor.extractUrl(container.getItems()[2], PreferredQuality.MEDIUM)
        println result
    }


test a single video, right?
How can modify them for a playlist?
Sorry, I am not so much familiar with Groovy, I am trying to learn how to use it.
Thanks

Re: YouTube plugin using Data API V3

PostPosted: Sun May 10, 2015 2:26 am
by stefana888
Maybe this is the wrong thread... youtube-dl uses identical URLs with that in the web browser for selecting a playlist. Not to mix up with the Youtube API.

Re: YouTube plugin using Data API V3

PostPosted: Tue May 12, 2015 6:00 pm
by stefana888
New working version of the YouTube plugin

Changes:

1. Videos bug fixed
Now handles different Json result for videos and playlist id's without causing error

2. Channel names (user name) added:
  Code:
https://www.googleapis.com/youtube/v3/channels?forUsername=USER_NAME

** Test code added in main **

  Code:
import groovy.json.JsonSlurper

import org.serviio.library.metadata.*
import org.serviio.library.online.*

/**
 * YouTube.com content URL extractor plugin.
 *
 * It uses YouTube API v3 for retrieving playlists.
 *
 * @see http://en.wikipedia.org/wiki/Youtube#Quality_and_codecs
 * 
 * @author Petr Nejedly
 *
 */
class YouTube extends WebResourceUrlExtractor {

    final VALID_RESOURCE_URL = '^https?://www.googleapis.com/youtube/.*$'
   /* Listed in order of quality */
    final availableFormats = ['37', '46', '22', '45', '35', '34', '18', '44', '43', '6', '5']
   
    String getExtractorName() {
        return getClass().getName()
    }
   
    boolean extractorMatches(URL feedUrl) {
        return feedUrl ==~ VALID_RESOURCE_URL
    }
   
   public int getVersion() {
      return 3;
   }
   
   @Override
   WebResourceContainer extractItems(URL resourceUrl, int maxItemsToRetrieve) {
      final user_agent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
      final api_key = "AIzaSyAPeRAlHMSa_WQMbarc_ffse7-t0-DOuVQ" // this is Serviio API key, don't use it for anything else please, they are easy to get

      if (maxItemsToRetrieve == -1) maxItemsToRetrieve = 50
   
      // Handle channel name urls
      if (resourceUrl.toString().contains("channels")) {
          def channelUrl = new URL(resourceUrl.toString() + "&part=contentDetails" + "&key=" + api_key) 
     def channeljson = new JsonSlurper().parseText(openURL(channelUrl, user_agent))
     def uploadPlaylist = channeljson.items[0].contentDetails.relatedPlaylists.uploads
         
     resourceUrl = new URL("https://www.googleapis.com/youtube/v3/playlistItems?playlistId=$uploadPlaylist")
      }    
          
      def apiUrl = new URL(resourceUrl.toString() + "&part=snippet" + "&maxResults=" + maxItemsToRetrieve + "&key=" + api_key)

      def json = new JsonSlurper().parseText(openURL(apiUrl, user_agent))

      def items = []

      json.items.each() {
         if (it.snippet.title != "Deleted video") {
             items.add(new WebResourceItem(title : it.snippet.title,
                                           additionalInfo : ['videoId' : resourceUrl.toString().contains("videos") ? it.id : it.snippet.resourceId.videoId,
                                                             'thumb' : it.snippet.thumbnails.high.url]))
         }
      }
    
     def containerThumbnailUrl = items?.find { it -> it.additionalInfo['thumb'] != null }?.additionalInfo['thumb']

      return new WebResourceContainer(items: items, thumbnailUrl: containerThumbnailUrl)
    }

   @Override
   protected ContentURLContainer extractUrl(WebResourceItem wrItem, PreferredQuality requestedQuality) {
      
        def contentUrl
      def expiryDate
      def expiresImmediately
      def cacheKey
      
        def videoId = wrItem.additionalInfo['videoId']
      def thumbnailUrl = wrItem.additionalInfo['thumb']
      
        for (elType in ['&el=embedded', '&el=detailpage', '&el=vevo', '']) {           
            def videoInfoUrl = "http://www.youtube.com/get_video_info?&video_id=$videoId$elType&ps=default&eurl=&gl=US&hl=en"
            //log("Loading video info: $videoInfoUrl")
            def videoInfoWebPage = new URL(videoInfoUrl).getText()
            def parameters = [:]
       
            videoInfoWebPage.split('&').each{item -> addParameter(item, parameters, '=')}
            if(parameters.containsKey('token')) {      
               def formatUrlMapString = parameters['fmt_url_map']   
            def urlEncodedUrlMapString = parameters['url_encoded_fmt_stream_map']
            def adaptiveFormatsMapString = parameters['adaptive_fmts']
            def allFormatUrlMap = [:]
            
            def test = URLDecoder.decode(adaptiveFormatsMapString,'UTF-8').split(',').each{item -> addParameter(item, allFormatUrlMap, '\\|')}
            
            if (formatUrlMapString != null && formatUrlMapString.length() > 0 ) {
               URLDecoder.decode(formatUrlMapString,'UTF-8').split(',').each{item -> addParameter(item, allFormatUrlMap, '\\|')}                  
            } else {            
               if (urlEncodedUrlMapString != null && urlEncodedUrlMapString.length() > 0 ) {                
                  processStreamData(urlEncodedUrlMapString, allFormatUrlMap)
               }   
               if (adaptiveFormatsMapString != null && adaptiveFormatsMapString.length() > 0 ) {
                  processStreamData(adaptiveFormatsMapString, allFormatUrlMap)
               }
            }
            // get available formats for requested quality, sorted by quality from highest
            def formatUrlMap = new LinkedHashMap()
            if(requestedQuality == PreferredQuality.HIGH) {
               // best quality, get the first from the list
               sortAvailableFormatUrls(availableFormats, allFormatUrlMap, formatUrlMap)
               def selectedUrl = formatUrlMap.entrySet().toList().head()
               contentUrl = selectedUrl.getValue()
               cacheKey = getCacheKey(videoId, selectedUrl.getKey())
            } else if (requestedQuality == PreferredQuality.MEDIUM) {
               // work with subset of available formats, starting at the position of format 35 and then take the best quality from there
               sortAvailableFormatUrls(availableFormats.getAt(4..availableFormats.size-1), allFormatUrlMap, formatUrlMap)
               def selectedUrl = formatUrlMap.entrySet().toList().head()
               contentUrl = selectedUrl.getValue()
               cacheKey = getCacheKey(videoId, selectedUrl.getKey())
            } else {
               // worst quality, take the last url
               sortAvailableFormatUrls(availableFormats, allFormatUrlMap, formatUrlMap)
               def selectedUrl = formatUrlMap.entrySet().toList().last()
               contentUrl = selectedUrl.getValue()
               cacheKey = getCacheKey(linkUrl, selectedUrl.getKey())
            }          
            if(contentUrl != null) {
               expiresImmediately = true
               if(contentUrl.startsWith('http')) {
                  // http URL
                  def contentUrlParameters = [:]
                  contentUrl.split('&').each{item -> addParameter(item, contentUrlParameters, '=')}
                  if( contentUrlParameters['expire'] != null ) {   
                     //log(Long.parseLong(contentUrlParameters['expire']).toString())
                     expiryDate = new Date(Long.parseLong(contentUrlParameters['expire'])*1000)
                     expiresImmediately = false
                  }
               } else {
                     // rtmp URL
                     def rtmpMatcher = contentUrl =~ 'rtmpe?://.*?/(.*)'
                  def app = rtmpMatcher[0][1]
                  // TODO load swf player URL from the HTML page
                     contentUrl = "$contentUrl app=$app swfUrl=http://s.ytimg.com/yt/swfbin/watch_as3-vflg0Q-LP.swf swfVfy=1"
               }   
            }
               
                break
            }
        }   
        return new ContentURLContainer(fileType: MediaFileType.VIDEO, contentUrl: contentUrl, thumbnailUrl: thumbnailUrl, expiresOn: expiryDate, expiresImmediately: expiresImmediately, cacheKey: cacheKey)
    }
 
    def addParameter(parameterString, parameters, separator) {
        def values = parameterString.split(separator)
        if( values.length == 2 ) {   
             parameters.put(values[0], values[1])
        }
    }   
   
   def processStreamData(String urlEncodedUrlMapString, Map streamsMap ) {
      URLDecoder.decode(urlEncodedUrlMapString,'UTF-8').split(',').each{item ->
         def streamParams = [:]
         item.split('&').each{item2 -> addParameter(item2, streamParams, '=')}
         String urlKeyName = streamParams.containsKey('url') ? 'url' : 'conn'
         //String stream = URLDecoder.decode(streamParams['stream'],'UTF-8') //TODO stream is playpath
         String streamUrl = URLDecoder.decode(streamParams[urlKeyName],'UTF-8')
         String signature = streamParams['sig']
         if(signature) streamUrl = streamUrl + "&signature=" + signature
         streamsMap.put(streamParams['itag'], streamUrl )
      }
   }

   def String getCacheKey(String videoId, String qualityId) {
      "youtube_${videoId}_${qualityId}"
   }
      
   def sortAvailableFormatUrls(List formatIds, Map sourceMap, Map targetMap) {
      formatIds.each{formatId ->
         if(sourceMap.containsKey(formatId) ) {            
             targetMap.put(formatId, sourceMap.get(formatId))
         }
      }
   }
   
    static void main(args) {
      // this is just to test
        YouTube extractor = new YouTube()
      
      assert extractor.extractorMatches( new URL("https://www.googleapis.com/youtube/v3/playlistItems?playlistId=YOUR_PLAYLIST_ID_HERE") )
      assert !extractor.extractorMatches( new URL("http://google.com/feeds/api/standardfeeds/top_rated?time=today") )
      
      
      WebResourceContainer container = extractor.extractItems( new URL("https://www.googleapis.com/youtube/v3/playlistItems?playlistId=PL5D6FB3CCE0053D01"), 10)
      println container
      ContentURLContainer result = extractor.extractUrl(container.getItems()[2], PreferredQuality.MEDIUM)
      println result
      
                println ""
                container = extractor.extractItems( new URL("https://www.googleapis.com/youtube/v3/videos?chart=mostPopular"), 10)
      println container
      result = extractor.extractUrl(container.getItems()[2], PreferredQuality.MEDIUM)
      println result

                println ""
                container = extractor.extractItems( new URL("https://www.googleapis.com/youtube/v3/channels?forUsername=NFL"), 10)
                println container
                result = extractor.extractUrl(container.getItems()[2], PreferredQuality.MEDIUM)
      println result
    }

}

Re: YouTube plugin using Data API V3

PostPosted: Tue May 12, 2015 9:20 pm
by tuareg64ve
This link is SkyNews live channel:

https://www.googleapis.com/youtube/v3/v ... w4hmqVPe0E

This request to API v3 works fine:

https://www.googleapis.com/youtube/v3/v ... entDetails

Putting in SERVIIO I get this issue:

  Code:
2015-05-12 23:15:31,935 WARN  [FeedUpdaterWorker] An error occured while parsing the online resource https://www.googleapis.com/youtube/v3/videos?id=sw4hmqVPe0E, will try again soon: Unexpected error while invoking plugin (YouTube): null
org.serviio.library.online.metadata.OnlineResourceParseException: Unexpected error while invoking plugin (YouTube): null
   at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:78)
   at org.serviio.library.online.OnlineLibraryManager.findResource(OnlineLibraryManager.java:186)
   at org.serviio.library.online.OnlineLibraryManager.findResourceInCacheOrParse(OnlineLibraryManager.java:203)
   at org.serviio.library.online.metadata.FeedUpdaterWorker.getOnlineItems(FeedUpdaterWorker.java:191)
   at org.serviio.library.online.metadata.FeedUpdaterWorker.run(FeedUpdaterWorker.java:108)
   at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
2015-05-12 23:15:32,668 DEBUG [SearchManager] Committing search index


any idea?

Re: YouTube plugin using Data API V3

PostPosted: Wed May 13, 2015 12:10 am
by stefana888
tuareg64ve wrote:This link is SkyNews live channel:

https://www.googleapis.com/youtube/v3/v ... w4hmqVPe0E

any idea?


Don't know what causes that, the link works for me. Server restart??

Re: YouTube plugin using Data API V3

PostPosted: Wed May 13, 2015 10:41 pm
by tuareg64ve
Unfortunately server restart does not help!
Have you tried that video in your Serviio ?

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 12:02 am
by stefana888
tuareg64ve wrote:Unfortunately server restart does not help!
Have you tried that video in your Serviio ?


Yes I have tested it because I saw it was a live video. The problem occurs before the plugin code is entered. Something must be wrong with the Serviio setup or plugin installation. No Youtube at all is working, correct?

Note. The Youtube plugin assumes all videos are not live. It should play but same ffmpeg instance running after stop can eventually fill up disc. You may start another transcoding job or manually kill the ffmpeg process.

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 6:37 am
by tuareg64ve
No Youtube at all is working, correct?


Correct.Any chance to fix it?
Is this a major issue or it could be fixed with few changes?

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 7:35 am
by stefana888
tuareg64ve wrote:
No Youtube at all is working, correct?


Correct.Any chance to fix it?
Is this a major issue or it could be fixed with few changes?


Make sure plugintype is 'Web tesource'. If it still not work. Delete all plugins, restart server and reinstall plugins. You can also try reinstall Serviio.

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 7:49 am
by tuareg64ve
stefana888 wrote:
tuareg64ve wrote:Make sure plugintype is 'Web tesource'. If it still not work. Delete all plugins, restart server and reinstall plugins. You can also try reinstall Serviio.


Plugintype is "Web resource".
I have just reinstalled Serviio and the plugins for the upgrade to 1.5.2.
Installation is on Linux, so there is no need of installation procedure. It is just a copy of files.
Everything is working except Youtube live streams.
Why should I have to do it again?

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 8:35 am
by stefana888
tuareg64ve wrote:
stefana888 wrote:
tuareg64ve wrote:Make sure plugintype is 'Web tesource'. If it still not work. Delete all plugins, restart server and reinstall plugins. You can also try reinstall Serviio.


Plugintype is "Web resource".
I have just reinstalled Serviio and the plugins for the upgrade to 1.5.2.
Installation is on Linux, so there is no need of installation procedure. It is just a copy of files.
Everything is working except Youtube live streams.
Why should I have to do it again?

Sky news live works for me.

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 9:08 am
by tuareg64ve
stefana888 wrote:Sky news live works for me.


NFL works for me, but Sky News Live still does not work.
Could you post me your version of Youtube.groovy ?

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 10:27 am
by stefana888
tuareg64ve wrote:
stefana888 wrote:Sky news live works for me.


NFL works for me, but Sky News Live still does not work.
Could you post me your version of Youtube.groovy ?


I now see that there is a problem with the official plugin (in the extractor) on that live video. This is not related to the Youtube v3 API. I use a selfmade v3 API youtube-dl tool-based plugin that suits my specific needs. You could get a youtube-dl tool-based plugin too from the available plugins section (YoutubeDL).

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 11:31 am
by tuareg64ve
I have already installed YoutubeDL.groovy
Now I do not have time to check, but I will check and let you know
Thanks

Re: YouTube plugin using Data API V3

PostPosted: Thu May 14, 2015 11:48 am
by stefana888
tuareg64ve wrote:I have already installed YoutubeDL.groovy
Now I do not have time to check, but I will check and let you know
Thanks


......http live streaming (HLS) was not supported in the extractor of the official plugin. Live content needs to be added.....