FAQ  •  Register  •  Login

BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

<<

totototo

Serviio newbie

Posts: 14

Joined: Fri Jan 04, 2013 3:19 pm

Post Fri Jan 04, 2013 3:30 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

There seems to be an issue with Beeg plugin on my NAS running Serviio 1.1
Log repeatedly says:
WARN [FeedUpdaterThread] An error occured while scanning for online item information, will continue
java.io.IOException: Unexpected error while invoking plugin (Beeg): The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:58)
at org.serviio.library.online.OnlineLibraryManager.findResource(OnlineLibraryManager.java:171)
at org.serviio.library.online.OnlineLibraryManager.findResourceInCacheOrParse(OnlineLibraryManager.java:186)
at org.serviio.library.online.metadata.FeedUpdaterThread.getOnlineItems(FeedUpdaterThread.java:159)
at org.serviio.library.online.metadata.FeedUpdaterThread.run(FeedUpdaterThread.java:96)
Caused by: java.lang.RuntimeException: The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.PluginExecutionProcessor.execute(PluginExecutionProcessor.java:64)
at org.serviio.library.online.WebResourceUrlExtractor.parseWebResource(WebResourceUrlExtractor.java:29)
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:56)
... 4 more
when v5 plugin is activated with http://www.beg.com web resource. It also max-up CPU with Java.
I have other web ressource plugins running, and no issue with them.
On another machine (Mac) same plugin, same Serviio version, there is no issue...

Any idea?
Thanks.
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Fri Jan 04, 2013 9:13 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Probably trying to add more video than the cpu or beeg server can do in 30 seconds which is all Serviio currently allows. Beeg is running slowly today. Try cutting back ... debug log will show you what is happening.
<<

totototo

Serviio newbie

Posts: 14

Joined: Fri Jan 04, 2013 3:19 pm

Post Fri Jan 04, 2013 10:33 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Not sure I understand what you mean by "cut back"?
Debug log does not seem really helpful actually
2013-01-04 23:22:40,965 DEBUG [OnlineLibraryManager] Resource http://beeg.com/ not in cache yet, loading it
2013-01-04 23:22:40,965 DEBUG [WebResourceParser] Parsing web resource 'http://beeg.com/'
2013-01-04 23:22:40,967 DEBUG [FeedItemUrlExtractor] Playlist extractor: - transmitted from WebResourceParser - playlistUrl: null
2013-01-04 23:22:40,967 DEBUG [WebResourceParser] Found matching url extractor (Beeg) for resource http://beeg.com/
2013-01-04 23:22:40,968 DEBUG [FeedItemUrlExtractor] Beeg: Starting parsing resource: http://beeg.com/
2013-01-04 23:22:40,969 DEBUG [FeedItemUrlExtractor] Beeg: Parsing with beeg V5
2013-01-04 23:22:43,514 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:22:43,611 DEBUG [AbstractCDSLibraryIndexingListener] Library updated, notifying CDS
2013-01-04 23:22:44,011 DEBUG [EventDispatcher] Sending event notification #23 for subscription d4335c1d-d395-41d7-9ac2-bae438fac33f to endpoint http://127.0.0.1:9000/
2013-01-04 23:22:44,015 DEBUG [RequestExecutor] Sending HTTP request to 127.0.0.1:9000
2013-01-04 23:22:44,023 DEBUG [EventDispatcher] Event notification sent and received successfully
2013-01-04 23:22:48,574 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:22:53,534 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:22:58,484 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:23:03,032 DEBUG [WebServer] Incoming connection from /192.168.3.200:34660
2013-01-04 23:23:03,032 DEBUG [RendererDAOImpl] Reading a Renderer with ip address 192.168.3.200
2013-01-04 23:23:03,034 DEBUG [DeviceDescriptionRequestHandler] DeviceDescription request received for device 6489fbb2-c75d-3879-9842-9162558132c2 from 192.168.3.200 (headers = [Connection: close,Cache-Control: no-cache,Accept: */*,Accept-Encoding: deflate, gzip,Accept-Language: en,Host: 192.168.3.200:8895,User-Agent: iTunes/4.7.1 (Linux; N; Linux; armv5tel-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/7.7.0/33614,Cookie: Squeezebox-player=90%3A84%3A0d%3A71%3A17%3Aa0,Cookie2: $Version="1",Icy-Metadata: 1])
2013-01-04 23:23:03,034 DEBUG [RendererDAOImpl] Reading a Renderer with ip address 192.168.3.200
2013-01-04 23:23:03,036 DEBUG [RendererManager] Looking for a renderer profile for Http headers: [Connection: close,Cache-Control: no-cache,Accept: */*,Accept-Encoding: deflate, gzip,Accept-Language: en,Host: 192.168.3.200:8895,User-Agent: iTunes/4.7.1 (Linux; N; Linux; armv5tel-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/7.7.0/33614,Cookie: Squeezebox-player=90%3A84%3A0d%3A71%3A17%3Aa0,Cookie2: $Version="1",Icy-Metadata: 1]
2013-01-04 23:23:03,037 DEBUG [RendererDAOImpl] Reading a Renderer with ip address 192.168.3.200
2013-01-04 23:23:03,097 DEBUG [DeviceDescriptionRequestHandler] Sending DeviceDescription XML back using profile 'Generic DLNA profile'
2013-01-04 23:23:03,544 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:23:08,494 DEBUG [RendererDAOImpl] Retrieving list of all stored renderers
2013-01-04 23:23:10,972 WARN [FeedUpdaterThread] An error occured while scanning for online item information, will continue
java.io.IOException: Unexpected error while invoking plugin (Beeg): The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:58)
at org.serviio.library.online.OnlineLibraryManager.findResource(OnlineLibraryManager.java:171)
at org.serviio.library.online.OnlineLibraryManager.findResourceInCacheOrParse(OnlineLibraryManager.java:186)
at org.serviio.library.online.metadata.FeedUpdaterThread.getOnlineItems(FeedUpdaterThread.java:159)
at org.serviio.library.online.metadata.FeedUpdaterThread.run(FeedUpdaterThread.java:96)
Caused by: java.lang.RuntimeException: The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.PluginExecutionProcessor.execute(PluginExecutionProcessor.java:64)
at org.serviio.library.online.WebResourceUrlExtractor.parseWebResource(WebResourceUrlExtractor.java:29)
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:56)
... 4 more
I agree this plugin somehow makes Java stress CPU and throw exceptions.
Now it's very surprising this happens only with this plugin. Other crowded web-resource plugins do not lead to such issue.

About Beeg server-side, I don't think it's the case, as my Mac get through it in parallel with no issue.
Thanks for your thoughts.
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Fri Jan 04, 2013 11:19 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

The log says your system is doing a lot of other stuff that is preventing the beeg groovy (which strated at the 40 second mark) from running, and so when the 1:10 mark is reached , Serviio terminates it. I've asked zip to look at this.
<<

totototo

Serviio newbie

Posts: 14

Joined: Fri Jan 04, 2013 3:19 pm

Post Fri Jan 04, 2013 11:28 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Actually it's Beeg plugin alone that takes Java up to 99% CPU.
Indeed, if I disable Beeg plugin in console (not restarting the server), Java gets down to 1% after few seconds (as usual).
If I re-enable Beeg (always same server session), then Java gets back to 99% forever (or until I switch-off Beeg).
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Fri Jan 04, 2013 11:38 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

That part
  Code:
def idMatch = idMatcher[0][1] =~ '(?s)(.*?)[,\\]]'
def thumbMatch = thumbMatcher[0][1] =~ "(?s)(.*?)[,\\]]"

is CPU heavy, extracting links and names for 200 or I don't know how much videos they have on one page. And if your CPU is slow it won't finish that in the allowed 30 seconds.
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sat Jan 05, 2013 3:33 am

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

That way it should be faster http://pastie.org/private/punveqmymn72atwx9armmw.
However it's ugly as hell...

@totototo
Added some logging, it should say Got HTML, Got Script, etc. Check where it fails.
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Sat Jan 05, 2013 1:07 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

The regexes can be slow, true. I'm not sure if you could cut the page after the point where the user's required number of items ends. I also found (see my ITV plugin) that sometimes the regex extraction just hangs and causes 100% cpu - not sure if it's my regex or the Groovy engine, but it was fixed by cutting off the Html file after the point all my required data end.
<<

totototo

Serviio newbie

Posts: 14

Joined: Fri Jan 04, 2013 3:19 pm

Post Sat Jan 05, 2013 1:27 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Thanks for looking into it: very much appreciated.
Here is the log with your changes:
2013-01-05 14:10:13,720 DEBUG [WebResourceParser] Parsing web resource 'http://beeg.com/'
2013-01-05 14:10:14,319 DEBUG [FeedItemUrlExtractor] Playlist extractor: - transmitted from WebResourceParser - playlistUrl: null
2013-01-05 14:10:14,319 DEBUG [WebResourceParser] Found matching url extractor (Beeg) for resource http://beeg.com/
2013-01-05 14:10:14,320 DEBUG [FeedItemUrlExtractor] Beeg: Starting parsing resource: http://beeg.com/
2013-01-05 14:10:14,321 DEBUG [FeedItemUrlExtractor] Beeg: Parsing with beeg V6
2013-01-05 14:10:15,160 DEBUG [FeedItemUrlExtractor] Beeg: Got HTML
2013-01-05 14:10:15,206 DEBUG [FeedItemUrlExtractor] Beeg: Got Script
2013-01-05 14:10:15,207 DEBUG [FeedItemUrlExtractor] Beeg: Got Matchers

2013-01-05 14:10:44,327 WARN [FeedUpdaterThread] An error occured while scanning for online item information, will continue
java.io.IOException: Unexpected error while invoking plugin (Beeg): The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:58)
at org.serviio.library.online.OnlineLibraryManager.findResource(OnlineLibraryManager.java:171)
at org.serviio.library.online.OnlineLibraryManager.findResourceInCacheOrParse(OnlineLibraryManager.java:186)
at org.serviio.library.online.metadata.FeedUpdaterThread.getOnlineItems(FeedUpdaterThread.java:159)
at org.serviio.library.online.metadata.FeedUpdaterThread.run(FeedUpdaterThread.java:96)
Caused by: java.lang.RuntimeException: The operation took more than 30000 ms and has been cancelled.
at org.serviio.library.online.PluginExecutionProcessor.execute(PluginExecutionProcessor.java:64)
at org.serviio.library.online.WebResourceUrlExtractor.parseWebResource(WebResourceUrlExtractor.java:29)
at org.serviio.library.online.WebResourceParser.parse(WebResourceParser.java:56)
... 4 more

2013-01-05 14:10:50,489 DEBUG [FeedItemUrlExtractor] Beeg: Got Strings
2013-01-05 14:10:50,490 DEBUG [FeedItemUrlExtractor] Beeg: Got IDs
2013-01-05 14:10:50,490 DEBUG [FeedItemUrlExtractor] Beeg: Got URLs
So the issue seems to be between Got Matchers and Got Strings.

Still your changes make a huge difference in CPU usage, as now it is maxed-out only during the Matcher/String phases, and "released" in other phases (as shown in graph).
CPU.png
CPU.png (49.39 KiB) Viewed 33618 times
Previously it was just maxed-out all the time. A step in a good direction for sure.

Thanks
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sat Jan 05, 2013 5:37 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

It looks like your CPU is way too slow. And I have no idea how to further speed up the extraction. Maybe jhb50 would try something.
And it's not an issue with the plugin, it's just that there are some very long strings to process on that beeg page.
Btw., I have an old P3 running as a home server and it finishes running the plugin in 10 seconds, so maybe you should check what else is slowing down your CPU.
<<

gpsis

User avatar

Streaming enthusiast

Posts: 22

Joined: Fri Apr 06, 2012 12:03 am

Post Sat Jan 05, 2013 6:51 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Hey guys I install a plugin about a week ago and it initially worked ok.

Went back a few days later and although the Online content folder was populated, I get an error that states the file is corrupt or it is the wrong format.

This is strange since I was attempting to view the same content that played a few days days prior.

I powered down the laptop and media player and restarted each, waited about 30 minutes or so and tried again and had the same issue.

Later that night I tried again and it worked fine, but a few days later the same issue is back...anyone got a clue?
Regards!

Device: Sony Media Player SMP-N100
Win7 Dual Core 2.9GHz
3GBR
Malwarebytes Anti-Spyware
BitDefender Total Security
Serviio allowed in all security software
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sat Jan 05, 2013 8:58 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Version 6 posted which should resolve performance issue.

As per below total elapsed extract time is less than 10 seconds on my system, even for 360 videos , 8 seconds of which is reading in the web page, so cpu load is negligible.

2013-01-05 11:51:14,918 DEBUG [FeedItemUrlExtractor] Beeg: Starting parsing resource: http://www.beeg.com/section/long-videos ... t=10&for=5
2013-01-05 11:51:14,918 DEBUG [FeedItemUrlExtractor] Beeg: Parsing with beeg V6
2013-01-05 11:51:14,926 DEBUG [FeedItemUrlExtractor] Beeg: Startup = 1 msec
2013-01-05 11:51:22,711 DEBUG [FeedItemUrlExtractor] Beeg: Read URL = 7790 msec
2013-01-05 11:51:22,714 DEBUG [FeedItemUrlExtractor] Beeg: Parse Web Page = 4 msec
2013-01-05 11:51:22,727 DEBUG [FeedItemUrlExtractor] Beeg: Create 5 Items = 13 msec
2013-01-05 11:51:22,727 DEBUG [FeedItemUrlExtractor] Beeg: Total Item Extract Time = 7808 msec



2013-01-05 11:22:57,342 DEBUG [FeedItemUrlExtractor] Beeg: Starting parsing resource: http://beeg.com?for=400
2013-01-05 11:22:57,342 DEBUG [FeedItemUrlExtractor] Beeg: Parsing with beeg V6
2013-01-05 11:22:57,343 DEBUG [FeedItemUrlExtractor] Beeg: Startup = 0 msec
2013-01-05 11:23:05,522 DEBUG [FeedItemUrlExtractor] Beeg: Read URL = 8179 msec
2013-01-05 11:23:05,527 DEBUG [FeedItemUrlExtractor] Beeg: Parse Web Page = 5 msec
2013-01-05 11:23:06,430 DEBUG [FeedItemUrlExtractor] Beeg: Create 360 Items = 903 msec
2013-01-05 11:23:06,430 DEBUG [FeedItemUrlExtractor] Beeg: Total Time = 9087 msec
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sat Jan 05, 2013 10:41 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Version 7 posted, which uses the fact that all the beeg streams use the same file name template. This allows the ContentURL's to be generated rather than extracted which results in 360 URL's being generated in 1 second, rather than the 22 minutes it took on my system each time the groovy was refreshed.
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sat Jan 05, 2013 11:21 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Nice, but the search fails.

And add that to fix the titles.
  Code:
Put '' here
def thumbMatch = thumbMatcher[0][1] =~ "(?s)'(.*?)'[,\\]]"

Add videoTitle = videoTitle.replaceAll("\\\\'", "'")
after videoTitle = thumbMatch[i][1].trim()


And generating links probably wouldn't work for all streams, cause half of streams are from: http://NN.007i.net/480p/
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sat Jan 05, 2013 11:33 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Search works fine for me http://beeg.com/search?q=blond
and all my videos are fromhttp://NN.video.mystreamservice.com/480p/.
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sat Jan 05, 2013 11:48 pm

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Gonna check why the search is not working here.

And I believe that they are doing some load balancing or moving servers, cause just yesterday half of videos where redirected to that other server. But just checked right now and indeed all of them are from the first server. Got to see if they would change something.
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sun Jan 06, 2013 12:29 am

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

The search was failing because you forgot to remove \\] just as I did)
Here:
  Code:
def thumbMatcher = html =~ '(?s)var tumbalt =\\[(.*?)\\];'

Should be just
def thumbMatcher = html =~ '(?s)var tumbalt =\\[(.*?);'
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sun Jan 06, 2013 1:04 am

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Yes, I forgot to remove that second closing ] you added but it did not impact my search. What were you searching for?

Maybe I'll add a parm to allow the user to switch between generating and extracting the url's.
<<

mqojdn

Streaming enthusiast

Posts: 44

Joined: Thu Jan 03, 2013 6:39 pm

Post Sun Jan 06, 2013 1:58 am

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

It was failing for anything, even for that http://beeg.com/search?q=blond.
The strange thing is that it was working on the Ubuntu VM, where I was checking the scripts with Groovy, and failing on the Windows server. It looks like there is some crap with different Java/Groovy string handling or something like that.
<<

totototo

Serviio newbie

Posts: 14

Joined: Fri Jan 04, 2013 3:19 pm

Post Sun Jan 06, 2013 10:31 am

Re: BEEG - AN ADULT STREAMS VIDEO WEB RESOURCE

Hey guys, works like a charm on my ARM-based NAS now!
Great optimization job.
Thanks!
PreviousNext

Return to Available plugins

Who is online

Users browsing this forum: No registered users and 21 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.