FAQ  •  Register  •  Login

how do I download a link?

<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Fri Apr 13, 2012 5:16 pm

how do I download a link?

In attemting to extract the parameters for an rtmp link in a web resource groovy I found the link http://castamp.com/embed.php?c=puurnld4 (and other c values) could not be opened as a url. ie: String linkHtml = new URL(srcUrl).getText() Explorer also returns a blank page.

I can however download the link using my downloader utility and it contains the necessary rtmp parameter values, so my question is what is the best way to download that link using groovy?
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Fri Apr 13, 2012 6:20 pm

Re: how do I download a link?

the web page is empty but has a source HTML (right click in the browser). It's possible they're expectiv some (browser like) headers, eg User-Agent
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Fri Apr 13, 2012 10:03 pm

Re: how do I download a link?

Actually the webpage does load in the browser but is quickly replaced with a blank page as per the html below. So the question remains how do I access that html in groovy when String linkHtml = new URL(srcUrl).getText()
gives me the null page ?

  Code:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title>CastAMP.com</title>
<script type="text/javascript">
if(self==top) {

window.location = "about:blank";


}
</script>
<noscript><meta http-equiv="refresh" content="0; URL=about:blank"></noscript>
</head>
 
<body>

<div style="text-align:center;width:700px;">

<iframe src="468x60.php" id="header_iframe" frameborder='0' marginheight='0' marginwidth='0' scrolling='no' width='468' height='60'></iframe>
</div>
<div>
<div style="position:relative">
   <div id="ad_overlay" style="width:700px;height:398px;position:absolute;top:0px;z-index:100;background-color:rgba(169, 152, 128, 0.4);display:none;">
      <div id="ad_overlay_countdown" style="700px;position:absolute;top:20px;left:150px;text-align:center;color:#fff;font-size:11px">
This ad will close in <span id="countdownnum"></span> seconds.   <a href="javascript:void(0)" onclick="removeOverlayHTML()"><img style="border:0;z-index:101;display:none" src="http://castamp.com/x.png" alt="x" id="xad" /></a></div>


      <div id="ad_overlay_content" style="width:300px;height:250px;position:absolute;top:35px;left:150px;background:#fff;"></div>
   </div>
      <div id="ad_footer" style="width:700px;height:60px;position:absolute;top:270px;left:6px;background:url();display:none">
      <div id="ad_footer_content" style="position:absolute;width:468px;height:60px;">
<iframe id='footer_iframe' frameborder='0' marginheight='0' marginwidth='0' scrolling='no' width='468' height='60'></iframe>

</div>   
   

   </div>
   <script type="text/javascript" src="swfobject2.js"></script>
      <div id="player">
        <h2>Loading Player...</h2>
   </div>
   <script type='text/javascript'>

  var so = new SWFObject('http://bitcast-r.v1.sjc1.bitgravity.com/malar/Players/player.swf','mpl','700','398','9');
  so.addParam('allowfullscreen','true');
  so.addParam('allowscriptaccess','always');
  so.addParam('wmode','opaque');
//  so.addVariable('skin','http://live.castamp.com/fsjw43.xml');
  so.addVariable('controlbar','over');
  so.addVariable('backcolor','877c6b');
  so.addVariable('dock','false');
  so.addVariable('autostart','true');
  so.addVariable('stretching','uniform');
  so.addVariable('file', 'puurnld4');
  so.addVariable('streamer', 'rtmp://stream.castamp.com:1935/amp');
  //so.addVariable('bufferlength','3');
  so.addVariable('rtmp.tunneling','false');
  so.addVariable('logo.file', 'http://castamp.com/images/castamp.png');
  so.addVariable('logo.hide','false');
  so.addVariable('logo.position','top-right');

  so.addVariable('logo.link','http://castamp.com/home.html');
  so.addVariable('abouttext','CastAMP.com &#45; Free Broadcasting');
  so.addVariable('aboutlink','http://castamp.com/home.html');
  so.write('player');
</script>
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Fri Apr 13, 2012 10:49 pm

Re: how do I download a link?

not sure how they are doing this, but it doesn't seem to be possible. curl and wget fail as well
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sat Apr 14, 2012 4:08 am

Re: how do I download a link?

Using Wireshark here are the requests made by my downloader utility (which downloads the file) and Curl which fails.

Does this give any info that will help solve this problem?

Can anyone write a curl command to duplicate the downloader request?
I've tried but have not succeeded.

  Code:

http:\\www.castamp.com/embed.php?c=puurnld4

CURL
----
GET /embed.php?c=puurnld4 HTTP/1.1
User-Agent: curl/7.23.1 (x86_64-pc-win32 libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5
Host: castamp.com
Accept: */* 

Downloader
----------
GET /embed.php?c=ladytnb HTTP/1.1\r\n
Accept: */*\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)\r\n
Host: castamp.com\r\n
Connection: Keep-Alive\r\n
Cookie: __utma=109905047.1530474507.1329370460.1334327012.1334331785.9; __utmz=109905047.1329370460.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)\r\n

<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sat Apr 14, 2012 3:17 pm

Re: how do I download a link?

More research finds that
  Code:
curl -X GET "http://castamp.com/embed.php?c=puurnld4"  -H "Accept-Encoding: gzip" -v  > D:\curl.txt


will download a gzip'd version of the file, and I can get the gzip string in groovy with
  Code:
String linkHtml =  new URL(srcUrl).getText(requestProperties:['Accept-Encoding': 'gzip,deflate'])


Can someone tell me how I get groovy to uncompress the gzip format string back to text??
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Sat Apr 14, 2012 7:17 pm

Re: how do I download a link?

I'm using this in Java (you can use it in Groovy too):

  Code:
new GZIPInputStream (is)


You have to pass 'is' which would be the gzipped InputStream, so you have to figure out how to return InputStream from URL rather than text. Looks like then you can call .text on the gzip stream to read its contents like:


  Code:
new GZIPInputStream (is).text


More on Groovy streams is here: http://groovy.codehaus.org/JN2025-Streams
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sun Apr 15, 2012 3:17 pm

Re: how do I download a link?

Thanks..works great

  Code:
import java.util.zip.GZIPInputStream;
String decodedtext = new GZIPInputStream (new URL(address).newInputStream(requestProperties:['Accept-Encoding':
'gzip,deflate'])).text
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sun Apr 15, 2012 7:06 pm

Re: how do I download a link?

Seems I spoke too soon.

The GZIP code
  Code:
String linkHtml = new GZIPInputStream (new URL(srcUrl).newInputStream(requestProperties:['Accept-Encoding': 'gzip,deflate'])).text         

works fine in my groovy in test mode and generates the correct ContentURLContainer
  Code:

ContentURLContainer [fileType=VIDEO, contentUrl=rtmp://50.115.119.42/live/bonanza1.stream swfUrl=http://bitcast-r.v1.sjc1.bitgravity.com/malar/Players/player.swf pageURL=http://castamp.com/home.html playpath=bonanza1.stream, thumbnailUrl=https://sites.google.com/site/hahaflags/en.gif, live=true, expiresOn=Tue May 15 13:12:00 EDT 2012, expiresImmediately=true, cacheKey=http://www.hahasport.com/c-11.html?&estreams=4_http://www.hahasport.com/v-2/18/156/v-395700.html]


however when executed within Serviio the log shows this error:

  Code:
2012-04-15 12:47:00,137 DEBUG [WebResourceParser] Unexpected error during url extractor plugin invocation (HaHaSport) for item Sun 5:00-18:00 (Table 1)World Snooker Championship Qualifying-World Snooker: No signature of method: java.net.URL.newInputStream() is applicable for argument types: (java.util.LinkedHashMap) values: [[requestProperties:[Accept-Encoding:gzip,deflate]]]
Possible solutions: newInputStream(), withInputStream(groovy.lang.Closure)


Why would Serviio process this groovy differently?
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Sun Apr 15, 2012 11:02 pm

Re: how do I download a link?

My standalone version of groovy which executes my plug-in correctly is version 1.8.3. What version does Serviio have builtin which fails with my plug-in per my previous post? The failing method in Serviio was implemented in Groovy 1.8.1 so perhaps Serviio is not yet at that level.
<<

jhb50

DLNA master

Posts: 2843

Joined: Thu Jun 30, 2011 9:32 pm

Post Mon Apr 16, 2012 4:41 am

Re: how do I download a link?

Further testing shows that using the Serviio groovy-all.jar for my standalone testing generates the same error, which would appear to prove the Serviio groovy does not have the 1.8.1 release enhancements. The fix would be to include the groovy-1.8.3.jar in the 1.0 release.

Return to Plugin development

Who is online

Users browsing this forum: No registered users and 15 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.