FAQ  •  Register  •  Login

Native filesystem monitoring for libraries updates

<<

lukakama

Serviio newbie

Posts: 5

Joined: Fri May 09, 2014 7:28 am

Post Fri May 09, 2014 9:26 am

Native filesystem monitoring for libraries updates

Hi,
these days I had some time to develop an add-on for Serviio in order to use native file monitoring features to trigger library updates. This should then allows any HDD with resources monitored by a Serviio instance to correctly go on sleep when in idle :) .

The add-on requires at least a Java 7 runtime, and it should works on any O.S. By far, it has been tested it on:
  • Windows 7 - 64bit
  • Embedded Linux - Kernel 2.6.31.8 - ARMv5 (NAS Zyxel NSA310)
ONLY LOCAL DIRECTORIES ARE FULLY SUPPORTED (network shares are partially supported with some limitations as described here: https://docs.google.com/spreadsheets/d/1QbTpL6onqhJU54N6IQtHG1bmK0XTBYLHbPmmjPufUlM/edit?usp=sharing)

Actually, the add-on is just a jar to add to the Serviio's classpath, and it works using an alternative main Java entry class which, in turn, starts Serviio and, immediately after, starts its own monitoring feature for libraries configured inside Serviio.
Once started, it will trigger a Serviio full libraries scan after 5 seconds since the last filesystem change detected on any of the monitored directory (the 5 seconds delay is needed to avoid to request multiple consecutive scans when multiple files/directories are added/changed/removed). It can also detect long-copying operations, waiting the copy completion before trigger the library update.

The add-on is also capable to detect Serviio library configuration changes, in order to update directories being monitored. However, the configuration change monitoring is quite hackish, so it could not work on Serviio versions other then 1.4.1.2.

If ZIP give me the permission, I will be glad to attach needed files (released under Apache License 2.0) and installation instructions, creating also a github repository for patches and contributions.

========= INSTALL INSTRUCTIONS =========
  • Disable Serviio automatic library update
  • Stop Serviio server
  • Download the jar add-on from https://www.dropbox.com/s/zewom1tnuql1q ... -1.0.0.jar
  • Put the add-on jar into "<SERVIIO_PATH>\lib" directory
  • Edit "<SERVIIO_PATH>\bin\serviio.bat" (Windows) or "<SERVIIO_PATH>/bin/serviio.sh" (Linux) and change every occurrence of org.serviio.MediaServer to com.lukakama.serviio.watchservice.WatchServiceMainWrapper
  • ONLY ON LINUX - Edit any Lunix daemon script and change every occurrence of org.serviio.MediaServer to com.lukakama.serviio.watchservice.WatchServiceMainWrapper if any
  • Start Serviio server

Source code on GitHub: https://github.com/lukakama/serviio-watch-service
Last edited by lukakama on Thu May 15, 2014 10:55 am, edited 5 times in total.
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Fri May 09, 2014 12:31 pm

Re: Native filesystem monitoring for libraries updates

Ok with me, if it's a separate Jar file and doesn't modify any original files.

Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.
<<

lukakama

Serviio newbie

Posts: 5

Joined: Fri May 09, 2014 7:28 am

Post Fri May 09, 2014 3:26 pm

Re: Native filesystem monitoring for libraries updates

zip wrote:Ok with me, if it's a separate Jar file and doesn't modify any original files.

Great! The add-on just uses Serviio's classes, it doesn't overwrite or modify any file.

I will add the jar and instructions as soon as possible :) .

zip wrote:Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.


I made some tests and it seems to depend on JVM / Guest OS / Host OS combinations.
Here is a status summary (I will try to perform other tests using my NAS as guest): https://docs.google.com/spreadsheets/d/ ... sp=sharing
<<

lukakama

Serviio newbie

Posts: 5

Joined: Fri May 09, 2014 7:28 am

Post Sat May 10, 2014 4:58 pm

Re: Native filesystem monitoring for libraries updates

lukakama wrote:
zip wrote:Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.


I made some tests and it seems to depend on JVM / Guest OS / Host OS combinations.
Here is a status summary (I will try to perform other tests using my NAS as guest): https://docs.google.com/spreadsheets/d/ ... sp=sharing


Well, it seems that native monitoring on network shares has some limitations. From my tests, it seems that using a Windows 7 guest, it is possible to monitor only up to 62 directories of a share on a Windows 7 host and only 40 on a Unix host (Samba shares). Also, Unix samba shares can be monitored only using Java 8.

So, I must say that the add-on should be used only on Serviio installations with local libraries, as like it should be for embedded NAS.
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Tue May 13, 2014 6:12 am

Re: Native filesystem monitoring for libraries updates

I want to do similar function for Linux based Serviio server. I started another topic about Linux implementation, but I want to advice form You. For triggering update action I use such command:

  Code:
inotifywait -e close_write -e delete -e modify -e move /path_to_watching_directory1 /path_to_watching_directory2 etc...


Work nice, but of course I have to write whole script.
So my question is, how to execute library update? I try something like this, without positive results:

  Code:
wget http://127.0.0.1:23423/rest/action/forceLibraryRefresh


Can You help me to create proper syntax for library refreshing?
<<

lukakama

Serviio newbie

Posts: 5

Joined: Fri May 09, 2014 7:28 am

Post Tue May 13, 2014 7:33 pm

Re: Native filesystem monitoring for libraries updates

I'm sorry, but I don't know how to help you :( : my add-on is directly connected to Serviio run-time code, so it uses undocumented APIs to force a library update.

However, the add-on should works well also with Linux based Serviio servers (I'm using it on my NAS - Embedded Linux - Kernel 2.6.31.8) .
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Tue May 13, 2014 8:27 pm

Re: Native filesystem monitoring for libraries updates

mackowiakp wrote:Can You help me to create proper syntax for library refreshing?

You have to do a POST (not GET) request, with a XML/Json body describing the action. See example in the API website.
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Wed May 14, 2014 7:01 am

Re: Native filesystem monitoring for libraries updates

OK. So I used such syntax:

  Code:
curl  -X POST --data "forceLibraryRefresh" http://127.0.0.1:23423/rest/action


and the result is:

  Code:
<?xml version="1.0" encoding="UTF-8" ?>
<result/>


Bus as far as I know it is not right result. Sorry for my question but I am very new in REST implementation. Can You correct my syntax?
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Wed May 14, 2014 9:16 am

Re: Native filesystem monitoring for libraries updates

According to API documentation I finally use such syntax:

  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>{forceLibraryRefresh}</name> </action>' http://127.0.0.1:23423/rest/action


But the result is wrong:

  Code:
HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Date: Wed, 14 May 2014 09:16:02 GMT
Server: Linux, UPnP/1.0 DLNADOC/1.50, Serviio/1.4.1.2
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/xml; charset=UTF-8

<?xml version="1.0" encoding="UTF-8" ?>
<result/


What I am doing wrong?
<<

lukakama

Serviio newbie

Posts: 5

Joined: Fri May 09, 2014 7:28 am

Post Wed May 14, 2014 4:08 pm

Re: Native filesystem monitoring for libraries updates

I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Wed May 14, 2014 4:50 pm

Re: Native filesystem monitoring for libraries updates

lukakama wrote:I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action


YOU ARE THE MASTER!!! Just removing brackets is necessary! THX
<<

will

DLNA master

Posts: 2138

Joined: Mon Aug 30, 2010 11:18 am

Location: UK

Post Wed May 14, 2014 7:36 pm

Re: Native filesystem monitoring for libraries updates

mackowiakp wrote:
lukakama wrote:I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action


YOU ARE THE MASTER!!! Just removing brackets is necessary! THX


While it does work because serviio will just return xml, your accept header doesn't make sense, you cannot accept both xml and json. However, as you aren't looking to parse the server's response, you don't care what you get.

A slightly more concice version:

  Code:
curl -X POST -H "Content-Type: application/xml" -d "<action><name>forceLibraryRefresh</name></action>" http://localhost:23423/rest/action
Will

ServiiDroid (Android Console) Developer: Download | Home | Support
ServiiGo (Android 3G/4G/WiFi Playback App) Developer: Download | Home | Support
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Thu May 15, 2014 3:51 am

Re: Native filesystem monitoring for libraries updates

Yep. It is more simple. Just "Keep it ASAP" (ASAP -> As Simple As Possible). THX!
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Thu May 15, 2014 5:13 am

Re: Native filesystem monitoring for libraries updates

Be so kind to explain what parameter I have to include in forceOnlineResourceRefresh call? What is id number of online source? Is it number of source in Serviio console "Online Library" table?
<<

mackowiakp

Serviio lover

Posts: 71

Joined: Tue Feb 11, 2014 8:31 pm

Post Thu May 15, 2014 10:41 am

Re: Native filesystem monitoring for libraries updates

Finally I have a script for automatic database update for Linux. It starts from cron after system starts.
It works on native filesystem, mounted via loop, SMB or NFS and any mix. If any remarks - Welcome!

  Code:
#!/bin/bash
sleep 600
while true; do
TRIG=`inotifywait -rqq -e close_write -e delete -e modify -e move /home/media/disk-1/Filmy /home/media/disk-2/Muzyka /home/media/disk-2/Zdjecia|wc -w`
while true; do
        sleep 30
        TRIG=`inotifywait -t 30 -rq -e close_write -e delete -e modify -e move /home/media/disk-1/Filmy /home/media/disk-2/Muzyka /home/media/disk-2/Zdjecia|wc -w`
        if [ $TRIG -eq "0" ]
        then break
        fi

done
while true; do
        RUN=`curl -sS -X GET -H "Content-Type: application/xml" http://localhost:23423/rest/library-status|grep false|wc -l`
        if [ $RUN -eq "2" ]
        then break
        fi
sleep 15
done
        #echo "Update bazy Serviio"
        curl -sS -X POST -H "Content-Type: application/xml" -d "<action><name>forceLibraryRefresh</name></action>" http://localhost:23423/rest/action >/dev/null
done

Return to Third-party tools integration

Who is online

Users browsing this forum: No registered users and 24 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.