Page 1 of 1

Native filesystem monitoring for libraries updates

PostPosted: Fri May 09, 2014 9:26 am
by lukakama
Hi,
these days I had some time to develop an add-on for Serviio in order to use native file monitoring features to trigger library updates. This should then allows any HDD with resources monitored by a Serviio instance to correctly go on sleep when in idle :) .

The add-on requires at least a Java 7 runtime, and it should works on any O.S. By far, it has been tested it on:
  • Windows 7 - 64bit
  • Embedded Linux - Kernel 2.6.31.8 - ARMv5 (NAS Zyxel NSA310)
ONLY LOCAL DIRECTORIES ARE FULLY SUPPORTED (network shares are partially supported with some limitations as described here: https://docs.google.com/spreadsheets/d/1QbTpL6onqhJU54N6IQtHG1bmK0XTBYLHbPmmjPufUlM/edit?usp=sharing)

Actually, the add-on is just a jar to add to the Serviio's classpath, and it works using an alternative main Java entry class which, in turn, starts Serviio and, immediately after, starts its own monitoring feature for libraries configured inside Serviio.
Once started, it will trigger a Serviio full libraries scan after 5 seconds since the last filesystem change detected on any of the monitored directory (the 5 seconds delay is needed to avoid to request multiple consecutive scans when multiple files/directories are added/changed/removed). It can also detect long-copying operations, waiting the copy completion before trigger the library update.

The add-on is also capable to detect Serviio library configuration changes, in order to update directories being monitored. However, the configuration change monitoring is quite hackish, so it could not work on Serviio versions other then 1.4.1.2.

If ZIP give me the permission, I will be glad to attach needed files (released under Apache License 2.0) and installation instructions, creating also a github repository for patches and contributions.

========= INSTALL INSTRUCTIONS =========
  • Disable Serviio automatic library update
  • Stop Serviio server
  • Download the jar add-on from https://www.dropbox.com/s/zewom1tnuql1q ... -1.0.0.jar
  • Put the add-on jar into "<SERVIIO_PATH>\lib" directory
  • Edit "<SERVIIO_PATH>\bin\serviio.bat" (Windows) or "<SERVIIO_PATH>/bin/serviio.sh" (Linux) and change every occurrence of org.serviio.MediaServer to com.lukakama.serviio.watchservice.WatchServiceMainWrapper
  • ONLY ON LINUX - Edit any Lunix daemon script and change every occurrence of org.serviio.MediaServer to com.lukakama.serviio.watchservice.WatchServiceMainWrapper if any
  • Start Serviio server

Source code on GitHub: https://github.com/lukakama/serviio-watch-service

Re: Native filesystem monitoring for libraries updates

PostPosted: Fri May 09, 2014 12:31 pm
by zip
Ok with me, if it's a separate Jar file and doesn't modify any original files.

Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.

Re: Native filesystem monitoring for libraries updates

PostPosted: Fri May 09, 2014 3:26 pm
by lukakama
zip wrote:Ok with me, if it's a separate Jar file and doesn't modify any original files.

Great! The add-on just uses Serviio's classes, it doesn't overwrite or modify any file.

I will add the jar and instructions as soon as possible :) .

zip wrote:Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.


I made some tests and it seems to depend on JVM / Guest OS / Host OS combinations.
Here is a status summary (I will try to perform other tests using my NAS as guest): https://docs.google.com/spreadsheets/d/ ... sp=sharing

Re: Native filesystem monitoring for libraries updates

PostPosted: Sat May 10, 2014 4:58 pm
by lukakama
lukakama wrote:
zip wrote:Also, does this work with remote drives? I haven't implemented it like that because people who tested it confirmed the java 7 file monitor doesn't work with remote drives.


I made some tests and it seems to depend on JVM / Guest OS / Host OS combinations.
Here is a status summary (I will try to perform other tests using my NAS as guest): https://docs.google.com/spreadsheets/d/ ... sp=sharing


Well, it seems that native monitoring on network shares has some limitations. From my tests, it seems that using a Windows 7 guest, it is possible to monitor only up to 62 directories of a share on a Windows 7 host and only 40 on a Unix host (Samba shares). Also, Unix samba shares can be monitored only using Java 8.

So, I must say that the add-on should be used only on Serviio installations with local libraries, as like it should be for embedded NAS.

Re: Native filesystem monitoring for libraries updates

PostPosted: Tue May 13, 2014 6:12 am
by mackowiakp
I want to do similar function for Linux based Serviio server. I started another topic about Linux implementation, but I want to advice form You. For triggering update action I use such command:

  Code:
inotifywait -e close_write -e delete -e modify -e move /path_to_watching_directory1 /path_to_watching_directory2 etc...


Work nice, but of course I have to write whole script.
So my question is, how to execute library update? I try something like this, without positive results:

  Code:
wget http://127.0.0.1:23423/rest/action/forceLibraryRefresh


Can You help me to create proper syntax for library refreshing?

Re: Native filesystem monitoring for libraries updates

PostPosted: Tue May 13, 2014 7:33 pm
by lukakama
I'm sorry, but I don't know how to help you :( : my add-on is directly connected to Serviio run-time code, so it uses undocumented APIs to force a library update.

However, the add-on should works well also with Linux based Serviio servers (I'm using it on my NAS - Embedded Linux - Kernel 2.6.31.8) .

Re: Native filesystem monitoring for libraries updates

PostPosted: Tue May 13, 2014 8:27 pm
by zip
mackowiakp wrote:Can You help me to create proper syntax for library refreshing?

You have to do a POST (not GET) request, with a XML/Json body describing the action. See example in the API website.

Re: Native filesystem monitoring for libraries updates

PostPosted: Wed May 14, 2014 7:01 am
by mackowiakp
OK. So I used such syntax:

  Code:
curl  -X POST --data "forceLibraryRefresh" http://127.0.0.1:23423/rest/action


and the result is:

  Code:
<?xml version="1.0" encoding="UTF-8" ?>
<result/>


Bus as far as I know it is not right result. Sorry for my question but I am very new in REST implementation. Can You correct my syntax?

Re: Native filesystem monitoring for libraries updates

PostPosted: Wed May 14, 2014 9:16 am
by mackowiakp
According to API documentation I finally use such syntax:

  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>{forceLibraryRefresh}</name> </action>' http://127.0.0.1:23423/rest/action


But the result is wrong:

  Code:
HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Date: Wed, 14 May 2014 09:16:02 GMT
Server: Linux, UPnP/1.0 DLNADOC/1.50, Serviio/1.4.1.2
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/xml; charset=UTF-8

<?xml version="1.0" encoding="UTF-8" ?>
<result/


What I am doing wrong?

Re: Native filesystem monitoring for libraries updates

PostPosted: Wed May 14, 2014 4:08 pm
by lukakama
I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action

Re: Native filesystem monitoring for libraries updates

PostPosted: Wed May 14, 2014 4:50 pm
by mackowiakp
lukakama wrote:I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action


YOU ARE THE MASTER!!! Just removing brackets is necessary! THX

Re: Native filesystem monitoring for libraries updates

PostPosted: Wed May 14, 2014 7:36 pm
by will
mackowiakp wrote:
lukakama wrote:I could be wrong, but the action name should not be enclosed in curly brackets:
  Code:
curl --include --request POST --header "Content-Type: application/xml" --header "Accept: application/xml | application/json" --data-binary '<action> <name>forceLibraryRefresh</name> </action>' http://127.0.0.1:23423/rest/action


YOU ARE THE MASTER!!! Just removing brackets is necessary! THX


While it does work because serviio will just return xml, your accept header doesn't make sense, you cannot accept both xml and json. However, as you aren't looking to parse the server's response, you don't care what you get.

A slightly more concice version:

  Code:
curl -X POST -H "Content-Type: application/xml" -d "<action><name>forceLibraryRefresh</name></action>" http://localhost:23423/rest/action

Re: Native filesystem monitoring for libraries updates

PostPosted: Thu May 15, 2014 3:51 am
by mackowiakp
Yep. It is more simple. Just "Keep it ASAP" (ASAP -> As Simple As Possible). THX!

Re: Native filesystem monitoring for libraries updates

PostPosted: Thu May 15, 2014 5:13 am
by mackowiakp
Be so kind to explain what parameter I have to include in forceOnlineResourceRefresh call? What is id number of online source? Is it number of source in Serviio console "Online Library" table?

Re: Native filesystem monitoring for libraries updates

PostPosted: Thu May 15, 2014 10:41 am
by mackowiakp
Finally I have a script for automatic database update for Linux. It starts from cron after system starts.
It works on native filesystem, mounted via loop, SMB or NFS and any mix. If any remarks - Welcome!

  Code:
#!/bin/bash
sleep 600
while true; do
TRIG=`inotifywait -rqq -e close_write -e delete -e modify -e move /home/media/disk-1/Filmy /home/media/disk-2/Muzyka /home/media/disk-2/Zdjecia|wc -w`
while true; do
        sleep 30
        TRIG=`inotifywait -t 30 -rq -e close_write -e delete -e modify -e move /home/media/disk-1/Filmy /home/media/disk-2/Muzyka /home/media/disk-2/Zdjecia|wc -w`
        if [ $TRIG -eq "0" ]
        then break
        fi

done
while true; do
        RUN=`curl -sS -X GET -H "Content-Type: application/xml" http://localhost:23423/rest/library-status|grep false|wc -l`
        if [ $RUN -eq "2" ]
        then break
        fi
sleep 15
done
        #echo "Update bazy Serviio"
        curl -sS -X POST -H "Content-Type: application/xml" -d "<action><name>forceLibraryRefresh</name></action>" http://localhost:23423/rest/action >/dev/null
done