Library refresh on 1.5

Zip. Thanks for new version of Serviio the library refresh seems a lot neater. One problem though when copying large files to a shared folder it seems to be killing the network performance on my NAS because Serviio seems to be adding the files to the library before I've finished copying all of the files. Would it be possible to delay the time from file being copied/updated to the time to Serviio inventory's the file to allow file copies to complete?

Yeah, I can confirm that. NAS performance goes down to a thin trickle when automatic refresh is on. Besides, HDDs never go to sleep. Wonder what that Delta scan hooks into the OS for, if it actually does so.

Back to manual refresh as with Serviio 1.4.1.2. Every feature is a bug unless it can be turned off.

P.S.: Yes, the log file says

  Code:

INFO  [LibraryDeltaScanner] Added repository '[shared folder]' to delta scanner (NATIVE)

when automatic is on.

What is the OS of the NAS?

Can ou post a bit of detailed log of the file being added before it finished copying? Normally it should keep checking if the file is complete and then add it (run ffmpeg etc)

Thanks Zip,

The NAS is a Zyxel NSA325 OS is "Linux 2.6.31.8 #2 Mon Aug 11 16:32:01 CST 2014 armv5tel GNU/Linux".

Please find attached a debug log from the time the files where being copied across the network to the NAS from my Win7 laptop. Please let me know if you need anything else. Thanks

Normally it should keep checking if the file is complete and then add it (run ffmpeg etc)

What if there are several files being copied, to Serviio-shared or other folders? Or if you start a copy while Serviio is indexing? This may not be a problem on a PC, but a 1.6GHz NAS is practically blocked for the time it takes Serviio to refresh its library (automatic or manual). Isn't it possible to carry out the refreshing as a background task that is entirely suspended as long as any data transfer is going on?

[zip]

this is the log bit where is does checking (lock is not supported on the OS, it checks if the file size keeps changing during a period of time)

  Code:

2015-02-03 20:26:20,144 DEBUG [LibraryHelper] Checking if the file is complete using file lock
2015-02-03 20:26:20,145 DEBUG [LibraryHelper] Checking if the file is complete using file size
2015-02-03 20:26:20,940 DEBUG [LibraryHelper] File is complete with size 1073739776

Did your 1.5 setup actually finish, as it'll re-index the entire library first, which might take some time.

[zip]

Normally it should keep checking if the file is complete and then add it (run ffmpeg etc)

What if there are several files being copied, to Serviio-shared or other folders? Or if you start a copy while Serviio is indexing? This may not be a problem on a PC, but a 1.6GHz NAS is practically blocked for the time it takes Serviio to refresh its library (automatic or manual). Isn't it possible to carry out the refreshing as a background task that is entirely suspended as long as any data transfer is going on?

Serviio has single queue for adding files to the library, so all is sequential.

new event -> queue <- process next event in the queue

new event -> queue <- process next event in the queue

Hmmm. Could you be a bit more explicit? In other words, why does the transfer from the PC to the NAS slow down from normally about 55 MB/s to 1.5, when auto refresh is on? It goes up again immediately, when, while the transfer is running, you switch it off.

this is the log bit where is does checking (lock is not supported on the OS, it checks if the file size keeps changing during a period of time)

  Code:

2015-02-03 20:26:20,144 DEBUG [LibraryHelper] Checking if the file is complete using file lock
2015-02-03 20:26:20,145 DEBUG [LibraryHelper] Checking if the file is complete using file size
2015-02-03 20:26:20,940 DEBUG [LibraryHelper] File is complete with size 1073739776

Did your 1.5 setup actually finish, as it'll re-index the entire library first, which might take some time.

It's strange that it doesn't support file lock. Is this JAVA 8 that doesn't support file lock? The NAS supports inotify. Before Serviio 1.5 I used a script which ran inotifywait from the inotify tools to capture changes to the filesytem of my shared folders and update library accordingly.

My install has finished as far as I am aware. I've left it on all weekend.

[zip]

It's strange that it doesn't support file lock. Is this JAVA 8 that doesn't support file lock?

only Windows supports lock as far as I'm aware (ie you cannot write to a file that is already written to), Linux doesn't care.

So if you disable automatic updates, the throughput is normal?

So if you disable automatic updates, the throughput is normal?

That's correct. If I disable automatic updates throughput returns to normal. Thanks for the help

[zip]

Can you run a test with only one shared folder with a couple files? I want to know if the amount of monitored files makes any difference.

Can you run a test with only one shared folder with a couple files? I want to know if the amount of monitored files makes any difference.

Hi Zip, I made a new shared folder called Test then used robocopy to copy the files from my Win7 laptop to the NAS. First copy to the empy Test shared folder with "Library refresh" enabled, second without. Then I did a 3rd test copying the files to my video share that has over 4000 files in it with "Library refresh enabled" and forth test without it. Results below...

  Code:

1/ Empty "Test" shared folder adding first files - Library automatically updated
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         2         1         0         0         0
   Files :         9         9         0         0         0         0
   Bytes :   4.237 g   4.237 g         0         0         0         0
   Times :   0:01:40   0:01:40                       0:00:00   0:00:00


   Speed :            45261358 Bytes/sec.
   Speed :            2589.875 MegaBytes/min.

   Ended : Mon Feb 09 20:24:49 2015

2/ Empty "Test" shared folder adding first files - No Library Update
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :         9         9         0         0         0         0
   Bytes :   4.237 g   4.237 g         0         0         0         0
   Times :   0:01:27   0:01:26                       0:00:00   0:00:00


   Speed :            52381889 Bytes/sec.
   Speed :            2997.315 MegaBytes/min.

   Ended : Mon Feb 09 20:27:48 2015

3/ Large video library ~4000 files - Library automatically updated
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :         9         9         0         0         0         0
   Bytes :   4.237 g   4.237 g         0         0         0         0
   Times :   0:01:57   0:01:56                       0:00:00   0:00:01


   Speed :            39155464 Bytes/sec.
   Speed :            2240.493 MegaBytes/min.

   Ended : Mon Feb 09 20:32:26 2015

4/ Large video library ~4000 files - No Library Update
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :         9         9         0         0         0         0
   Bytes :   4.237 g   4.237 g         0         0         0         0
   Times :   0:01:26   0:01:26                       0:00:00   0:00:00


   Speed :            52618141 Bytes/sec.
   Speed :            3010.834 MegaBytes/min.

   Ended : Mon Feb 09 20:35:36 2015

As you can see on the large shared folder with "Library Refresh" enabled it makes quite a big difference 53mb/sec down to 39mb/sec. With Windows Explorer the difference is even greater from 53mb/sec to 30mb/sec.

I hope that helps

[zip]

I meant to have a clean Serviio installation with only a few shared files - to check if observing the one folder / few files takes so much bandwidth as 4000 files being observed.

Hello,

I uploaded the last version 1.5: very nice Apps. It was the first time I discover it after changing of media server (Allshare) due to no more sustained by Samsung in W7/8.1.
All were good, Except when I tried to update my library (add some folder and /or new mp3 files in folder alreadyexisting), nothing ran, and more ever the console stayed locked (or blocked). Necessary to abort the task, to retreive my computer.
Did you receive other comments about this update mechanism in error? How can I proceed to solve it, please?
Config: ASUS intel core i3, W8.1pro 64 bits

Thanks for your help

Hello,

After checking a lot of things (firewall, access rights, shared folder, run as admin ...), I had an idea to cut and past the 2 folders not seen /detected by the Derby updater, and to paste them in a new folder out of Serviio perimeter. After, I copy and paste to the initial folder, and ...
The Derby update program reacted correctly and force the update of the library database YES
To resume: the updater has yet some issues to detect all cases of changes based on directory, folders or files. But here you have a bypass to wait the correction
Regards

I meant to have a clean Serviio installation with only a few shared files - to check if observing the one folder / few files takes so much bandwidth as 4000 files being observed.

Hi Zip

Sorry for the delay on getting back to you. I was kinda of worried about wiping out my library but creating a new folder did the job.

  Code:

1/ Empty "Test" library adding first files - Library automatically updated
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :        11        11         0         0         0         0
   Bytes :   6.795 g   6.795 g         0         0         0         0
   Times :   0:02:45   0:02:44                       0:00:00   0:00:00


   Speed :            44251309 Bytes/sec.
   Speed :            2532.080 MegaBytes/min.

   Ended : Mon Feb 23 11:08:05 2015

2/ Empty "Test" library adding first files - No Library Update
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :        11        11         0         0         0         0
   Bytes :   6.795 g   6.795 g         0         0         0         0
   Times :   0:02:18   0:02:18                       0:00:00   0:00:00


   Speed :            52578442 Bytes/sec.
   Speed :            3008.562 MegaBytes/min.

   Ended : Mon Feb 23 11:10:58 2015

So pretty similar difference from 53mb/sec down to 44mb/sec with only 11 files in the library. Hope that helps

[zip]

Thanks, with 1.5.1 you can now force POLLER scanning, so basically doing pre-1.5 library scan. Check out the system variables docs on the website.

I'm running the 1.5 evaluation release on Yosemite. A Samsung TV is accessing files on a shared folder on the Mac. I cannot get it to update the server even when selecting "auto update". I presume this is the problem...Due to current Java implementation, the NATIVE scanner is not available on these platforms. POLLER is used for all shared folders on these systems.

So I have to press "force refresh" to get the files refreshed.

Any advice on how I can fix this?

Version 1.5.1 locks my pc up ..

Crazy after 1.4.2 serviio really is NO GOOD

The new version also does not automatiq refresh the maps ...

Strange

[zip]

I'm running the 1.5 evaluation release on Yosemite. A Samsung TV is accessing files on a shared folder on the Mac. I cannot get it to update the server even when selecting "auto update". I presume this is the problem...Due to current Java implementation, the NATIVE scanner is not available on these platforms. POLLER is used for all shared folders on these systems.

So I have to press "force refresh" to get the files refreshed.

Any advice on how I can fix this?

The MAC installation should update files every 5 minutes - is that not the case?