Too many open files

Hi,

I've been trying out serviio 9.x and 10.x on my FreeBSD box, and for most part the software works great. However, I have noticed that if I include a shared directory containing 10's of thousands of files, eg a photo library, the open-file count spikes until my system reports that there are too many open-files. It appears that the library scanner is opening each file and not closing the file once it inspects the content. In fact on a smaller shared-library, it appears to be keeping an open-file descriptor for each file(?)

I had thought of attempting to fix the code myself, until I realised that the project is not open-sourced.

Could the kind developer please take a look?

Cheers
--
Jonathan

[zip]

can you troubleshoot a bit more? Are all the files immediately opened when you share the folder, or does the count rise as is scans one file at a time?

can you troubleshoot a bit more? Are all the files immediately opened when you share the folder, or does the count rise as is scans one file at a time?

I'm not quite sure what you mean here. The photo-folder had been added onto "Library" > "Shared folders" with "Media type" of Image; at which point the open-file count starts to climb.

When I reboot my host, the serviio server starts up automatically; at which point I also notice the open-file count starts to spike against the configured serviio-user.

I have also noticed in some of the logs that:

  Code:

2018-12-20 20:05:28,411 ERROR [ProcessExecutor] Fatal error in process starting: Cannot run program "/usr/local/bin/dcraw": error=2, No such file or directory
java.io.IOException: Cannot run program "/usr/local/bin/dcraw": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at org.serviio.external.ProcessExecutor.run(ProcessExecutor.java:136)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:251)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 1 more
2018-12-20 20:05:28,412 WARN  [MediaServer] DCRAW not found. Support for raw image files will be missing.


I'm not sure whether that is relevant to the open-file descriptor problem, though.

[zip]

Can you try to turn on Poller for the shared folder in the Serviio console and see if the effect is the same?

Also, what Linux command do you use to diagnose this?

Can you try to turn on Poller for the shared folder in the Serviio console and see if the effect is the same?

If I use the Poller mechanism for the shared folder, the open-files issue disappears. Thanks for the workaround!

Also, what Linux command do you use to diagnose this?

Assuming that $serviiouser is the uid for the serviio server, on FreeBSD:

  Code:

$ fstat -u $serviiouser | wc -l


On Linux, the equivalent would be:

  Code:

$ lsof -u $serviiouser | wc -l


[zip]

Thanks,

So on Linux I think Java uses inotify to track file changes. I have no clue how it's implemented on the low level there, it might be that it keeps the file handles open.