FAQ  •  Register  •  Login

Too many open files

<<

daemonblade

Serviio newbie

Posts: 3

Joined: Fri Dec 28, 2018 6:49 pm

Post Fri Dec 28, 2018 7:27 pm

Too many open files

Hi,

I've been trying out serviio 9.x and 10.x on my FreeBSD box, and for most part the software works great. However, I have noticed that if I include a shared directory containing 10's of thousands of files, eg a photo library, the open-file count spikes until my system reports that there are too many open-files. It appears that the library scanner is opening each file and not closing the file once it inspects the content. In fact on a smaller shared-library, it appears to be keeping an open-file descriptor for each file(?)

I had thought of attempting to fix the code myself, until I realised that the project is not open-sourced. :D

Could the kind developer please take a look?

Cheers
--
Jonathan
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Sun Dec 30, 2018 5:08 pm

Re: Too many open files

can you troubleshoot a bit more? Are all the files immediately opened when you share the folder, or does the count rise as is scans one file at a time?
<<

daemonblade

Serviio newbie

Posts: 3

Joined: Fri Dec 28, 2018 6:49 pm

Post Tue Jan 01, 2019 9:16 am

Re: Too many open files

zip wrote:can you troubleshoot a bit more? Are all the files immediately opened when you share the folder, or does the count rise as is scans one file at a time?


I'm not quite sure what you mean here. The photo-folder had been added onto "Library" > "Shared folders" with "Media type" of Image; at which point the open-file count starts to climb.

When I reboot my host, the serviio server starts up automatically; at which point I also notice the open-file count starts to spike against the configured serviio-user.

I have also noticed in some of the logs that:
  Code:
2018-12-20 20:05:28,411 ERROR [ProcessExecutor] Fatal error in process starting: Cannot run program "/usr/local/bin/dcraw": error=2, No such file or directory
java.io.IOException: Cannot run program "/usr/local/bin/dcraw": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at org.serviio.external.ProcessExecutor.run(ProcessExecutor.java:136)
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:251)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 1 more
2018-12-20 20:05:28,412 WARN  [MediaServer] DCRAW not found. Support for raw image files will be missing.


I'm not sure whether that is relevant to the open-file descriptor problem, though.
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Wed Jan 02, 2019 10:28 pm

Re: Too many open files

Can you try to turn on Poller for the shared folder in the Serviio console and see if the effect is the same?

Also, what Linux command do you use to diagnose this?
<<

daemonblade

Serviio newbie

Posts: 3

Joined: Fri Dec 28, 2018 6:49 pm

Post Mon Jan 07, 2019 3:51 am

Re: Too many open files

zip wrote:Can you try to turn on Poller for the shared folder in the Serviio console and see if the effect is the same?


If I use the Poller mechanism for the shared folder, the open-files issue disappears. Thanks for the workaround!

zip wrote:Also, what Linux command do you use to diagnose this?


Assuming that $serviiouser is the uid for the serviio server, on FreeBSD:
  Code:
$ fstat -u $serviiouser | wc -l


On Linux, the equivalent would be:
  Code:
$ lsof -u $serviiouser | wc -l
<<

zip

User avatar

Serviio developer / Site Admin

Posts: 17212

Joined: Sat Oct 24, 2009 12:24 pm

Location: London, UK

Post Sun Jan 27, 2019 5:01 pm

Re: Too many open files

Thanks,

So on Linux I think Java uses inotify to track file changes. I have no clue how it's implemented on the low level there, it might be that it keeps the file handles open.

Return to Serviio Support & Help

Who is online

Users browsing this forum: No registered users and 52 guests

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software for PTF.