Apostrophe in file name

I have named my files with the episode titles of series, which sometimes contain an apostrophe (') character. The call that serviio makes to ffmpeg seems to fail on these files. I upgraded to the latest version of serviio (1.5) but that didn't fix the problem.

Example: movie and subtitle file:
193 The Doctor's Daughter.mkv
193 The Doctor's Daughter.ass

The following is from the logfile (after setting the log level to DEBUG):

  Code:

2015-01-18 11:07:37,624 DEBUG [ServiioHttpService] Incoming request from /10.1.33.60:48079: GET /resource/69912/MEDIA_ITEM/MPEG_TS_SD_NA_ISO-0/ORIGINAL HTTP/1.1, headers = [User-Agent: UPnP/1.0 DLNADOC/1.50,Host: 10.1.33.1:8895,Accept: */*,TimeSeekRange.dlna.org: npt=0-,X-AV-Physical-Unit-Info: pa="Blu-ray Disc Player",X-AV-Client-Info: av=5.0; cn="Sony Corporation"; mn="Blu-ray Disc Player"; mv="2.0",getcontentFeatures.dlna.org: 1,Pragma: getIfoFileURI.dlna.org,transferMode.dlna.org: Streaming]]
2015-01-18 11:07:37,624 DEBUG [RendererDAOImpl] Reading a Renderer with ip address 10.1.33.60
2015-01-18 11:07:37,625 DEBUG [RendererDAOImpl] Reading a Renderer with ip address 10.1.33.60
2015-01-18 11:07:37,627 DEBUG [ResourceDeliveryProcessor] Resource request accepted. Using client 'Identifier=10.1.33.60, Profile=Sony BDP-Sx70/SMP-N100 (2010)'
2015-01-18 11:07:37,627 DEBUG [ResourceDeliveryProcessor] Request for resource 69912 and type 'MEDIA_ITEM' received
2015-01-18 11:07:37,627 DEBUG [MediaResourceRetrievalStrategy] Getting information about media item 69912 (local)
2015-01-18 11:07:37,627 DEBUG [MediaItemDAOImpl] Reading a MediaItem (id = 69912)
2015-01-18 11:07:37,628 DEBUG [VideoDAOImpl] Reading a Video (id = 69912)
2015-01-18 11:07:37,629 DEBUG [VideoDeliveryEngine] Retrieving resource information for item 69912, format MPEG_TS_SD_NA_ISO and profile Sony BDP-Sx70/SMP-N100 (2010)
2015-01-18 11:07:37,629 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,630 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,630 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,631 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,631 DEBUG [AbstractTranscodingDeliveryEngine] Getting media info for transcoded version of file 193 The Doctor's Daughter.mkv
2015-01-18 11:07:37,631 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,632 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,632 DEBUG [VideoDeliveryEngine] Found Format profile for transcoded file "193 The Doctor's Daughter.mkv": MPEG_TS_SD_EU_ISO
2015-01-18 11:07:37,632 DEBUG [VideoDeliveryEngine] Found Format profile for transcoded file "193 The Doctor's Daughter.mkv": MPEG_TS_SD_NA_ISO
2015-01-18 11:07:37,632 DEBUG [VideoDeliveryEngine] Found Format profile for transcoded file "193 The Doctor's Daughter.mkv": MPEG_TS_SD_KO_ISO
2015-01-18 11:07:37,633 DEBUG [MediaResourceRetrievalStrategy] Getting information about media item 69912 (local)
2015-01-18 11:07:37,633 DEBUG [MediaItemDAOImpl] Reading a MediaItem (id = 69912)
2015-01-18 11:07:37,634 DEBUG [VideoDAOImpl] Reading a Video (id = 69912)
2015-01-18 11:07:37,634 DEBUG [VideoDeliveryEngine] Delivering item '69912' for client 'Identifier=10.1.33.60, Profile=Sony BDP-Sx70/SMP-N100 (2010)'
2015-01-18 11:07:37,635 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,636 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,636 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,637 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,638 DEBUG [VideoDeliveryEngine] Delivering file '193 The Doctor's Daughter.mkv' using transcoding
2015-01-18 11:07:37,638 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,639 DEBUG [SubtitlesService] Found external subtitle file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.ass
2015-01-18 11:07:37,642 DEBUG [AbstractTranscodingDeliveryEngine] No suitable transcoding job exists yet, start one for client 'Identifier=10.1.33.60, Profile=Sony BDP-Sx70/SMP-N100 (2010)'
2015-01-18 11:07:37,645 DEBUG [MediaItemDAOImpl] Getting file of media item 69912
2015-01-18 11:07:37,647 DEBUG [FFMPEGWrapper] Invoking FFmpeg to transcode video file: /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.mkv
2015-01-18 11:07:37,648 DEBUG [ProcessExecutor] Starting ffmpeg -threads auto -i /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor's Daughter.mkv -y -threads auto -c:v mpeg2video -pix_fmt yuv420p -b:v 17000k -maxrate:v 17000k -bufsize:v 17000k -vf subtitles=filename='/mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor\\\\\\\'s Daughter.ass':original_size=1872x1080:charenc=UTF-8 -r 24000/1001 -g 15 -c:a ac3 -b:a 320k -ac 2 -map 0:0 -map 0:1 -sn -f mpegts /tmp/Serviio/transcoding-temp-69912-bdp2010-ORIGINAL-546e4d11f7a7bd0a3a564e4e3873942b.stf
2015-01-18 11:07:37,730 WARN  [ProcessExecutor] Process ffmpeg has a return code of 1! This is a possible error. Detailed output follows.
[AVFilterGraph @ 0x1bb4fa0] [Eval @ 0x7fff38f4d920] Undefined constant or missing '(' in 'auto'
[AVFilterGraph @ 0x1bb4fa0] Unable to parse option value "auto"
[Parsed_subtitles_0 @ 0x1bad720] Unable to open /mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor\\\s Daughter.ass
[AVFilterGraph @ 0x1bb4fa0] Error initializing filter 'subtitles' with args 'filename=/mnt/data/serviio-video/Dr Who/e. Series 4/193 The Doctor\\\\\\\s Daughter.ass:original_size=1872x1080:charenc=UTF-8'
Error opening filters!
2015-01-18 11:07:37,730 DEBUG [TranscodingJobListener] Transcoding finished; successful: false

I'm not sure if it is a result of the debug output, but it looks like there are no quotes around the .mkv file name and there are seven backslashes in front of the apostrophe in the .ass file. Later on in the log file output, the apostrophe in the .ass file seems to have been lost.

I saw an entry in the forum about a problem with apostrophes in 2013, which was fixed in 1.4, but I was still having this problem with 1.4.1 and now also with 1.5.

Hopefully you will be able to look into this issue? Please let me know if you need any additional information.

Thanks,
Eric

[atc98092]

It's definitely something to do with Linux. I have numerous files with an apostrophe in them and they play with no issues. This is from a Windows based system.

My Linux is weak, but I thought that only two backslashes (\\) were necessary to properly escape and use punctuation? It's been years since I've played much with it, so I'm guessing here. Looks like something in the script was accidently altered.

[zip]

I will look into it again.

https://bitbucket.org/xnejp03/serviio/i ... s-for-file

I have just discovered a similar issue.... having a file that was not being listed by Serviio, which had an apostrophe in the file name (the word was What's). Sat down to watch it this evening, but it wasn't showing up in serviio, but was in my samba share of the same folder. After a bit of playing around, I thought I'd remove the apostrophe from the file name and get serviio to refresh, and lo and behold it saw it.

Looking through the log, the file was -not- recognised or added by serviio when it was downloaded (earlier today) - it was as if it simply wasn't there.

I'm running serviio on Ubuntu 14.04 LTS. Serviio is on 1.4.1

Same thing with Umlauts (ä, ö, ü) in file names, both in 1.4.1.2 and 1.5.

[zip]

Same thing with Umlauts (ä, ö, ü) in file names, both in 1.4.1.2 and 1.5.

http://serviio.org/component/content/article?id=21#q9

Thank you. I set the LANG variable in /ffp/etc/profile by changing

  Code:

export LANG=en_US

to

  Code:

export LANG=en_US.utf8

I get

  Code:

~# locale -a
C
POSIX
en_US.utf8


but Serviio stays with ASCII

  Code:

INFO  [MediaServer] ------------------------------------------------------------------------
INFO  [MediaServer] Serviio DLNA media streaming server v 1.5 (rev. 19ceb8acec8c)
INFO  [MediaServer] Petr Nejedly 2009-2015
INFO  [MediaServer] http://www.serviio.org
INFO  [MediaServer]
INFO  [MediaServer] Java 1.8.0_06-Oracle Corporation arm
INFO  [MediaServer] OS Linux 2.6.31.8
INFO  [MediaServer] File encoding: US-ASCII
INFO  [MediaServer] Headless mode enabled: true
INFO  [MediaServer] User: root
INFO  [MediaServer] User home dir: /root
INFO  [MediaServer] Temp dir: /tmp
INFO  [MediaServer] ------------------------------------------------------------------------

Did I miss anything?

[zip]

follow the posts, it says something about

export LC_ALL="en_US.UTF-8"

I tried both

export LANG=en_US.utf8
export LC_ALL="en_US.UTF-8"

together, and only one of them at a time. LC_ALL alone results in no locale setting at all. LANG alone results in the same as both lines together:

  Code:

:~# locale -a
C
en_US.utf8
POSIX


I also tried LANG=en_US.UTF-8, but it results in the locale en_US (no UTF-8). It seems that LANG has to be set to en_US.utf8, and LC_ALL="en_US.UTF-8" provides just another conventional name for the same value.

None of these, however, changed the encoding for Serviio; it's always ASCII. Looks like one of those probs you get solved eventually by pure chance, usually months (or years) later. Luckily it's not a big deal, it's just the file names. All meta-data, eg track titles, are read and output correctly.

[zip]

make sure you set the locale for the shell / user running the serviio server

got it at last. Culprit was the Fonz Fun Plug 0.7 (FFP) operating in this case on a Zyxel NSA 325-v2. The solution obtains to this environment only, and is basically found here: http://forum.nas-central.org/viewtopic.php?f=249&t=14977. However, you don't need to use the test packages provided. The library update is already available via slacker.

1.
Call slacker:

  Code:

slacker -a solibs


Don't worry if the slacker user interface gives you strange characters instead of 'graphic' lines and corners, it will look fine when you're done. Anyway, install / upgrade all packages shown to the latest versions. The relevant one seems to be br2:uClibc-solibs-0.9.33.3_git-arm-3.txz, but it won't hurt to install / upgrade the others as well.

2.
Create a file called "utf8profile.sh" in ffp/etc/profile.d. Paste the following into it and save:

  Code:

# UTF-8 encoding related settings
export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="C"
export NCURSES_NO_UTF8_ACS="1"
# Aliases for compatability and better performance with UTF-8
alias screen='screen -U'
alias ls='LC_ALL=C ls -hv --color --show-control-chars --group-directories-first'
alias sort='LC_ALL=C sort'
alias grep='LC_ALL=C grep'
alias awk='LC_ALL=C awk'
alias sed='LC_ALL=C sed'
alias updatedb='LC_ALL=C updatedb'
alias mc='mc -u'


This is from the link to Nas-Central mentioned above. Don't forget to put a CR-LF on the last line. Make the file executable.

3.
In /ffp/etc edit the file called "profile". Change line #5 "export LANG=en_US" to

  Code:

export LANG=en_US.UTF-8

4.
In the Seviio start-up file, /ffp/start/serviiod.sh, make sure that under stop) you have:

  Code:

serv_kill=`ps aux | grep serviio.sh | grep -v grep | awk '{print $2}'`

– with the ending "serviio.sh"
and likewise under status)

  Code:

serv_pid=`ps aux | grep serviio.sh | grep -v grep | awk '{print $2}'`

With 1.5, this will cause Serviio to be reported falsely as "stopped" when the status is queried, but "start" and "stop" will work reliably. – No idea why that is, with 1.4.1.2 status was reported correctly.

5. Restart NAS

The Serviio log file should now report "File encoding: UTF-8". And a library refresh should then register any media files with any non-ASCII characters in their names, including the apostrophe.

Crowed to soon. Obviously I didn't restart the NAS this time but used serviiod.sh to stop and re-start. Well, it works this way, but not on a NAS reboot. Locale seems to be set for root, but start-up scripts are not executed as root. Sigh, you have to manually stop and restart.

Ah, and forget step 4 above, with the serviio.sh ending. It makes no difference. Only, without the ending the status is reported right.

Well, this is it now. Serviio using UTF-8 right after boot, on a Zyxel NSA325 v2. Tested with 3 cold boots.

First, execute steps 1 to 3 in my above post: Upgrade uClibc, create utf8profile.sh and edit profile. Then:

4. Change Serviio startup script
In /ffp/start, edit your startup script, eg serviiod.sh. At the beginning, if you like right after the variable definitions, insert:

  Code:

export LANG=en_US.UTF-8
export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="C"
export NCURSES_NO_UTF8_ACS="1"

These are the same locale variables that are also set in utf8profile.sh and profile. But they're undefined at boot time, that is, before the root user logs in. As regular profile definitions will not be invoked by the su command in step 6 below, the variables need to be set right here, as well.

5. Make the start-up script you just edited un-executable.

  Code:

chmod 666 /ffp/start/serviiod.sh

6. Have the start-up script called by another script via the su ("switch user") command.
Create a new file in /ffp/start, call it serviio-root-starter.sh or whatever. Paste the folling into it:

  Code:

#!/ffp/bin/sh
/ffp/bin/su -c "sh /ffp/start/serviiod.sh start" -l root -s /ffp/bin/sh


This is the decisive part. The start-up script will now be executed by root. Could also be done with sudo, except there is no sudo in FFP or in Zyxel's Linux 2.6.31.8 armv5tel.

7. Make the new script executable:

  Code:

chmod 777 /ffp/start/serviio-root-starter.sh

8. Edit /ffp/etc/login.defs.
In line 152 (if still original), change the value of ENV_SUPATH to include all paths required by Java and Serviio:

  Code:

ENV_SUPATH   PATH=/ffp/opt/java/bin:/ffp/sbin:/usr/sbin:/sbin:/ffp/bin:/usr/bin:/bin

Note that the first path is the Java path, i.e. the same that is also set set in /ffp/etc/profile.d/java.sh.

9. Reboot NAS.
If you don't get UTF-8 on your Zyxel now, I don't know.

Of course, zip was right suspecting the shell / user environment.

[zip]

Great, this might make a nice wiki article . Hint hint

Thank you. I'll streamline it a bit and forward it to the wiki.

I found it is much easier, su command is not necessary. The user is root at boot time already, only the locale is undefined. Therefore it is enough to insert the definitions in the start-up script (serviiod.sh), as in setp 4 above:

  Code:

export LANG=en_US.UTF-8
export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="C"
export NCURSES_NO_UTF8_ACS="1"


That's all, leave it at that. No need also to specify a path; the default system path, though without the Java path, is enough for Serviio to start up all right.

Since I have tried exporting only LANG, to no avail, it seems that UTF-8 also depends on the other three variables. And on the uClibc upgrade, of course.