Problem with embedded Greek subtitles

First of thank you for making such a high quality program! I tried many solutions as a DLNA server and yours was by far the best!

I face one problem though. Lets start with the details:

OS: Linux Mint 17.2 64bit
Renderer Device: LG 3D TV 47LW980S
Small sample video with embedded subtitles (greek and english, UTF-8 both), log file and subtitles as stand alone files (8MB zip file):
http://www.filedropper.com/sample_10

The problem is that the Greek subtitles I embed in the mkv file give weird results. Characters are garbled, they are twice as many as they should be, only punctuation marks display properly. The subtitles are in UTF-8 format and it seems to me that something goes wrong to their decoding.
If I use external greek subtitles they display fine. Same if I choose the embedded english subtitles. Embedded greek subtitles play fine when I use other means to provide the file to my TV such e.g. a USB drive.
Here are my settings (transcoding is disabled):

As I said, if I change "gre,gr,greek" to "eng,en,english" the english embedded subtitles display fine.

Also, if I choose to burn-in the greek subtitles from the settings they appear fine, but I do not want this, since 3D movies only appear to one eye (Up-Down) or appear half to one eye and half to the other (SBS).

Thank you for your time and efforts!

[zip]

Can you post the detailed log and find the ffmpeg command used to extract the subtitles? You should then be able to find the temporary subtitle file on your disk and check if it's really UTF-8

Here are 2 logs:

logs.zip

(65.83 KiB) Downloaded 279 times

The ffmpeg command that I see in the logs is:
ffmpeg -i /media/Data/temp/sample.mkv -y -an -vn -map 0:2 -c:s srt -f srt pipe:
If I run this I get the subtitles inside the console (they appear OK).

If I modify it to create a file:
ffmpeg -i /media/Data/temp/sample.mkv -y -an -vn -map 0:2 -c:s srt -f srt subs.srt
I get a UTF-8 subtitle file.

A problem I found was when I originally had my library inside /media/Data/Movies/To keep/temp
The gap in the folder name "To keep" was creating an error when I executed the above ffmpeg commands to the console.
I had to modify the command to this to make it run (I added "):
ffmpeg -i "/media/Data/Movies/To keep/temp" -y -an -vn -map 0:2 -c:s srt -f srt subs.srt
In serviio logs I didn't find a problem with the execution of this command though. Subtitles were displayed to my TV (still garbled) so I guess ffmpeg command was executed properly. Logs about it are inside the zip file from my initial post.
Changing the library to a path without gaps (/media/Data/temp/) didn't cause any improvement, still garbled subtitles.
I have to mention here that back when I installed serviio I had to place serviio installation in a gap-less path also because it didn't run at all otherwise. So it might be a good idea to check your code for proper handling of paths with gaps.

I also tried to locate where the temporary srt file was created on my disk. I found these lines in the logs:

  Code:

2015-08-13 00:39:14,470 DEBUG [ServiioHttpService] Incoming request from /192.168.1.107:42709: GET /resource/12/SUBTITLE.srt HTTP/1.1, headers = [Host: 192.168.1.108:8895,Pragma: no-cache,Cache-Control: no-cache,Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, text/html, text/xml, text/plain, application/json, application/atom+xml, application/xml, */*,Accept-Language: ko-kr, ko;q=0.8, en-us;q=0.5, en;q=0.3,transferMode.dlna.org: Interactive,Connection: Close,User-Agent: Mozilla/5.0 (compatible; LG-HttpClient-v1.0.3 UPnP/1.1; MSIE 8.0; Windows NT 5.1; LG_UA; AD_LOGON=LGE.NET; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; LG_UA; AD_LOGON=LGE.NET; LG NetCast.TV-2011)]]

However /resource/12/SUBTITLE.srt is nowhere to be found (there isn't even a /resource folder) so my guess is that it is a virtual file. The " pipe:" at the end of the ffmpeg command also indicate this.

How can I help you further to resolve this?

[zip]

If you save that UTF-8 file you created with FFmpeg next to the video with the same file name (and .srt extension), does it display them correctly? (disable enbedded subs for this test)

It came as a surprise to me, but the subtitles do not display correctly!
I attach both subtitles so you can check them yourself.
I checked the files again and I found out that both are UTF-8 BUT the one extracted with ffmpeg is plain UTF-8 while the original that plays correctly is UTF-8-BOM (I used notepad++ to check them).
I converted the ffmpeg extracted file in UTF-8-BOM format and it displayed correctly!
So I think that in order to have everything working correctly ffmpeg command should be modified to extract the subtitles in UTF-8-BOM format. Could you add this as an option to serviio (UTF-8-BOM instead of UTF-8)? I changed the "Subtitle character encoding option:" from "UTF-8" to "UTF-8-BOM" but it didn't work.

[zip]

It doesn't look like we can specify output encoding for ffmpeg. the field in the console is for the input char encoding.

I found a work around that might come handy to everyone with the same problem. Also it might give a solution to ffmpeg shortcomings.
I came across a script for linux and modified it to extract all subtitles in a folder recursively (that means everything, even in subdirectories). This script uses mkvtoolnix (https://www.bunkus.org/videotools/mkvtoolnix/).
It searches for specific words in Greek ( και | η | το ) with the command below (you can modify this line with words for your language):

  Code:

    langtest=`egrep -ic ' και | η | το ' "$subtitlename".srt.tmp`

Then it extracts these subtitles from the mkv files giving them the same name with each mkv (It also stores with alternative names all other embedded subtitles it finds).
This way I started extracting all my embedded subtitles to overcome ffmpeg problems.

Also the files extracted with mkvextract are UTF-8-BOM by default. This is also a strong indication that UTF-8-BOM might be the standard for devices, besides my LG TV, since mkvtoolnix and mkvextract are widely used. Dear zip, can you make serviio use mkvextract (if available and installed to the system) instead of ffmpeg to extract embedded subtitles?

Here is the script below. I also attach it as a zip.

  Code:

#!/bin/bash
# Extract subtitles from each MKV file in the given directory

# If no directory is given, work in local dir
if [ "$1" = "" ]; then
  DIR="."
else
  DIR="$1"
fi

# Get all the MKV files in this dir and its subdirs
find "$DIR" -type f -name '*.mkv' | while read filename
do
  # Find out which tracks contain the subtitles
  mkvmerge -i "$filename" | grep 'subtitles' | while read subline
  do
    # Grep the number of the subtitle track
    tracknumber=`echo $subline | egrep -o "[0-9]{1,2}" | head -1`

    # Get base name for subtitle
    subtitlename=${filename%.*}

    # Extract the track to a .tmp file
    `mkvextract tracks "$filename" $tracknumber:"$subtitlename.srt.tmp" > /dev/null 2>&1`
    `chmod g+rw "$subtitlename.srt.tmp"`


    # Do a super-primitive language guess: GREEK
    langtest=`egrep -ic ' και | η | το ' "$subtitlename".srt.tmp`
    trimregex=""



    # Check if subtitle passes our language filter (10 or more matches)
    if [ $langtest -ge 10 ]; then
      `mv "$subtitlename.srt.tmp" "$subtitlename.srt" > /dev/null 2>&1`
      `chmod g+rw "$subtitlename.srt"`
    else
      # Not our desired language: add a number to the filename and keep anyway, just in case
      `mv "$subtitlename.srt.tmp" "$subtitlename.$tracknumber.srt" > /dev/null 2>&1`
    fi
  done
done

Even better, I found the following command that converts a UTF-8 file to UTF-8-BOM!

  Code:

sed -i '1s/^/\xef\xbb\xbf/' utf8_subtitle.srt

This will make it easier to add this as an option to serviio.

Edit:
BOM is the standard for windows as I found out.
With some more searching I found this that serves both lnux and windows:
https://code.google.com/p/utf-bom-utils/

[zip]

I created a ticket: https://bitbucket.org/xnejp03/serviio/i ... -subtitles

Thank you for looking into it!