OK, I've been through the Roku profiles, and I can't figure out what's triggering transcoding for your files. There's only one wildcard that transcodes all H.265, and it's in the "Roku 1080 Media Player (2016+)" and the "Roku TV (non 4K)" profile, neither of which you are using. Every other Matches line has an audio qualifier, which doesn't match your files.
What I'd like to ask takes a little time, but it might help identify what's going on. If you enable debug logging, I might be able to see exactly where in the profile it's getting caught. See the instructions in the link to turn debug logging on and off. You don't want to leave it on, as your log files get huge quickly. Once you've enabled it, stop the Serviio service (it can be stopped in the Windows Services app), delete your current log file (or rename it if you'd like to keep it), then start Serviio back up. Now, attempt to play one of the files that gives you a problem. Once the issue happens, go ahead and stop the Serviio service again. Now save the log file somewhere else, disable debug, and start Serviio again. Now post that log file as an attachment here, and also tell me the exact file name of the video you tried to play. Makes it faster to search the large log.
If the file is too large to attach, you can right click the file and select Send To... Compressed (zipped) Folder. This will shrink a plain text file like the log drastically, and should make it small enough to attach here.
Dan
LG NANO85 4K TV, Samsung JU7100 4K TV, Sony BDP-S3500, Sharp 4K Roku TV, Insignia Roku TV, Roku Ultra, Premiere and Stick, Nvidia Shield, Yamaha RX-V583 AVR.
Primary server: Intel i5-6400, 16 gig ram, Windows 10 Pro, 22 TB hard drive space | Test server Windows 10 Pro, AMD Phenom II X4 965, 8 gig ram
HOWTO: Enable debug logging HOWTO: Identify media file contents