Unless you have a commitment to only using open source software, I’d recommend Plex over Jellyfin. Mostly because I’ve found the client software for Jellyfin to be lacking, especially on AppleTV.
For the issues with the GoogleTV, you mention that it’s on WiFi, would it be possible to use a wired connection or get another set top box for it? Some TVs have the WiFi antenna behind the screen causing interference, so even though other devices get a strong signal the TV doesn’t.
Also, how’s the hardware on your server? Is the CPU powerful enough or do you have a GPU for transcoding? Also, is the server on WiFi or wired?
It’s worth noting that a lot of settop boxes have limited codec support, which might be forcing transcoding even if everything should otherwise support direct play.
Ah, it’s one of those. I was thinking it was the TV’s OS.
I don’t have a lot of experience with those smaller dongles, but as I understand it they’re fairly low power devices that are more meant for streaming relatively low bitrate media from the internet or from a phone. It may not have the horsepower for playing back high bitrate media from Jellyfin or Plex.
Others may have a solution that’ll work for that device, but my gut response is to say you should consider replacing it with something more powerful.