Testing v= 2.340 for LED wall (Problem: vertical larger than 2.160 px / 16:9 UHD-1) on XT1144 (4th Gen) / XT1145 (5th Gen). Since the tutorials I found were lacking simple steps to follow here is some documentation for custom resolution in General and also specific tests for v > 2.160.
A. Apply custom resolution to a (XT1144, FW 8.5.47) player, single screen
1. Script from Github for single screen: https://github.com/brightsign/video-mode-plugin/blob/master/SetSingleScreenVideoMode.brs#L1 (Download as .brs File on 3 dots top right). Multiple output script not tested.
Use a text editor to change the Modline to the HDMI timing you would like the BS-Player to take on output: https://tomverbeure.github.io/video_timings_calculator (I used CVT-RB / reduced blanking for my tests).
Example: to archive 1.920 x 2.340 I replaced the existing line in the .brs-File with the following:
vm.SetCustomModeline("Modeline 1920x2340_4996 249 1920 1968 2000 2080 2340 2343 2353 2396 +hsync -vsync").
2. In BrightAuthor 5.0.2.0 Presentation Settings you need to make 2 changes to your project:
- Main > Force Resolution: no (not 100% tested if force = yes might also work)
- Autorun > Video Mode Plugins (bottom) > Function Name: SetVideoMode Path: select the modified SetSingleScreenVideoMode.brs above.
3. Now when you push the presentation to the player it boots with the Timing you specified via the Modline. It seems to ignore the EDID from the attached display. The XT1144 successfully shows 1.920 x 2.340p/49.97 to the Barco ImagePro 4K connected (see picture below), the output downscaled to an UHD-1 monitor also looked fine. A live input via HDMI displayed correctly as well.

BUT: since Brightauthor only works within the Canvas you selected it vertically maxes out at something_h x 2.160. Using a XT1145 project in UHD-2 to push to the XT1144 resulted in garbage on HDMI output.
You can see the defined BG color on the full 1.920 x 2.340 pixels but Images/Video including Live input can only be moved within the Canvas (and Zone) from your project (which is max = 2.160 on XT1144 projects).
B. v = 2.340 on XT1145
5th gen Player with UHD-2 is on its way, I will update my results here. Based on what we learned with the XT1144 I expect the custom resolution script to work, together with a BrightAuthor Canvas of 7.680 x 4.320 where we use the top left corner as a zone to work with 1.920 x 2.340 using videos, stills and HDMI live input.