display corruption on many Unicode characters

So JOE's idea of double-wide characters matches xterm in Ubuntu 16.04 (xterm -version says XTerm(322)). When I paste your examples into JOE running in an xterm, editing works properly. However only two symbols show up, and they are single-wide: the musical notes and the baseball home plate. All of the rest are not in the font, but are treated as single wide. (If you think what I'm saying here is not true, please let me know!)

The default font Gnome-terminal in 16.04 is using ("Monospace Regular 12") has these characters, but gnome-terminal does a weird broken thing: It displays them wider than a normal character, but advances the cursor as if they were single-wide, so the cursor ends up in the middle of the character. So if you paste "🔊A", the A is displayed on top of the right half of the speaker symbol.

So what does VIM do? VIM seems to know that these characters are wide, but because gnome-terminal is broken, they are displayed correctly only sometimes. As you type them in it displays correctly, but if you hit ^L refresh they are messed up (the following character overlaps). VIM in xterm is also somewhat broken, in that the cursor column number doesn't match what it is on the screen.

Anyway, so I have no good fix for this. It would be nice if we can query the terminal emulator for the width of the character and take the font width knowledge out of the editor- actually this would work in xterm: print the character, then use "ESC [ 6 n" to have it report the cursor position. You could imagine having the editor build up a database of known widths this way. Unforutnately because gnome-terminal is not advancing the cursor properly, this would not work for it, so it's not a universal solution.

Does anybody have any suggestions for this?

Is this not related to [#363]?

Yeah, so I've updated JOE to allow you select which unicode version you want to build it with. I don't think it will solve the above problem with gnome-terminal in Ubuntu 16.04... but perhaps they fixed it in more recent versions of Ubuntu.

I've also added a program: joe/util/checkwidths
It determines the width of each character by printing it to the terminal emulator and reading back the cursor position.
It generates a table of widths on stderr.

Anyway I'm marking this closed. There is not much JOE can do if the character widths of the font used by the terminal emulator do not match widths specified in EastAsianWidth.txt.

I've reached out to the original submitter and here are their updated comments:

The bug stems from joe trying to reinvent its own wcwidth() -- everyone else
instead uses the definition from glibc, which at least makes it consistent
between producers and consumers as long as you don't cross network lines,
have the same release on both sides, or don't use characters that have just
been added to Unicode.

Thus, all other editors I've tried, on any modern terminal, handle it well:
cursor movement, display, editing. Worst case, a glyph that's wide in a
font is considered to be single-width, but that's at most temporary spillage
or clipping.

Also, joe uses EastAsianWidth.txt, this has just been explicitly disallowed
by the Unicode standard: https://www.unicode.org/reports/tr11/tr11-34.html