Bug#913237: xterm: exec-formatted yields a tilde character in zsh and emacs

[Vincent Lefevre]

Package: xterm
Version: 337-1
Severity: normal

In my XTerm configuration, I have:

*VT100*translations: #override \n\
Meta<Btn1Down>: exec-formatted("browser %s", PRIMARY)

The problem is that when exec-formatted is invoked from zsh or emacs
(when run in xterm, e.g. with "emacs -nw"), a ~ character appears.

Other shells such as dash, bash and ksh are not affected. Curses
applications such as mutt and tack are not affected either.

-- System Information:
Debian Release: buster/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.18.0-2-amd64 (SMP w/12 CPU cores)
Locale: LANG=POSIX, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=POSIX (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages xterm depends on:
ii libc6 2.27-8
ii libfontconfig1 2.12.6-0.1
ii libfreetype6 2.6.3-3.2
ii libice6 2:1.0.9-2
ii libtinfo6 6.1+20181013-1
ii libutempter0 1.1.6-3
ii libx11-6 2:1.6.7-1
ii libxaw7 2:1.0.13-1+b2
ii libxft2 2.3.2-2
ii libxinerama1 2:1.1.4-1
ii libxmu6 2:1.1.2-2
ii libxpm4 1:3.5.12-1
ii libxt6 1:1.1.5-1
ii xbitmaps 1.1.1-2

Versions of packages xterm recommends:
ii x11-utils 7.7+4

Versions of packages xterm suggests:
pn xfonts-cyrillic <none>

-- no debconf information

[Thomas Dickey]

Post by Vincent Lefevre
Package: xterm
Version: 337-1
Severity: normal
*VT100*translations: #override \n\
Meta<Btn1Down>: exec-formatted("browser %s", PRIMARY)
The problem is that when exec-formatted is invoked from zsh or emacs
(when run in xterm, e.g. with "emacs -nw"), a ~ character appears.
Other shells such as dash, bash and ksh are not affected. Curses
applications such as mutt and tack are not affected either.

I don't see how this could happen unless you combined the action with
some pasting (such as bracketed-paste). xterm's formatting of the
string is shell-agnostic, and the exec'd "browser" command would only
depend on what the formatted URL looked like.

In either case, steps-to-reproduce seem obscure.

[Vincent Lefevre]

Post by Thomas Dickey
I don't see how this could happen unless you combined the action with
some pasting (such as bracketed-paste).

I paste nothing.

Post by Thomas Dickey
xterm's formatting of the string is shell-agnostic, and the exec'd
"browser" command would only depend on what the formatted URL looked
like.
In either case, steps-to-reproduce seem obscure.

With zsh, one can reproduce the issue with:

$ xterm -e zsh -f

then in the xterm:

zira% bindkey -e

What's strange is that in the Xterm log, I can get either ^G~, or
just ~, or nothing.

[Thomas Dickey]

Post by Vincent Lefevre

Post by Thomas Dickey
I don't see how this could happen unless you combined the action with
some pasting (such as bracketed-paste).

I paste nothing.

Post by Thomas Dickey
xterm's formatting of the string is shell-agnostic, and the exec'd
"browser" command would only depend on what the formatted URL looked
like.
In either case, steps-to-reproduce seem obscure.

$ xterm -e zsh -f

If you added a "-l" option, that would turn on xterm's logging feature

xterm -l -e zsh -f

which could be interesting. But the bug report deals with programs
run from xterm, which the shell wouldn't see -- unless it's reading
xterm's output in some way.

Post by Vincent Lefevre
zira% bindkey -e

Perhaps "zira%" is your shell prompt.

What does "bindkey -e" have to do with exec-formatted?

[Vincent Lefevre]

Post by Thomas Dickey

Post by Vincent Lefevre
$ xterm -e zsh -f

If you added a "-l" option, that would turn on xterm's logging feature
xterm -l -e zsh -f
which could be interesting.

Here's what the log file gives (output with hd):

00000000 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
00000010 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000060 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000070 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000080 4b 1b 5b 3f 32 30 30 34 68 62 08 62 69 6e 64 6b |K.[?2004hb.bindk|
00000090 65 79 20 2d 65 1b 5b 3f 32 30 30 34 6c 0d 0d 0a |ey -e.[?2004l...|
000000a0 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
000000b0 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
000000c0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000100 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000110 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000120 4b 1b 5b 3f 32 30 30 34 68 07 7e |K.[?2004h.~|
0000012b

The 07 7e is what the exec-formatted generates (the Ctrl-G doesn't
seem to have any effect in zsh).

If after "binkey -e", I type "echo '" then Ctrl-V, I get:

00000120 4b 1b 5b 3f 32 30 30 34 68 65 08 65 63 68 6f 20 |K.[?2004he.echo |
00000130 27 1b 5b 37 6d 5e 5b 1b 5b 32 37 6d 5b 32 30 31 |'.[7m^[.[27m[201|
00000140 7e |~|

So, actually, the effect of exec-formatted is more complex than just
a tilde (or Ctrl-G tilde).

Post by Thomas Dickey
But the bug report deals with programs run from xterm, which the
shell wouldn't see -- unless it's reading xterm's output in some
way.

The shell sees the tilde, as if it were entered with the keyboard.

Note that I can replace

exec-formatted("browser %s", PRIMARY)

by

exec-formatted("/bin/true", PRIMARY)

and I get the same behavior in xterm, i.e. the command doesn't matter.

Post by Thomas Dickey

Post by Vincent Lefevre
zira% bindkey -e

Perhaps "zira%" is your shell prompt.

Yes, the hostname follwed by "%" is the default zsh prompt.

Post by Thomas Dickey
What does "bindkey -e" have to do with exec-formatted?

"bindkey -e" selects Emacs editing mode. It just makes the issue
visible. But with the "echo '" + Ctrl-V test, the same issue is
reproducible without needing "bindkey -e":

00000000 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
00000010 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000060 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000070 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000080 4b 1b 5b 3f 32 30 30 34 68 65 08 65 63 68 6f 20 |K.[?2004he.echo |
00000090 27 5e 08 1b 5b 37 6d 5e 1b 5b 37 6d 5b 1b 5b 32 |'^..[7m^.[7m[.[2|
000000a0 37 6d 5b 32 30 31 7e |7m[201~|
000000a7

The effect of exec-formatted is what appears after "27 5e 08".

[Thomas Dickey]

Post by Vincent Lefevre

Post by Thomas Dickey

Post by Vincent Lefevre
$ xterm -e zsh -f

If you added a "-l" option, that would turn on xterm's logging feature
xterm -l -e zsh -f
which could be interesting.

00000000 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
00000010 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000060 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000070 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000080 4b 1b 5b 3f 32 30 30 34 68 62 08 62 69 6e 64 6b |K.[?2004hb.bindk|
00000090 65 79 20 2d 65 1b 5b 3f 32 30 30 34 6c 0d 0d 0a |ey -e.[?2004l...|
000000a0 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
000000b0 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
000000c0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000100 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000110 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000120 4b 1b 5b 3f 32 30 30 34 68 07 7e |K.[?2004h.~|
0000012b
The 07 7e is what the exec-formatted generates (the Ctrl-G doesn't
seem to have any effect in zsh).

exec-formatted doesn't send a ^G, it simply does an execvp() for the arguments.

The escape "[?2004h" is turning on bracketed paste (which I asked about).

Post by Vincent Lefevre
00000120 4b 1b 5b 3f 32 30 30 34 68 65 08 65 63 68 6f 20 |K.[?2004he.echo |
00000130 27 1b 5b 37 6d 5e 5b 1b 5b 32 37 6d 5b 32 30 31 |'.[7m^[.[27m[201|
00000140 7e |~|
So, actually, the effect of exec-formatted is more complex than just
a tilde (or Ctrl-G tilde).

Post by Thomas Dickey
But the bug report deals with programs run from xterm, which the
shell wouldn't see -- unless it's reading xterm's output in some
way.

The shell sees the tilde, as if it were entered with the keyboard.

The shell is using bracketed paste, and is reading xterm's output,
as I suggested.

Post by Vincent Lefevre
Note that I can replace
exec-formatted("browser %s", PRIMARY)
by
exec-formatted("/bin/true", PRIMARY)
and I get the same behavior in xterm, i.e. the command doesn't matter.

so pasting works fairly consistently :-)

Post by Vincent Lefevre

Post by Thomas Dickey

Post by Vincent Lefevre
zira% bindkey -e

Perhaps "zira%" is your shell prompt.

Yes, the hostname follwed by "%" is the default zsh prompt.

Post by Thomas Dickey
What does "bindkey -e" have to do with exec-formatted?

"bindkey -e" selects Emacs editing mode. It just makes the issue
visible. But with the "echo '" + Ctrl-V test, the same issue is
00000000 1b 5b 31 6d 1b 5b 37 6d 25 1b 5b 32 37 6d 1b 5b |.[1m.[7m%.[27m.[|
00000010 31 6d 1b 5b 6d 20 20 20 20 20 20 20 20 20 20 20 |1m.[m |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
*
00000060 20 20 20 20 0d 20 0d 0d 1b 5b 6d 1b 5b 32 37 6d | . ...[m.[27m|
00000070 1b 5b 32 34 6d 1b 5b 4a 7a 69 72 61 25 20 1b 5b |.[24m.[Jzira% .[|
00000080 4b 1b 5b 3f 32 30 30 34 68 65 08 65 63 68 6f 20 |K.[?2004he.echo |
00000090 27 5e 08 1b 5b 37 6d 5e 1b 5b 37 6d 5b 1b 5b 32 |'^..[7m^.[7m[.[2|
000000a0 37 6d 5b 32 30 31 7e |7m[201~|
000000a7

That looks as expected, if you've got two different things writing to
the terminal at the same time:

https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Bracketed-Paste-Mode

but the execvp command by itself doesn't do the writing

Post by Vincent Lefevre
The effect of exec-formatted is what appears after "27 5e 08".