Large pastes into readline enabled programs causes breakage fromv2.6.31 onwards

[Margarita Manterola]

Hi,

The problem:
Large pastes (5k or more) into a readline enabled program fail when
running kernels larger than v2.6.31-rc5.  "Fail" means that some lines
are incomplete.  From v2.6.39-rc1 onwards, "some lines" become "almost
all lines after the first 4k".  This turns up at least in Fedora,
Debian, Ubuntu and Gentoo.  From our findings, it should happen in any
readline enabled program on a system running kernels later than the
mentioned ones.

The problematic commits in the kernel tree:
1 - 2009-07-27 (never shipped) -
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=3a54297478e6578f96fd54bf4daa1751130aca86

After this commit, pastes start breaking.  For a 35k file, about 50%
of the times one or two lines are partially incomplete.

2 - 2009-07-29 (v2.6.31-rc5) -
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=e043e42bdb66885b3ac10d27a01ccb9972e2b0a3

This commit reverts the previous one, but adds one extra call to
flush_to_ldisc.  Pastes still break, commenting out the function call
prevents breakage *up to 2.6.39-rc1*.

3 - 2011-03-22 (v2.6.39-rc1) -
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f23eb2b2b28547fc70df82dd5049eb39bec5ba12

This commit changes many schedule/flush/cancel_delayed_work calls into
schedule/flush/cancel_work.  After this commit, the big breakage
starts: for the 35k example file, it starts breaking at aprox. 4k and
then every line is partially incomplete or directly not there.

Still after this commit, commenting out the tty_flush_to_ldisc(tty)
call added by e043e42bdb66885b3ac10d27a01ccb9972e2b0a3 prevents the
breakage.

4 - 2011-04-04 (v2.6.39-rc2) -
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a5660b41af6a28f8004e70eb261e1202ad55c5e3

This commit modifies the behaviour of how the ttys are polled.  After
this commit, commenting out the tty_flush_to_ldisc(tty) call doesn't
prevent breakage anymore.

But then re-adding the call to schedule_work(&tty->buf.work) that was
removed in this commit, prevents the breakage even up to 3.11-rc2.
I'm attaching a diff of the patch that we applied, just to show what
had to be done, this is not a proposed fix, because this does cause a
busy loop that is particularly noticeable in VMs.

We haven't yet found a good fix for this issue, but we believe that
pasting into readline enabled programs shouldn't cause characters to
get lost, and it should be possible to do that properly without the
busy loop.

***
This was originally reported as a bug in readline, but it was found
that going back to very old kernels prevented the issue, regardless of
the version of readline.

Original Report (2012-06-25):
http://lists.gnu.org/archive/html/bug-readline/2012-06/msg00006.html
Follow Up thread (2013-07-22):
http://lists.gnu.org/archive/html/bug-readline/2013-07/msg00006.html

I'm attaching here a very simple readline enabled program that helps
with performing tests.  Compile, run, then copy and paste a large
enough file into it, close and diff.

Looking at the code in readline, the issue is triggered by these lines
in rltty.c, while preparing the input:

tiop->c_lflag &= ~(ICANON | ECHO);
(...)
tiop->c_iflag &= ~(ICRNL | INLCR);

If those two lines are replaced by:

tiop->c_lflag &= ~(ECHO);
(...)
tiop->c_iflag &= ~(INLCR);

Then the pastes work fine: no lines are missing.  Of course, this
means that readline doesn't work properly, but this is just to note
that those are the terminal settings that cause the issue to pop-up.

Credit: this investigation was done together with Maximiliano Curia.

-- 
Regards,
Margarita Manterola
/* mini program to demonstrate problem with cut and paste,  Frank Lübeck */
/* Simplified by Margarita Manterola. */
/* Compile:
 *    gcc -Wall -o minirl -O2 minirl.c -lreadline
 *    gcc -Wall -o minirl -O2 minirl.c $CPPFLAGS $LDFLAGS -lreadline -lncurses
 * Run:
 *    ./minirl TESTOUT
 * and paste the content of any text file (say 10kB without long lines),
 * end program with Ctrl-d.
 *
 * Then
 *    diff original-file TESTOUT
 * shows that in some lines trailing characters get lost. Or lines completely
 * lost.
*/

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <readline/readline.h>

int main(int argc, char** argv) {

  char* s;
  FILE *out;

  if (argc == 1) return 1;
  out = fopen(argv[1], "w");
  while (1) {
    s = readline("> ");
    if (!s) break;
    fprintf(out,"%s\n",s);
    free(s);
  }
  fflush(out);
  return 0;
}

Attachment: prevent_readline_paste_breakage.diff
Description: Binary data