Merge lp://qastaging/~jameinel/bzr/2.1-client-reconnect-819604 into lp://qastaging/bzr/2.1

Start of bug #819604, allow bzr clients to reconnect if the connection is gone when we go to write a new request.

This is some initial work for having clients reconnect when their bzr+ssh (or bzr://) connection gets closed underneath them.

To start with, the trickiest bit is that it is really quite hard to detect when things are closed. Mostly because of buffering, etc, at lots of different levels (internal memory buffering, socket/pipe buffering, 'ssh.exe' buffering, latency of close message from server back to client, etc.)

I'll also clarify that this isn't intended to handle all possible sorts of connection failures. What we really are trying to handle is allowing a server to be upgraded 'live'. Such that the server will gently disconnect us only between complete requests. Versus having requests terminated randomly in the middle of content. It may help with some of those cases, and we certainly don't want to cause corruption, etc if one of those happens. But it isn't the goal of *this* change.

This specific patch has a few aspects, and does help some cases in real-world testing.

1) Backport a minimum amount of the SmartSSHClientMedium proxies to _real_medium patch from Andrew. In bzr-2.2 Andrew updated the code so that when spawning an 'ssh' subprocess, we use a socketpair when possible instead of pipes. (That way we can read without blocking, allowing us to use large buffers, etc.)

It also meant that he shared the logic from SmartSimplePipeStreamMedium and SmartTCPClientMedium, which I wanted to do, to avoid having to re-implement the logic multiple times.

This is a pretty small part of the patch.

2) Update some of the lower level code so that we get ConnectionReset rather than various IOError and ValueError when writing to a closed connection.

3) Update the _SmartClient._send_request logic. If we get a ConnectionReset while we are trying to write the request, then we can safely retry the request. On the assumption that as long as the server doesn't see the final 'e' terminal byte, it will reject the request, because it is incomplete.

4) The one caveat is if there is a 'body_stream'. body_stream is an iterable, so we can't just rewind it and resume it.

On the plus side, there is only one caller RemoteStreamSink.insert_stream. In theory we could update that caller. In practice, it *also* takes a 'stream' object, which can't simply be rewound. Though we do a 'no-op' stream just before the real content to determine that the remote server actually supports the RPC we want to use.

That would allow us to close the gap a little bit. So we can detect ConnectionReset all the way to the point that we actually start streaming the content.

The other bit that would be possible, is to update ProtocolThreeRequester.call_with_body_stream to allow it to set a flag to indicate whether it has actually started consuming the stream yet. Then update it to flush just before it iterates the body stream to help force detecting closed connections. (otherwise we are likely to have buffered at least the first part of the body_stream in local memory, since the local buffer is 1MB and the first flush is only called after the first chunk of the body_stream.)

This just closes a window where we won't reconnect during 'bzr push'.

5) The next step is that (especially for sockets), we write out the request successfully, and then notice it is closed when we try to read back the response.

Here I'm a lot more concerned about non-idempotent requests. Because the server might have fully read the request, and sent back a response, but we didn't get to see the response.

However, our RPC design is meant to be stateless (all state is passed in the request, not assumed to be stored on the server side). Which means things *should* be idempotent. For something like 'put_bytes', we may write the same content twice, but we shouldn't corrupt anything.

The only one I can think of is 'append_bytes', and the associated open_write_stream() code. (RemoteTransport.open_write_stream uses AppendBasedFileStream as its implementation.)

And that code shouldn't really be used with up-to-date bzr clients and servers. So I'm much more ok with it just dying if it isn't safe to retry.