Merge lp://qastaging/~ipython-dev/ipython/inputhook into lp://qastaging/ipython/0.11

New features
------------
This branch contains a radical refactor of how IPython integrates with GUI event loop. While it has been tested extensively, it will require a bit of polishing, but we want to get this into trunk ASAP so people can begin to play with it at SciPy.

* Added a new module :mod:`IPython.lib.inputhook` to manage the integration
  with GUI event loops using `PyOS_InputHook`. See the docstrings in this
  module for details.

Bug fixes
---------

Backwards incompatible changes
------------------------------

* Removed :mod:`shellglobals` as it was obsolete.
* Removed all the threaded shells in :mod:`IPython.core.shell`. These are no
  longer needed because of the new capabilities in
  :mod:`IPython.lib.inputhook`.
* Should we remove the old threading flags like `-pylab`, `-qthread`, etc.?

I have done more work to clean up the user interface. Here is the latest changes:

New features
------------

* Added a new module :mod:`IPython.lib.inputhook` to manage the integration
  with GUI event loops using `PyOS_InputHook`. See the docstrings in this
  module for details.
* For users, GUI event loop integration is now handled through the new
  :command:`%gui` magic command. Type ``%gui?`` at an IPython prompt for
  documentation.

To use matplotlib with the new GUI event loop integration do the following
in IPython::

    %gui -a wx
    import matplotlib
    matplotlib.use('wxagg')
    from matplotlib import pyplot
    pyplot.interactive(True)
    # Then do your plotting

Bug fixes
---------

* Keyboard interrupts now work with GUI support enabled across all platforms
  and all GUI toolkits reliably.

Backwards incompatible changes
------------------------------

* Removed :mod:`shellglobals` as it was obsolete.
* Removed all the threaded shells in :mod:`IPython.core.shell`. These are no
  longer needed because of the new capabilities in
  :mod:`IPython.lib.inputhook`.
* The old threading command line flags (pylab/wthread/etc.) have been
  deprecated. Use :mod:`IPython.inputhook` or the new :command:`%gui` magic
  command instead.

I'm fried so the notes below are less complete than they should... But I want them in now, in case you see this tomorrow early before I'm up, we can talk about it further later.

Notes on the review of the inputhook branch...

- shouldn't -a be the default for %gui?

- why %gui tk? I thought it wasn't really needed since Python already does
  this...
- why %gui with no args? What does it mean?

- Small tests with standalone wx/gtk/qt codes besides matplotlib? It would be
  good to have little snippets with each toolkit in our test suite, even if we
  need to run those by hand each time. At least knowing where to find them
  would help.

- Is there no support for qt3?

- %gui gtk appears to be broken:

/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/inputhook.pyc in enable_gtk(self, app)
    177 # For older versions of gtk, use our own ctypes version

    178 from IPython.lib.inputhookgtk import inputhook_gtk
--> 179 add_inputhook(inputhook_gtk)
    180
    181 def disable_gtk(self):

NameError: global name 'add_inputhook' is not defined

- My little test script crashes with wx, if I hit Ctrl-C before it's finished.
  Here is the code, just put it in a file (I called it 'switchgui') and run it
  as indicated in the docstring.

"""Test the new %gui command. Run this in ipython as

%run switchgui [backend]

where the optional backend can be one of: qt4, gtk, tk, wx.
"""

import sys
import time

import matplotlib

try:
    import IPython.core.ipapi as ipapi
except ImportError:
    import IPython.ipapi as ipapi
    ip = ipapi.get()
else:
    ip = ipapi.get()
    try:
        guitk = sys.argv[1]
        ip.magic('gui -a %s' % guitk)
    except IndexError:
        guitk = ''
    backends = dict(wx='wxagg', qt4='qt4agg', gtk='gtkagg', tk='tkagg')
    backends[''] = 'tkagg'
    matplotlib.use(backends[guitk])
    matplotlib.interactive(True)

from matplotlib import pyplot as plt, mlab, pylab
import numpy as np

from numpy import *
from matplotlib.pyplot import *

x = np.arange(10)

print "A plot has been created"
plot(x)
plt.show()

print "I will now count until 10, please hit Ctrl-C before I'm done..."
print "IPython should stop counting and return to the prompt without crashing."
print
for i in range(1,11):
    print i,
    sys.stdout.flush()
    time.sleep(1)

> Notes on the review of the inputhook branch...
>
> - shouldn't -a be the default for %gui?
>

Probably.

- why %gui tk? I thought it wasn't really needed since Python already does
>

Just to provide a uniform interface. Also, when %gui is called, I
internally set a variable that points to the current gui, and it can be
retrieved using:

from IPython.lib.inputhook import current_gui
current_gui()

> this...
> - why %gui with no args? What does it mean?
>

It removes all gui event loop support.

> - Small tests with standalone wx/gtk/qt codes besides matplotlib? It would
> be
> good to have little snippets with each toolkit in our test suite, even if
> we
> need to run those by hand each time. At least knowing where to find them
> would help.
>

Yes, these should go into examples.

> - Is there no support for qt3?
>

Nope, it shouldn't be difficult to add, but I am not going to do it right
now.

> - %gui gtk appears to be broken:
>
> /home/fperez/usr/lib/python2.6/site-packages/IPython/lib/inputhook.pyc in
> enable_gtk(self, app)
> 177 # For older versions of gtk, use our own ctypes version
>
> 178 from IPython.lib.inputhookgtk import inputhook_gtk
> --> 179 add_inputhook(inputhook_gtk)
> 180
> 181 def disable_gtk(self):
>
> NameError: global name 'add_inputhook' is not defined
>

Ooops, I just fixed this (haven't pushed yet).

> - My little test script crashes with wx, if I hit Ctrl-C before it's
> finished.
> Here is the code, just put it in a file (I called it 'switchgui') and run
> it
> as indicated in the docstring.
>

There are a couple of issues with this:

* The thing that triggers the event loop to run is something being ready on
stdin. Thus, because your script doesn't have anything ready on stdin, the
event loop doesn't run.

* Another way of saying it is this: gui event loop is in the main thread.
Your code is in the main thread (it has to be because it is using the
gui!). Thus, as long as user code is running, the gui can't run.

We might be able to find a way of getting the event loop to run by piping
something to stdin though. We should look at this.

>
> """Test the new %gui command. Run this in ipython as
>
> %run switchgui [backend]
>
> where the optional backend can be one of: qt4, gtk, tk, wx.
> """
>
> import sys
> import time
>
> import matplotlib
>
> try:
> import IPython.core.ipapi as ipapi
> except ImportError:
> import IPython.ipapi as ipapi
> ip = ipapi.get()
> else:
> ip = ipapi.get()
> try:
> guitk = sys.argv[1]
> ip.magic('gui -a %s' % guitk)
> except IndexError:
> guitk = ''
> backends = dict(wx='wxagg', qt4='qt4agg', gtk='gtkagg', tk='tkagg')
> backends[''] = 'tkagg'
> matplotlib.use(backends[guitk])
> matplotlib.interactive(True)
>
> from matplotlib import pyplot as plt, mlab, pylab
> import numpy as np
>
> from numpy import *
> from matplotlib.pyplot import *
>
> x = np.arange(10)
>
> print "A plot has been created"
> plot(x)
> plt.show()
>
> print "I will now count until 10, please hit Ctrl-C before I'm done..."
> print "IPython should stop counting and return to the prompt without
> ...

On Fri, Aug 21, 2009 at 7:45 PM, Brian Granger<email address hidden> wrote:

> * Another way of saying it is this:  gui event loop is in the main thread.
> Your code is in the main thread (it has to be because it is using the
> gui!).  Thus, as long as user code is running, the gui can't run.
>
> We might be able to find a way of getting the event loop to run by piping
> something to stdin though.  We should look at this.

Unlikely, as event loop is run when you are doing raw_input
(regardless of whether you have data there or not). If you are not
entering raw_input part of repl, your event loop is starving. No easy
way around that.

--
Ville M. Vainio
http://tinyurl.com/vainio

> Unlikely, as event loop is run when you are doing raw_input
> (regardless of whether you have data there or not). If you are not
> entering raw_input part of repl, your event loop is starving. No easy
> way around that.
>

Yes this is right. Ignore what I said before it was wrong. Here is a
modified version of Fernando's script that does work.

"""Test the new %gui command. Run this in ipython as

%gui -a wx # select the gui here
%run switchgui
"""

import sys
import time

import matplotlib

from IPython.lib.inputhook import current_gui

guitk = current_gui()
if guitk is None:
    raise RuntimeError('You must enable gui event loop first using %gui')

backends = dict(wx='wxagg', qt4='qt4agg', gtk='gtkagg', tk='tkagg')
matplotlib.use(backends[guitk])
matplotlib.interactive(True)

from matplotlib import pyplot as plt, mlab, pylab
import numpy as np

from numpy import *
from matplotlib.pyplot import *

x = np.arange(10)

print "A plot has been created"
plot(x)
plt.show()

raw_input("Press Enter to continue")

# The plot won't show up here as the event loop only runs when the
# command prompt is sitting and doing nothing.

print "I will now count until 10, please hit Ctrl-C before I'm done..."
print "IPython should stop counting and return to the prompt without
crashing."
print
for i in range(1,11):
   print i,
   sys.stdout.flush()
   time.sleep(1)

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

On Fri, Aug 21, 2009 at 10:00 AM, Brian Granger<email address hidden> wrote:
>
> raw_input("Press Enter to continue")

Mmh, still segfaults:

In [1]: run switchgui.py wx
A plot has been created
Press Enter to continue

I will now count until 10, please hit Ctrl-C before I'm done...
IPython should stop counting and return to the prompt without crashing.

1 2^C
maqroll[ipython]> # OOPS, why am I here? segfault...

Did you do:

In [1]: %gui -a wx

First?

On Fri, Aug 21, 2009 at 11:33 AM, Fernando Perez <email address hidden>wrote:

> On Fri, Aug 21, 2009 at 10:00 AM, Brian Granger<email address hidden>
> wrote:
> >
> > raw_input("Press Enter to continue")
>
> Mmh, still segfaults:
>
> In [1]: run switchgui.py wx
> A plot has been created
> Press Enter to continue
>
> I will now count until 10, please hit Ctrl-C before I'm done...
> IPython should stop counting and return to the prompt without crashing.
>
> 1 2^C
> maqroll[ipython]> # OOPS, why am I here? segfault...
> --
> https://code.launchpad.net/~ipython-dev/ipython/inputhook/+merge/10127<https://code.launchpad.net/%7Eipython-dev/ipython/inputhook/+merge/10127>
> You proposed lp:~ipython-dev/ipython/inputhook for merging.
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

This is for the long-term record: Brian and I have just found out that
the Wx crash was apparently a Wx bug, that we can fortunately work
around ourselves.

So for now this is OK, we'll continue reviewing the code.

- Trivial problem in gtk example:

In [2]: run gui-gtk.py
---------------------------------------------------------------------------
NameError Traceback (most recent call last)

/home/fperez/ipython/repo/inputhook-lp/docs/examples/lib/gui-gtk.py in <module>()
     22 button.connect("clicked", hello_world, None)
     23
---> 24 window.add(self.button)
     25 button.show()
     26 window.show()

NameError: name 'self' is not defined
WARNING: Failure executing file: <gui-gtk.py>

I've fixed this and pushed the tiny change so I could keep my working branch synced; go ahead and pull before making further commits.

But I still wonder why (after the fix) this gtk example blocks completely ipython *after closing the gtk window*. That is, if I do *not* type '%gui gtk' first, and then '%run gui-gtk', the gtk window blocks the terminal, as it should. But even if I close the window, I can never seem to get the terminal back.

It may be that the actual destroy event hasn't been wired, but it would be good to have an example that works for our users to use as reference (that is, one which only blocks while the window is open if they forget the %gui call, but that doesn't leave ipython locked forever).

If I then try to Ctrl-C after closing the gtk window, I get this:

In [1]: run gui-gtk.py

^CERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (126, 0))

---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)

/home/fperez/ipython/repo/inputhook-lp/docs/examples/lib/gui-gtk.py in <module>()
     28 try:
     29 from IPython.lib.inputhook import appstart_gtk
---> 30 appstart_gtk()
     31 except ImportError:
     32 gtk.main()

/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/inputhook.pyc in appstart_gtk()
    163 pass
    164 else:
--> 165 gtk.main()
    166
    167 #-----------------------------------------------------------------------------

KeyboardInterrupt:
WARNING: Failure executing file: <gui-gtk.py>

I don't know if that gives you a hint as to what the problem may be. At least it allows one to recover the terminal.

Otherwise the gtk one seems fine.

- As you indicate in the docstring, it seems the gui-wx example can only be run once. Any idea why? It might be worth putting in a note requesting more info in the docstring, so at least in the future someone who knows more about this can let us know.

- Why does the Tk example actually require 'gui tk'? I would have thought that for tk, the 'do nothing' approach would be sufficient, since Python already has event loop integration with Tk built in. It might be worth dropping a note on this in the docstring and/or docs.

I think we are hitting diminishing returns here, though, and I don't want to delay things unnecessarily. You've done excellent work here, and as we saw at scipy, this is exceedingly hard to get right in all possible variations. I think we've converged on a reasonable, if not ideal, solution ...

Just some data points on the wx polling, I just finished testing with powertop on my Ubuntu 9.04 32-bit laptop, on battery. I killed everything I could, including compiz, so I was basically running nothing but a terminal and ipython. Notes:

- Once the keyboard quiets down, my baseline is ~ 25 wakeups-from-idle per second.

- If I type

gui wx

but do NOT open an app, I see a 10Hz CPU wakeup unconditionally (via powertop). This is a python-driven wakeup and it goes away when I type 'gui', so I'm 100% sure it's the 'gui wx' call doing it.

- Once I open up a Wx app (I typed %run gui-wx from the examples), the wakeups are then controlled by the inputhook_wx. That is, they start very fast but after the fallback kicks in, they slow down to the expected level. I'll send a patch next with one extra level of fallback and some changes.

- I see that even after closing the app, the inputhook seems to continue firing until one types 'gui' to disable it. I don't know if we can auto-detect anything here to make it less active when no actual windows are open but there is still keyboard activity...

OK, I'll send my small patch now from the email interface.

On Tue, Aug 25, 2009 at 7:09 PM, Fernando Perez<email address hidden> wrote:
>
> OK, I'll send my small patch now from the email interface.

Small patch attached, that adds an extra 'super-slow' 10-second sleep
layer, and changes the multiple-if to an elif chain, so the longer one
only kicks in after enough slow ones have passed.

I'm not necessarily proposing this change, just have a look to see if
it's any better...

OK, I think I have addressed all your points and have now done the merge.
We can continue to tweak this as time goes along.

Cheers,

Brian

> - Trivial problem in gtk example:
>
> In [2]: run gui-gtk.py
> ---------------------------------------------------------------------------
> NameError Traceback (most recent call last)
>
> /home/fperez/ipython/repo/inputhook-lp/docs/examples/lib/gui-gtk.py in
> <module>()
> 22 button.connect("clicked", hello_world, None)
> 23
> ---> 24 window.add(self.button)
> 25 button.show()
> 26 window.show()
>
> NameError: name 'self' is not defined
> WARNING: Failure executing file: <gui-gtk.py>
>
> I've fixed this and pushed the tiny change so I could keep my working
> branch synced; go ahead and pull before making further commits.
>

Got it.

> But I still wonder why (after the fix) this gtk example blocks completely
> ipython *after closing the gtk window*. That is, if I do *not* type '%gui
> gtk' first, and then '%run gui-gtk', the gtk window blocks the terminal, as
> it should. But even if I close the window, I can never seem to get the
> terminal back.
>
> It may be that the actual destroy event hasn't been wired, but it would be
> good to have an example that works for our users to use as reference (that
> is, one which only blocks while the window is open if they forget the %gui
> call, but that doesn't leave ipython locked forever).
>

Yep, I wired up the destroy event and it works fine now.

- As you indicate in the docstring, it seems the gui-wx example can only be
run once. Any idea why? It might be worth putting in a note requesting more
info in the docstring, so at least in the future someone who knows more
about this can let us know.

Yes, the wx window closing logic in this example is too harsh and renders wx
dead. I have added a note to the example, but I don't know wx well enough
to fix it.

- Why does the Tk example actually require 'gui tk'? I would have thought
> that for tk, the 'do nothing' approach would be sufficient, since Python
> already has event loop integration with Tk built in. It might be worth
> dropping a note on this in the docstring and/or docs.
>

Basically so we can provide a single interface to all of the different GUI
toolkits. By forcing people to "enable" (ha ha) tk, the currently active
GUI in ipython can then be proved using current_gui. At some level the same
is also true of PyQT4, which automatically handles this.

I think we are hitting diminishing returns here, though, and I don't want to
> delay things unnecessarily. You've done excellent work here, and as we saw
> at scipy, this is exceedingly hard to get right in all possible variations.
> I think we've converged on a reasonable, if not ideal, solution to this
> problem, and with help from mpl/enthought, it should be possible to make
> this transparent to all users who want pylab and/or Mayavi/Chaco/ETS.
>
> The last important thing missing here is some documentation: I think your
> solutions are fine, but we definitely need an entry in the manual (and
> possibly a FAQ entry pointing to it) explaining GUI loop integration. This
> is exteremely impo...

On Tue, Sep 1, 2009 at 9:09 AM, Brian Granger<email address hidden> wrote:
> OK, I think I have addressed all your points and have now done the merge.
> We can continue to tweak this as time goes along.
>

Awesome job, thanks! Landing this was critical, great job.

Cheers,

f