Merge lp://qastaging/~nipy-developers/nipy/trunk-neurospin into lp://qastaging/~nipy-developers/nipy/obsolete-do-not-use

trunk-neurospin is now 3-month old. The external C library has been lightened and renamed, the image registration module has been made independent from it, and many other developements have been going on.

When I try to run the tests on my computer, the process terminates
without finishing the test suite. No crash, just a termination. I suspect
there might be a 'exit' call at the C level.

Can you run the test suite on your computer?

Gaël

I'm having a look at it.
on the station, I get into trouble with funny things such as
ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined
symbol: PyUnicodeUCS2_DecodeUTF8
which I fail to understand, but could be a neurospinian curiosity
(don't laugh). There must exist a fix, since I have already used it ,
but I simply don't remeber at the moment.

on the laptop I get (after 12 minutes...):

from trunk-neurospin/nipy:

Ran 4370 tests in 673.157s

FAILED (failures=2, errors=7)

Not perfect, but it runs
Bertrand

Gael Varoquaux a écrit :
> When I try to run the tests on my computer, the process terminates
> without finishing the test suite. No crash, just a termination. I suspect
> there might be a 'exit' call at the C level.
>
> Can you run the test suite on your computer?
>
> Gaël
>

On Thu, Jul 09, 2009 at 06:15:19PM -0000, Bertrand Thirion wrote:
> I'm having a look at it.
> on the station, I get into trouble with funny things such as
> ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined
> symbol: PyUnicodeUCS2_DecodeUTF8
> which I fail to understand, but could be a neurospinian curiosity

Yes, that's exactly it. These are unicode errors due to the fact that we
have two different versions of Python, compiled with different unicode
setting.

> on the laptop I get (after 12 minutes...):

> from trunk-neurospin/nipy:

> Ran 4370 tests in 673.157s

> FAILED (failures=2, errors=7)

> Not perfect, but it runs

Well, 7 errors, and 2 failures out of 4370 tests is not bad!

OK, so it is something that happens only on my box. Zut! I need to track
it down :)

Gaël

NB : when I lauch the tests from trunk-neurospin/nipy/neurospin, the
outcome is:

======================================================================
ERROR: test_model_selection_mfx_spatial_rand_walk
(nipy.neurospin.group.tests.test_spatial_relaxation_onesample.test_multivariate_stat_saem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/data/home/thirion/mybzr/trunk-neurospin/nipy/neurospin/group/tests/test_spatial_relaxation_onesample.py",
line 111, in test_model_selection_mfx_spatial_rand_walk
    self.assertTrue(L0 > L00)
  File "/usr/lib/python2.5/unittest.py", line 309, in failUnless
    if not expr: raise self.failureException, msg
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

======================================================================
ERROR: Failure: ImportError (No module named neuroimaging.neurospin.utils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/loader.py",
line 364, in loadTestsFromName
    addr.filename, addr.module)
  File
"/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py",
line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File
"/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py",
line 84, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File
"/data/home/thirion/mybzr/trunk-neurospin/neuroimaging/neurospin/neuro/__init__.py",
line 1, in <module>
    """\
  File
"/data/home/thirion/mybzr/trunk-neurospin/neuroimaging/neurospin/neuro/image_classes.py",
line 10, in <module>
ImportError: No module named neuroimaging.neurospin.utils

----------------------------------------------------------------------
Ran 214 tests in 440.831s

FAILED (errors=2)

Bertrand Thirion a écrit :
> I'm having a look at it.
> on the station, I get into trouble with funny things such as
> ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined
> symbol: PyUnicodeUCS2_DecodeUTF8
> which I fail to understand, but could be a neurospinian curiosity
> (don't laugh). There must exist a fix, since I have already used it ,
> but I simply don't remeber at the moment.
>
> on the laptop I get (after 12 minutes...):
>
> from trunk-neurospin/nipy:
>
> Ran 4370 tests in 673.157s
>
> FAILED (failures=2, errors=7)
>
> Not perfect, but it runs
> Bertrand
>
>
> Gael Varoquaux a écrit :
>
>> When I try to run the tests on my computer, the process terminates
>> without finishing the test suite. No crash, just a termination. I suspect
>> there might be a 'exit' call at the C level.
>>
>> Can you run the test suite on your computer?
>>
>> Gaël
>>
>>
>
>
>

Bertrand,

The second error suggests that you should do a fresh install: python
is trying to import a module that doesn't exist any longer
(neuroimaging.neurospin.utils).

Alexis

On Thu, Jul 9, 2009 at 8:21 PM, Bertrand
Thirion<email address hidden> wrote:
> NB : when I lauch the tests from trunk-neurospin/nipy/neurospin, the
> outcome is:
>
> ======================================================================
> ERROR: test_model_selection_mfx_spatial_rand_walk
> (nipy.neurospin.group.tests.test_spatial_relaxation_onesample.test_multivariate_stat_saem)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File
> "/data/home/thirion/mybzr/trunk-neurospin/nipy/neurospin/group/tests/test_spatial_relaxation_onesample.py",
> line 111, in test_model_selection_mfx_spatial_rand_walk
>    self.assertTrue(L0 > L00)
>  File "/usr/lib/python2.5/unittest.py", line 309, in failUnless
>    if not expr: raise self.failureException, msg
> ValueError: The truth value of an array with more than one element is
> ambiguous. Use a.any() or a.all()
>
>
> ======================================================================
> ERROR: Failure: ImportError (No module named neuroimaging.neurospin.utils)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File
> "/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/loader.py",
> line 364, in loadTestsFromName
>    addr.filename, addr.module)
>  File
> "/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py",
> line 39, in importFromPath
>    return self.importFromDir(dir_path, fqname)
>  File
> "/usr/lib/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py",
> line 84, in importFromDir
>    mod = load_module(part_fqname, fh, filename, desc)
>  File
> "/data/home/thirion/mybzr/trunk-neurospin/neuroimaging/neurospin/neuro/__init__.py",
> line 1, in <module>
>    """\
>  File
> "/data/home/thirion/mybzr/trunk-neurospin/neuroimaging/neurospin/neuro/image_classes.py",
> line 10, in <module>
> ImportError: No module named neuroimaging.neurospin.utils
>
> ----------------------------------------------------------------------
> Ran 214 tests in 440.831s
>
> FAILED (errors=2)
>
>
>
>
> Bertrand Thirion a écrit :
>> I'm having a look at it.
>> on the station, I get into trouble with funny  things such as
>> ImportError: /usr/lib/python2.5/lib-dynload/cPickle.so: undefined
>> symbol: PyUnicodeUCS2_DecodeUTF8
>> which I fail to understand, but could be  a neurospinian curiosity
>> (don't laugh). There must exist a fix, since I have already used it ,
>> but I simply don't remeber at the moment.
>>
>> on the laptop I get (after 12 minutes...):
>>
>> from trunk-neurospin/nipy:
>>
>> Ran 4370 tests in 673.157s
>>
>> FAILED (failures=2, errors=7)
>>
>> Not perfect, but it runs
>> Bertrand
>>
>>
>> Gael Varoquaux a écrit :
>>
>>> When I try to run the tests on my computer, the process terminates
>>> without finishing the test suite. No crash, just a termination. I suspect
>>> there might be a 'exit' call at the C level.
>>>
>>> Can you run the test suite on your computer?
>>>
>...

On Thu, Jul 09, 2009 at 05:30:25PM -0000, Gael Varoquaux wrote:
> When I try to run the tests on my computer, the process terminates
> without finishing the test suite. No crash, just a termination. I suspect
> there might be a 'exit' call at the C level.

So, it was a sys.exit done in a module that was imported by nose while
looking for the tests:

http://bazaar.launchpad.net/~nipy-developers/nipy/trunk-neurospin/revision/1801

Just as a reminder for everybody: please protect all your
execution-related code from your scripts by:

if __name__ == '__main__':
    ...

All files should be importable, to be introspected. Nose is only one
example of where it is important.

Test suite is now running on my box.

Gaël

OK, remarks on the merge, as I go:

Examples
============

affine_matching example: can't run outside of neurospin, hardcoded path
and data. We will need to fix that in the long run.

Same thing for demo_blob_from_nifti.py, demo_histo_fit_nifti.py, erp.py,
onesample_group.py, script_parcel_multisubj.py, and
space_time_realign.py.

The glm.py example is dead: importing Image from nipy.neurospin fails.

Main code
==========

* nipy/neurospin/register/transform.py

    class Affine should inherit from object:

    class Affine(object):
        ...

* nipy/neurospin/spatial_models/hroi.py

    Depreciated code at the end? Could it be removed?

* nipy/neurospin/utils/mini_designmatrix.py

    Maybe the comments could be rewriten a bit. For instance translated
    from French to English?

* nipy/neurospin/utils/roi.py

    The set of tests at the end cannot be run by other people. Could we
    use artificial data instead?

* nipy/neurospin/utils/simul_2d_multisubject_fmri_dataset.py

    The 'non_random' keyword argument should be called 'seed', and should
    default to False.

Documentation
==============

Most of this code is not documented: it does not appear in the
sphinx-generated documentation. As a result, the chances that it
gets reused by people outside of neurospin is smaller: it is hard to know
what you are looking for when you are an outsider.

In the long run, it would be nice to start building a documentation.

Gaël

Thank you Gael. The fixes are not that large. I would not say that
documentation is a matter of long run.
I'm currently improving it, and I take it as a relatively short term
priority.

Bertrand

Gael Varoquaux a écrit :
> OK, remarks on the merge, as I go:
>
> Examples
> ============
>
> affine_matching example: can't run outside of neurospin, hardcoded path
> and data. We will need to fix that in the long run.
>
> Same thing for demo_blob_from_nifti.py, demo_histo_fit_nifti.py, erp.py,
> onesample_group.py, script_parcel_multisubj.py, and
> space_time_realign.py.
>
> The glm.py example is dead: importing Image from nipy.neurospin fails.
>
> Main code
> ==========
>
> * nipy/neurospin/register/transform.py
>
> class Affine should inherit from object:
>
> class Affine(object):
> ...
>
> * nipy/neurospin/spatial_models/hroi.py
>
> Depreciated code at the end? Could it be removed?
>
> * nipy/neurospin/utils/mini_designmatrix.py
>
> Maybe the comments could be rewriten a bit. For instance translated
> from French to English?
>
> * nipy/neurospin/utils/roi.py
>
> The set of tests at the end cannot be run by other people. Could we
> use artificial data instead?
>
> * nipy/neurospin/utils/simul_2d_multisubject_fmri_dataset.py
>
> The 'non_random' keyword argument should be called 'seed', and should
> default to False.
>
> Documentation
> ==============
>
> Most of this code is not documented: it does not appear in the
> sphinx-generated documentation. As a result, the chances that it
> gets reused by people outside of neurospin is smaller: it is hard to know
> what you are looking for when you are an outsider.
>
> In the long run, it would be nice to start building a documentation.
>
> Gaël
>

Hi Gaël,

Thanks a lot for your commitment.

I know about the hard-coded path issue -- at some point, I'll rewrite
my examples so that they use downloadable nipy data. Also, the glm.py
uses Edouard's datamind package, which is obviously not an official
nipy dependencie. In my mind, examples were just here to help
developers, but I agree it's better if they run.

Just a question:

> * nipy/neurospin/register/transform.py
>
>    class Affine should inherit from object:
>
>    class Affine(object):
>        ...

I do now:

class Affine():
    ...

Is that correct? Not sure I understand why any class should inherit
from something. As far as I remember, python 2.4 was not accepting
instructions like the one above.

Cheers,

Alexis

--
Alexis Roche, PhD
Researcher, CEA, Neurospin, Paris, France
Academic Guest, ETHZ, Computer Vision Lab, Zurich, Switzerland
http://alexis.roche.googlepages.com

On Fri, Jul 10, 2009 at 08:18:08AM -0000, Alexis Roche wrote:
> Hi Gaël,

> Thanks a lot for your commitment.

> I know about the hard-coded path issue -- at some point, I'll rewrite
> my examples so that they use downloadable nipy data. Also, the glm.py
> uses Edouard's datamind package, which is obviously not an official
> nipy dependencie. In my mind, examples were just here to help
> developers, but I agree it's better if they run.

> Just a question:

> > * nipy/neurospin/register/transform.py

> >    class Affine should inherit from object:

> >    class Affine(object):
> >        ...

> I do now:

> class Affine():
> ...

> Is that correct?

No, you need:

class Affine(object):

With 'object' really in there.

>Not sure I understand why any class should inherit from something.

That's historical, there use to be a 'old-style' objects, and later on,
'new-style object', that implement more clever MRO (method resolution
order) where added, and thus the base class 'object' was added.

Gaël

Also:

nipy/neurospin/statistical_mapping.py:23: undefined name 'zmap'
nipy/neurospin/statistical_mapping.py:25: undefined name 'nvoxels'

Gaël

> No, you need:
>
> class Affine(object):
>
> With 'object' really in there.

Done. I am a little confused here: is that a recommendation? Obviously
both 'class Affine:' and 'class Affine():' work.

> Also:
>
> nipy/neurospin/statistical_mapping.py:23: undefined name 'zmap'
> nipy/neurospin/statistical_mapping.py:25: undefined name 'nvoxels'

Done.

On Fri, Jul 10, 2009 at 08:45:18AM -0000, Alexis Roche wrote:
> > No, you need:

> > class Affine(object):

> > With 'object' really in there.

> Done. I am a little confused here: is that a recommendation? Obviously
> both 'class Affine:' and 'class Affine():' work.

They do work but they are not the same:

In [1]: class A:
   ...: pass
   ...:

In [2]: a = A()

In [3]: a.
a.__class__ a.__doc__ a.__module__

In [3]: class A(object):
   ...: pass
   ...:

In [4]: a = A()

In [5]: a.
a.__class__ a.__hash__ a.__repr__
a.__delattr__ a.__init__ a.__setattr__
a.__dict__ a.__module__ a.__sizeof__
a.__doc__ a.__new__ a.__str__
a.__format__ a.__reduce__ a.__subclasshook__
a.__getattribute__ a.__reduce_ex__ a.__weakref__

The later is a 'new-style class'. People nowadays expect new-style
class, so if you give them old style object, things will break (for
instance a sub-class calling the parent class __repr__). This is
why it is a recommandation always to use a new-style classes.

Gaël

Capito! Grazie.

Alexis

On Fri, Jul 10, 2009 at 4:36 PM, Gael
Varoquaux<email address hidden> wrote:
> On Fri, Jul 10, 2009 at 08:45:18AM -0000, Alexis Roche wrote:
>> > No, you need:
>
>> > class Affine(object):
>
>> > With 'object' really in there.
>
>> Done. I am a little confused here: is that a recommendation? Obviously
>> both 'class Affine:' and 'class Affine():' work.
>
> They do work but they are not the same:
>
> In [1]: class A:
>   ...:     pass
>   ...:
>
> In [2]: a = A()
>
> In [3]: a.
> a.__class__   a.__doc__     a.__module__
>
> In [3]: class A(object):
>   ...:     pass
>   ...:
>
> In [4]: a = A()
>
> In [5]: a.
> a.__class__         a.__hash__          a.__repr__
> a.__delattr__       a.__init__          a.__setattr__
> a.__dict__          a.__module__        a.__sizeof__
> a.__doc__           a.__new__           a.__str__
> a.__format__        a.__reduce__        a.__subclasshook__
> a.__getattribute__  a.__reduce_ex__     a.__weakref__
>
> The later is a 'new-style class'. People nowadays expect new-style
> class, so if you give them old style object, things will break (for
> instance a sub-class calling the parent class __repr__). This is
> why it is a recommandation always to use a new-style classes.
>
> Gaël
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

--
Alexis Roche, PhD
Researcher, CEA, Neurospin, Paris, France
Academic Guest, ETHZ, Computer Vision Lab, Zurich, Switzerland
http://alexis.roche.googlepages.com

More comments/questions, as I go through the code:

http://bazaar.launchpad.net/~nipy-developers/nipy/trunk-neurospin/annotate/head%3A/nipy/neurospin/utils/roi.py#L250

The two method __add__ and __multiply__ as well as the WeightedROI seem
quite wrong to me. Is there a reason for having them in, and not simply
deleting them?

Cheers,

Gaël

Also, after fixing a few trivial problems, I have some test failures:

resting ~/dev/nipy-neurospin/nipy/neurospin $ nosetests --with-coverage
--cover-package=nipy.neurospin
.........................................................................................F...........................................................................................F...............FF..............
======================================================================
FAIL: testvarious
(nipy.neurospin.clustering.tests.test_clustering.TestTypeProof)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/varoquau/dev/nipy-neurospin/nipy/neurospin/clustering/tests/test_clustering.py",
line 190, in testvarious
    self.assert_(sys.getrefcount(X) == 2)
AssertionError

======================================================================
FAIL: test_model_selection_mfx_spatial_rand_walk
(nipy.neurospin.group.tests.test_spatial_relaxation_onesample.TestMultivariateStatSaem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/varoquau/dev/nipy-neurospin/nipy/neurospin/group/tests/test_spatial_relaxation_onesample.py",
line 131, in test_model_selection_mfx_spatial_rand_walk
    self.assertTrue(np.all(L0 > L00))
AssertionError

======================================================================
FAIL: test_bsa.test_bsa_null_dev
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/varoquau/usr/lib/python2.6/site-packages/nose/case.py",
line 182, in runTest
    self.test(*self.arg)
  File
"/home/varoquau/dev/nipy-neurospin/nipy/neurospin/spatial_models/tests/test_bsa.py",
line 116, in test_bsa_null_dev
    assert(AF==None)
AssertionError:
-------------------- >> begin captured stdout << ---------------------
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position definition, assuming it is the
identity
warning: no sform found for position ...

All right. I forward the failure report for
test_model_selection_mfx_spatial_rand_walk to Merlin.

Alexis

On Sat, Jul 11, 2009 at 10:15 PM, Gael
Varoquaux<email address hidden> wrote:
> Also, after fixing a few trivial problems, I have some test failures:
>
> resting ~/dev/nipy-neurospin/nipy/neurospin $ nosetests --with-coverage
> --cover-package=nipy.neurospin
> .........................................................................................F...........................................................................................F...............FF..............
> ======================================================================
> FAIL: testvarious
> (nipy.neurospin.clustering.tests.test_clustering.TestTypeProof)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/clustering/tests/test_clustering.py",
> line 190, in testvarious
>    self.assert_(sys.getrefcount(X) == 2)
> AssertionError
>
> ======================================================================
> FAIL: test_model_selection_mfx_spatial_rand_walk
> (nipy.neurospin.group.tests.test_spatial_relaxation_onesample.TestMultivariateStatSaem)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/group/tests/test_spatial_relaxation_onesample.py",
> line 131, in test_model_selection_mfx_spatial_rand_walk
>    self.assertTrue(np.all(L0 > L00))
> AssertionError
>
> ======================================================================
> FAIL: test_bsa.test_bsa_null_dev
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/home/varoquau/usr/lib/python2.6/site-packages/nose/case.py",
> line 182, in runTest
>    self.test(*self.arg)
>  File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/spatial_models/tests/test_bsa.py",
> line 116, in test_bsa_null_dev
>    assert(AF==None)
> AssertionError:
> -------------------- >> begin captured stdout << ---------------------
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position definition,  assuming it is the
> identity
> warning: no sform found for position ...

I just pushed the corrections for
test_model_selection_mfx_spatial_rand_walk.
It should work fine now.

Merlin

-------- Message d'origine--------
De: <email address hidden> de la part de Alexis Roche
Date: sam. 11/07/2009 23:09
À: KELLER Merlin INRIA
Objet : Re: [Nipy-devel] [Merge]

All right. I forward the failure report for
test_model_selection_mfx_spatial_rand_walk to Merlin.

Alexis

On Sat, Jul 11, 2009 at 10:15 PM, Gael
Varoquaux<email address hidden> wrote:
> Also, after fixing a few trivial problems, I have some test failures:
>
> resting ~/dev/nipy-neurospin/nipy/neurospin $ nosetests --with-coverage
> --cover-package=nipy.neurospin
> .........................................................................................F...........................................................................................F...............FF..............
> ======================================================================
> FAIL: testvarious
> (nipy.neurospin.clustering.tests.test_clustering.TestTypeProof)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/clustering/tests/test_clustering.py",
> line 190, in testvarious
> self.assert_(sys.getrefcount(X) == 2)
> AssertionError
>
> ======================================================================
> FAIL: test_model_selection_mfx_spatial_rand_walk
> (nipy.neurospin.group.tests.test_spatial_relaxation_onesample.TestMultivariateStatSaem)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/group/tests/test_spatial_relaxation_onesample.py",
> line 131, in test_model_selection_mfx_spatial_rand_walk
> self.assertTrue(np.all(L0 > L00))
> AssertionError
>
> ======================================================================
> FAIL: test_bsa.test_bsa_null_dev
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/varoquau/usr/lib/python2.6/site-packages/nose/case.py",
> line 182, in runTest
> self.test(*self.arg)
> File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/spatial_models/tests/test_bsa.py",
> line 116, in test_bsa_null_dev
> assert(AF==None)
> AssertionError:
> -------------------- >> begin captured stdout << ---------------------
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity
> warning: no sform found for position definition, assuming it is the
> identity...

I just pushed another version of test_spatial_relaxation_onesample,
which takes much less time to run.

Merlin

Gael Varoquaux a écrit :
> Also, after fixing a few trivial problems, I have some test failures:
>
> resting ~/dev/nipy-neurospin/nipy/neurospin $ nosetests --with-coverage
> --cover-package=nipy.neurospin
> .........................................................................................F...........................................................................................F...............FF..............
> ======================================================================
> FAIL: testvarious
> (nipy.neurospin.clustering.tests.test_clustering.TestTypeProof)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/varoquau/dev/nipy-neurospin/nipy/neurospin/clustering/tests/test_clustering.py",
> line 190, in testvarious
> self.assert_(sys.getrefcount(X) == 2)
> AssertionError
>
>
Pour celui-là, je ne suis pas sûr de comprendre: suivant les cas, on
trouve 2 ou 3 références...

Bertrand

On Wed, Jul 15, 2009 at 05:51:09PM -0000, Bertrand Thirion wrote:
> > line 190, in testvarious
> > self.assert_(sys.getrefcount(X) == 2)
> > AssertionError

> Pour celui-là, je ne suis pas sûr de comprendre: suivant les cas, on
> trouve 2 ou 3 références...

Let's ignore this failure for now. I want to focus on other things before
I leave Berkeley. We can merge with this problem, I believe.

Just a heads up on this merge: I would like to get the data-refactor
branch in first, pull from it in the neurospin branch, rework the data
access in the neurospin branch, and then merge the neurospin branch.
Hopefully this should all be done before the end of the week.

How does that sound?

Gaël

> Let's ignore this failure for now. I want to focus on other things before
> I leave Berkeley. We can merge with this problem, I believe.
>
> Just a heads up on this merge: I would like to get the data-refactor
> branch in first, pull from it in the neurospin branch, rework the data
> access in the neurospin branch, and then merge the neurospin branch.
> Hopefully this should all be done before the end of the week.
>
> How does that sound?
>
OK.

Bertrand

Hi,

> Just a heads up on this merge: I would like to get the data-refactor
> branch in first, pull from it in the neurospin branch, rework the data
> access in the neurospin branch, and then merge the neurospin branch.
> Hopefully this should all be done before the end of the week.
>
> How does that sound?

Sounds good to me too.

Oh, sorry I mean

+1

luv,

Matthew

> Oh, sorry I mean
>
> +1

This is funny.

I mean +1 too.

Alexis

The doc of trunk-neurospin now seems to build fine... even on windows XP!

I had to fix a number of issues in Bertrand's docstrings, though. The
system seems fairly strict: it does not like colons in section titles,
nor does it like dashes that do not match title lengths... Bertrand,
you should check that your docstrings are formatted the way you want
them to be. A suggested approach would be to build the doc once in a
while :-)

Cheers,

Alexis

For editing ReST in my docstrings I enable ReST-mode in emacs. Then
for section headings (titles) you enter in the heading, and hit
Ctrl-=, (Control Key + Equal Key) and it fills in the dashes on the
line below for you. Repeatedly hit Ctrl-= and it cycles through the
different 'dashes' for different sized headings. This is handy also
for when you change the heading later, you just Ctrl-= and it resizes
the dashes. Way easier then trying to manually adjust them.

I added this in my ~/.emacs file enable rst-mode on all my text files.

;; rst-mode
(require 'rst)
(setq auto-mode-alist
      (append '(("\\.rst$" . rst-mode)
                ("\\.rest$" . rst-mode)
  ("\\.txt$" . rst-mode)) auto-mode-alist))

Then have a key binding to switch it on:

(global-set-key [(f11)] 'rst-mode)

I'm sure Vi has something similar.
Chris

On Thu, Jul 16, 2009 at 4:36 PM, Alexis Roche<email address hidden> wrote:
> nor does it like dashes that do not match title lengths...

Thanks for this.

B.

Christopher Burns a écrit :
> For editing ReST in my docstrings I enable ReST-mode in emacs. Then
> for section headings (titles) you enter in the heading, and hit
> Ctrl-=, (Control Key + Equal Key) and it fills in the dashes on the
> line below for you. Repeatedly hit Ctrl-= and it cycles through the
> different 'dashes' for different sized headings. This is handy also
> for when you change the heading later, you just Ctrl-= and it resizes
> the dashes. Way easier then trying to manually adjust them.
>
> I added this in my ~/.emacs file enable rst-mode on all my text files.
>
> ;; rst-mode
> (require 'rst)
> (setq auto-mode-alist
> (append '(("\\.rst$" . rst-mode)
> ("\\.rest$" . rst-mode)
> ("\\.txt$" . rst-mode)) auto-mode-alist))
>
> Then have a key binding to switch it on:
>
> (global-set-key [(f11)] 'rst-mode)
>
> I'm sure Vi has something similar.
> Chris
>
> On Thu, Jul 16, 2009 at 4:36 PM, Alexis Roche<email address hidden> wrote:
>
>> nor does it like dashes that do not match title lengths...
>>

Thanks Chris, it's useful.

Alexis

On Fri, Jul 17, 2009 at 4:30 AM, Christopher Burns<email address hidden> wrote:
> For editing ReST in my docstrings I enable ReST-mode in emacs.  Then
> for section headings (titles) you enter in the heading, and hit
> Ctrl-=, (Control Key + Equal Key) and it fills in the dashes on the
> line below for you.  Repeatedly hit Ctrl-= and it cycles through the
> different 'dashes' for different sized headings. This is handy also
> for when you change the heading later, you just Ctrl-= and it resizes
> the dashes.  Way easier then trying to manually adjust them.
>
> I added this in my ~/.emacs file enable rst-mode on all my text files.
>
> ;; rst-mode
> (require 'rst)
> (setq auto-mode-alist
>      (append '(("\\.rst$" . rst-mode)
>                ("\\.rest$" . rst-mode)
>                ("\\.txt$" . rst-mode)) auto-mode-alist))
>
> Then have a key binding to switch it on:
>
> (global-set-key [(f11)] 'rst-mode)
>
> I'm sure Vi has something similar.
> Chris
>
> On Thu, Jul 16, 2009 at 4:36 PM, Alexis Roche<email address hidden> wrote:
>> nor does it like dashes that do not match title lengths...
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

--
Alexis Roche, PhD
Researcher, CEA, Neurospin, Paris, France
Academic Guest, ETHZ, Computer Vision Lab, Zurich, Switzerland
http://alexis.roche.googlepages.com

This just to ask - where are we with this proposed merge? Is there any work that we can help with?

I am wondering the same thing. Gaël, can you update us on the merge?

Best,

Alexis

On Tue, Aug 11, 2009 at 12:21:08AM -0000, Alexis Roche wrote:
> I am wondering the same thing. Gaël, can you update us on the merge?

We should update the downloading code to use the data framework that
Matthew has set up recently. I have been wanting to do this, but haven't
found time.

That's the only thing I see. Last time I looked, all the tests in our
code where passing (which is really good work, by the way).

Gaël

Hi Gaël,

> We should update the downloading code to use the data framework that
> Matthew has set up recently. I have been wanting to do this, but haven't
> found time.

Does that entail updating the nipy-data tarball to include additional
files? I am interested in adding some more data files for my example
scripts to be able to run outside neurospin.

> That's the only thing I see. Last time I looked, all the tests in our
> code where passing (which is really good work, by the way).

You mean "which is really professional work", don't you? :-)

Cheers,

Alexis

On Thu, Aug 13, 2009 at 11:48:08PM -0000, Alexis Roche wrote:
> > We should update the downloading code to use the data framework that
> > Matthew has set up recently. I have been wanting to do this, but haven't
> > found time.

> Does that entail updating the nipy-data tarball to include additional
> files? I am interested in adding some more data files for my example
> scripts to be able to run outside neurospin.

Possibly, and if not, posting these files on a download location where
everybody can access them, and using the right download mechanism to get
them. The goal is exactly that example scripts run outside of neurospin.

> > That's the only thing I see. Last time I looked, all the tests in our
> > code where passing (which is really good work, by the way).

> You mean "which is really professional work", don't you? :-)

Correct :).

Gaël

>> Does that entail updating the nipy-data tarball to include additional
>> files? I am interested in adding some more data files for my example
>> scripts to be able to run outside neurospin.
>
> Possibly, and if not, posting these files on a download location where
> everybody can access them, and using the right download mechanism to get
> them. The goal is exactly that example scripts run outside of neurospin.

You should I hope have got a Dropbox invite - if you put files up
there for now, I can put them into the package and upload it.
Alternatively, the design of the package structure was meant to allow
you to:

download the current package
unpack
add some images
bump the version number
repackage
upload

We're going to change hosting soon, so let's keep it 'lightweight' for
now (I mean I'll do it by hand, if you send me the files).

>> You mean "which is really professional work", don't you? :-)
>
> Correct :).

Ah the hum of happy enterprise... ;)

Matthew

> You should I hope have got a Dropbox invite - if you put files up
> there for now, I can put them into the package and upload it.

OK, I just added two MR-T1 image files for the affine registration
example script, and one NPZ file for the onesample fMRI group analysis
script.

I have other examples based on the FIAC dataset. My understanding is
that there is a plan to include FIAC, or at least part of it, in
nipy-data. Is that correct?

Cheers,

Alexis

On Fri, Aug 14, 2009 at 7:36 AM, Alexis Roche<email address hidden> wrote:
>> You should I hope have got a Dropbox invite - if you put files up
>> there for now, I can put them into the package and upload it.
>
> OK, I just added two MR-T1 image files for the affine registration
> example script, and one NPZ file for the onesample fMRI group analysis
> script.

Thanks - I see them.

> I have other examples based on the FIAC dataset. My understanding is
> that there is a plan to include FIAC, or at least part of it, in
> nipy-data. Is that correct?

No - not really. We were going to post FIAC separately as a huge
tarball somewhere, but with an explicit: 'please download
http://some.place.org/fiac/fiac_huge_tarball.bz2 and unpack' for any
scripts that needed it. We thought it was just too large for
distribution (Debian, Windows etc) packaging, which was the intention
of the current structure.

We wanted to keep the current packages from getting very large too -
certainly less than 500M, I hope less than 50.

No problem to include example FIAC files in the packages if they are useful...

see you,

Matthew

> No problem to include example FIAC files in the packages if they are useful...

I just updated my Dropbox so as to include all the necessary files for
my example scripts to run. Even though I tried to keep storage as low
as possible, I am adding a total of around 78MB to the example_data
package. Please let me know if you think it's too much.

Best,

Alexis

On Mon, Aug 17, 2009 at 02:36:10PM -0000, Alexis Roche wrote:
> > No problem to include example FIAC files in the packages if they are useful...

> I just updated my Dropbox so as to include all the necessary files for
> my example scripts to run. Even though I tried to keep storage as low
> as possible, I am adding a total of around 78MB to the example_data
> package. Please let me know if you think it's too much.

I haven't had time to look at that, but is their a possibility to reuse
these files in other examples? I think that's what we need to be careful
about: having a rather small set of examples that we can use in many
places.

Gaël

> I haven't had time to look at that, but is their a possibility to reuse
> these files in other examples?

Those are fairly standard types of neuroimage data, so I'm sure they
can be reused. Specifically, I included:
- two T1 images from different subjects
- two fMRI sessions of a given FIAC subject, with the corresponding
design matrices (in NPZ format)
- one second-level fMRI group dataset taken from another protocol

Those files are either in NIFTI or in NPZ format, so they can be read
without additional packages.

Cheers,

Alexis

On Mon, Aug 17, 2009 at 7:36 AM, Alexis Roche<email address hidden> wrote:
>> No problem to include example FIAC files in the packages if they are useful...
>
> I just updated my Dropbox so as to include all the necessary files for
> my example scripts to run. Even though I tried to keep storage as low
> as possible, I am adding a total of around 78MB to the example_data
> package. Please let me know if you think it's too much.

I think that's OK - let's just keep an eye on it.

I've put a new data package up here:

https://cirl.berkeley.edu/mb312/nipy-data/

You'll see a Makefile in the dropbox folder now.

I've made a package validator, now here:

It would be rather easy to set y'all up with the ability to modify the
packages directly if you like, let me know.

Otherwise, it's a trivial build step for me, happy to do it.

> I've made a package validator, now here:

Sorry - here:

http://bazaar.launchpad.net/~nipy-developers/nipy/nipy-datarefactor/annotate/head%3A/tools/validata_data_pkg.py

Hi,

Where are we now regarding the merge? Gaël, do you need help on specific tasks?

Best,

Alexis

On Tue, Aug 18, 2009 at 5:18 AM, Matthew Brett<email address hidden> wrote:
>> I've made a package validator, now here:
>
> Sorry - here:
>
> http://bazaar.launchpad.net/~nipy-developers/nipy/nipy-datarefactor/annotate/head%3A/tools/validata_data_pkg.py
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

--
Alexis Roche, PhD
Researcher, CEA, Neurospin, Paris, France
Academic Guest, ETHZ, Computer Vision Lab, Zurich, Switzerland
http://alexis.roche.googlepages.com

On Mon, Aug 31, 2009 at 07:36:10AM -0000, Alexis Roche wrote:
> Where are we now regarding the merge? Gaël, do you need help on
> specific tasks?

I was just thinking this week end that I don't have time to polish the
things I wanted to polish. So I think that either you have time to move
the data downloading and examples to use the new API, or we should merge
anyhow and worry about this later.

Tell me what you want to do? If we decide to merge, I'll merge trunk in
trunk-neurospin, and Matthew can merge trunk-neurospin in trunk.

In any case, I'll just run the tests on 32 bits and 64 bits today to
check all is fine.

Sorry for taking that long. I am convinced that we should merge more
often, and I don't even respect my own convictions :).

Gaël

Hi Gaël,

> So I think that either you have time to move
> the data downloading and examples to use the new API, or we should merge
> anyhow and worry about this later.

I already did part of the job, but some examples require files that
Bertrand (I suppose) put together on
ftp://ftp.cea.fr/pub/dsv/madic/download/nipy/

It should be up to Bertrand to decide on the future of that small
database. The right approach would probably be to push it into
nipy-data, but I won't do it myself without his consent. Either way, I
don't see this as an urgent matter. This can be done after the merge.

> Tell me what you want to do?

I guess I support the quick merge option.

> Sorry for taking that long. I am convinced that we should merge more
> often, and I don't even respect my own convictions :).

Please do not apologize. trunk-neurospin turns out to be a fairly big
chunk and we should maybe consider splitting it into several
"lightweight" branches in order to ease merging in the future. I
believe it makes sense to have a subpackage called "neurospin", but
having a single branch for all the projects going on in neurospin
might not be ideal.

Cheers,

Alexis

On Mon, Aug 31, 2009 at 09:51:57AM +0200, Gael Varoquaux wrote:
> In any case, I'll just run the tests on 32 bits and 64 bits today to
> check all is fine.

OK, I have 6 test failures in the neurospin subpackage that look like
they are fairly easy to correct and non-specific to 64 bit. I believe
that you know this code well. Could you have a look please. The test log
is attached.

Thanks,

Gaël

..................................................................................................................................................................................................................EEEEEE...................................
======================================================================
ERROR: test_iconic_matcher.test_clamping_uint8
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
line 182, in runTest
    self.test(*self.arg)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 46, in test_clamping_uint8
    _test_clamping(I)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 34, in _test_clamping
    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, thI, thI,
bins=clI)
TypeError: __init__() got an unexpected keyword argument 'bins'

======================================================================
ERROR: test_iconic_matcher.test_clamping_uint8_nonstd
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
line 182, in runTest
    self.test(*self.arg)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 50, in test_clamping_uint8_nonstd
    _test_clamping(I, 10, 165)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 34, in _test_clamping
    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, thI, thI,
bins=clI)
TypeError: __init__() got an unexpected keyword argument 'bins'

======================================================================
ERROR: test_iconic_matcher.test_clamping_int16
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
line 182, in runTest
    self.test(*self.arg)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 54, in test_clamping_int16
    _test_clamping(I)
  File
"/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
line 34, in _test_clamping
    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, thI, thI,
bins=clI)
TypeError: __init__() got an unexpected keyword argument 'bins'

======================================================================
ERROR: test_iconic_matcher.test_clamping_int16_nonstd
----------------------------------------------------------------------
Trace...

Yes, it's trivial. I'm doing it, and testing everything. Should be
done within an hour or so.

On Mon, Aug 31, 2009 at 10:36 AM, Gael
Varoquaux<email address hidden> wrote:
> On Mon, Aug 31, 2009 at 09:51:57AM +0200, Gael Varoquaux wrote:
>> In any case, I'll just run the tests on 32 bits and 64 bits today to
>> check all is fine.
>
> OK, I have 6 test failures in the neurospin subpackage that look like
> they are fairly easy to correct and non-specific to 64 bit. I believe
> that you know this code well. Could you have a look please. The test log
> is attached.
>
> Thanks,
>
> Gaël
>
> ..................................................................................................................................................................................................................EEEEEE...................................
> ======================================================================
> ERROR: test_iconic_matcher.test_clamping_uint8
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
> line 182, in runTest
>    self.test(*self.arg)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 46, in test_clamping_uint8
>    _test_clamping(I)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 34, in _test_clamping
>    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, thI, thI,
> bins=clI)
> TypeError: __init__() got an unexpected keyword argument 'bins'
>
> ======================================================================
> ERROR: test_iconic_matcher.test_clamping_uint8_nonstd
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
> line 182, in runTest
>    self.test(*self.arg)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 50, in test_clamping_uint8_nonstd
>    _test_clamping(I, 10, 165)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 34, in _test_clamping
>    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, thI, thI,
> bins=clI)
> TypeError: __init__() got an unexpected keyword argument 'bins'
>
> ======================================================================
> ERROR: test_iconic_matcher.test_clamping_int16
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/volatile/varoquau/usr/lib/python2.5/site-packages/nose/case.py",
> line 182, in runTest
>    self.test(*self.arg)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 54, in test_clamping_int16
>    _test_clamping(I)
>  File
> "/volatile/varoquau/dev/nipy-neurospin/nipy/neurospin/register/tests/test_iconic_matcher.py",
> line 34, in _test_clamping
>    IM = IconicMatcher(I.array, I.array, I.toworld, I.toworld, ...

On Mon, Aug 31, 2009 at 10:39:47AM +0200, Alexis Roche wrote:
> Yes, it's trivial. I'm doing it, and testing everything. Should be
> done within an hour or so.

OK, once this is done, we are good to merge. I tested on 64 bits, there
is no problem there.

For what its worth, we have a 54% test coverage of the neurospin
codebase, with the worst offenders in terms of coverage being:

    nipy.neurospin.utils.mask (I need to take care of that)
    nipy.neurospin.statistical_mapping
    nipy.neurospin.spatial_models.structural_bfls
    nipy.neurospin.image_registration
    nipy.neurospin.graph.BPmatch
    nipy.neurospin.eda.dimension_reduction
    nipy.neurospin.viz (I need completely rewrite this)

Gaël

I just committed a revision (1872) with fixes for various issues,
including the testing errors you spotted.

Running nipy.neurospin.test() on my minGW system yields a fantastic
zero errors, zero failures.

Ran 251 tests in 401.532s

OK

<nose.result.TextTestResult run=251 errors=0 failures=0>

On Mon, Aug 31, 2009 at 10:21:09AM -0000, Alexis Roche wrote:
> I just committed a revision (1872) with fixes for various issues,
> including the testing errors you spotted.

> Running nipy.neurospin.test() on my minGW system yields a fantastic
> zero errors, zero failures.

> Ran 251 tests in 401.532s

Same thing here.

Matthew, you can merge revision 1872 when you're ready.

Gaël

On Mon, Aug 31, 2009 at 01:11:16PM +0200, Gael Varoquaux wrote:
> On Mon, Aug 31, 2009 at 10:21:09AM -0000, Alexis Roche wrote:
> > I just committed a revision (1872) with fixes for various issues,
> > including the testing errors you spotted.

> > Running nipy.neurospin.test() on my minGW system yields a fantastic
> > zero errors, zero failures.

> > Ran 251 tests in 401.532s

> Same thing here.

> Matthew, you can merge revision 1872 when you're ready.

Actually, make that 1869[*], Bertrand has fixed a few examples, and I
just realized that I had unpushed fixes on my laptop.

We've just had history folding, someone will have to tell me how one can
avoid that with long running branches continuously synced.

Gaël

Hey,

On Tue, Sep 1, 2009 at 12:27 AM, Gael
Varoquaux<email address hidden> wrote:
>
> We've just had history folding, someone will have to tell me how one can
> avoid that with long running branches continuously synced.

As best as I've been able to learn from our similar pains early on
with ipython, the only way to avoid this problem is to use the 'two
branch dance', as described summarily in the ipython guide and in more
detail by Chris in the nipy one:

http://ipython.scipy.org/doc/stable/html/development/overview.html

http://neuroimaging.scipy.org/doc/manual/html/devel/guidelines/bzr_workflow.html

The take-home message is:

"You shall never do pull and merge in the same branch checkout, EVER"

Because of this, the safest solution is to have two branches: one
which you use to track the remote one, and where you only do
push/pull, and one where you work locally and where you do all merging
work.

I'm interested in learning if there's a better way to do this, of
course, but that's the best of my current knowledge.

Cheers,

f

Hello,
I am lurking here because I am just monitoring the branches. I was
pointed to it because of the stats part. I think it became a scikit now.
> I'm interested in learning if there's a better way to do this, of
> course, but that's the best of my current knowledge.
Have you tried to ask at the bzr mailing list?

People there are eager to help and may point you to some tricks...

Best,
Timmie

On Tue, Sep 1, 2009 at 12:15 PM, Tim<email address hidden> wrote:
> Have you tried to ask at the bzr mailing list?
>
> People there are eager to help and may point you to some tricks...

Not really, no. We found a pretty reasonable working solution and I
just moved on. By now I have the pattern fairly cleanly embedded in
my brain, and I use it fine, it doesn't really bother me. I'm already
overstretched with mailing list activity as it is, so for things like
this I'll just settle for the 'works for me' solution :)

Cheers,

f

On Tue, Sep 01, 2009 at 06:51:12PM -0000, Fernando Perez wrote:
> As best as I've been able to learn from our similar pains early on
> with ipython, the only way to avoid this problem is to use the 'two
> branch dance', as described summarily in the ipython guide and in more
> detail by Chris in the nipy one:

> http://ipython.scipy.org/doc/stable/html/development/overview.html

> http://neuroimaging.scipy.org/doc/manual/html/devel/guidelines/bzr_workflow.html

> The take-home message is:

> "You shall never do pull and merge in the same branch checkout, EVER"

> Because of this, the safest solution is to have two branches: one
> which you use to track the remote one, and where you only do
> push/pull, and one where you work locally and where you do all merging
> work.

I while ago I used to think this 'dance' solved the problems. When I was
doing a lot of development on IPython it worked really well for me.
However, I haven't been able to get it working with the development
pattern in nipy. Let me explain the problem.

1) I branch out trunk to trunk-gael

2) I work on trunk-gael, 'bzr ci' as I work

3) I bug fix is committed to trunk

4) I need the bug fix. This is where I break the golden rule, and merge
from trunk to trunk-gael

5) I work on trunk-gael, 'bzr ci' as I work

6) I am ready for merging in trunk, but how do I avoid history folding?

The problem for me lies in 4). And it happens very often. Now I may be
missing something, and I'll love to be enlightened.

Gaël

Hey,

On Tue, Sep 1, 2009 at 1:06 PM, Gael
Varoquaux<email address hidden> wrote:
> 1) I branch out trunk to trunk-gael
>
> 2) I work on trunk-gael, 'bzr ci' as I work
>
> 3) I bug fix is committed to trunk
>
> 4) I need the bug fix. This is where I break the golden rule, and merge
> from trunk to trunk-gael
>
> 5) I work on trunk-gael, 'bzr ci' as I work
>
> 6) I am ready for merging in trunk, but how do I avoid history folding?

This should be OK, as long as you do:

cd trunk/
bzr merge ../trunk-gael
bzr push

because at that point, bzr will recognize that the revisions you'd
merged in (4) are already in trunk. If you look graphically at the
history it will look funny, but you shouldn't get any 'dropped
revisions' messages or any other problem.

The issue arises in the following scenario:

cd trunk # the one meant to track launchpad 1-to-1
work; commit; work
# At the same time, someone else pushes something to the same trunk
# now, you try to push:
bzr push
# boom

At this point, the two histories have diverged, and doing a forced
push or a merge is likely to cause problems.

But in your original scenario, I don't really think your #4 is a
breakage of the golden rule, as long as you don't have anyone else who
can push to trunk-gael to cause true history divergence. The problem
is a divergence between local and remote history, and I think that's
not happening in your case.

Cheers,

f

Hi,

>> Matthew, you can merge revision 1872 when you're ready.
>
> Actually, make that 1869[*], Bertrand has fixed a few examples, and I
> just realized that I had unpushed fixes on my laptop.

Guys - I'm a bit confused

Current revision of trunk-neurospin seems to be 1871

Do you really want me to merge 1869?

Matthew

On Wed, Sep 02, 2009 at 11:24:08PM -0000, Matthew Brett wrote:
> Hi,

> >> Matthew, you can merge revision 1872 when you're ready.

> > Actually, make that 1869[*], Bertrand has fixed a few examples, and I
> > just realized that I had unpushed fixes on my laptop.

> Guys - I'm a bit confused

> Current revision of trunk-neurospin seems to be 1871

I pushed a bug fix yesterday. 1871 is good.

Thanks,

Gaël

On Wed, Sep 02, 2009 at 11:18:08PM -0000, Fernando Perez wrote:
> Hey,

> On Tue, Sep 1, 2009 at 1:06 PM, Gael
> Varoquaux<email address hidden> wrote:
> > 1) I branch out trunk to trunk-gael

> > 2) I work on trunk-gael, 'bzr ci' as I work

> > 3) I bug fix is committed to trunk

> > 4) I need the bug fix. This is where I break the golden rule, and merge
> > from trunk to trunk-gael

> > 5) I work on trunk-gael, 'bzr ci' as I work

> > 6) I am ready for merging in trunk, but how do I avoid history folding?

> This should be OK, as long as you do:

> cd trunk/
> bzr merge ../trunk-gael
> bzr push

Yes, this is what I realised yesterday. 4) should be:

cd trunk/
bzr pull pl:nipy
bzr merge ../trunk-gael
bzr ci -m 'MRG: Sync trunk-gael with trunk'
cd ../trunk-gael
bzr pull ../trunk

It worked on a simple example.

The 'merge in one direction to be able to pull in the other' wasn't clear
to me. It is a directed graph, and the order in wich the edges are
created matter!

Gaël

On Wed, Sep 2, 2009 at 10:21 PM, Gael Varoquaux
<email address hidden> wrote:
> Yes, this is what I realised yesterday. 4) should be:
>
> cd trunk/
> bzr pull pl:nipy
> bzr merge ../trunk-gael
> bzr ci -m 'MRG: Sync trunk-gael with trunk'

!!!
At this point, you should immediately do a push! Since you are using
trunk as a local mirror of lp:nipy (you are 'pulling' from it, not
merging from it), you should then push up so that your local commit is
publicly available to others. Basically your local history should
match the one seen on the public site at lp:nipy.

> cd ../trunk-gael
> bzr pull ../trunk
>
> It worked on a simple example.

But it won't in others. In general, since trunk-gael is NOT a mirror
image of trunk, here you should do

cd ../trunk-gael
bzr mege ../trunk

That will always work. The pull operation is dangerous because for
bzr, push/pull mean: "make these two trees IDENTICAL, down to every
commit and revision number". That's why in this case you should use
merge and not pull, and why in the one above you should push after
your local commit, so your local history and the public one remain
identical.

Makes sense?

> The 'merge in one direction to be able to pull in the other' wasn't clear
> to me. It is a directed graph, and the order in wich the edges are
> created matter!

Yup :)

Cheers,

f

On Thu, Sep 03, 2009 at 07:18:09PM -0000, Fernando Perez wrote:
> On Wed, Sep 2, 2009 at 10:21 PM, Gael Varoquaux
> <email address hidden> wrote:
> > Yes, this is what I realised yesterday. 4) should be:

> > cd trunk/
> > bzr pull pl:nipy
> > bzr merge ../trunk-gael
> > bzr ci -m 'MRG: Sync trunk-gael with trunk'

> !!!
> At this point, you should immediately do a push! Since you are using
> trunk as a local mirror of lp:nipy (you are 'pulling' from it, not
> merging from it), you should then push up so that your local commit is
> publicly available to others. Basically your local history should
> match the one seen on the public site at lp:nipy.

Well, no. I am not ready to share my changes. They are buggy, and may not
be fully functionnal. I am going to break other peoples code, and get in
trouble if I do.

> Makes sense?

Nope. Your explaination does make sens, but it simply shows me that I
haven't solved my real problem. Or am I wrong?

Gaël

On Thu, Sep 3, 2009 at 3:39 PM, Gael
Varoquaux<email address hidden> wrote:
> On Thu, Sep 03, 2009 at 07:18:09PM -0000, Fernando Perez wrote:
>> On Wed, Sep 2, 2009 at 10:21 PM, Gael Varoquaux
>> <email address hidden> wrote:
>> > Yes, this is what I realised yesterday. 4) should be:
>
>> > cd trunk/
>> > bzr pull pl:nipy
>> > bzr merge ../trunk-gael
>> > bzr ci -m 'MRG: Sync trunk-gael with trunk'
>
>> !!!
>> At this point, you should immediately do a push!  Since you are using
>> trunk as a local mirror of lp:nipy (you are 'pulling' from it, not
>> merging from it), you should then push up so that your local commit is
>> publicly available to others.  Basically your local history should
>> match the one seen on the public site at lp:nipy.
>
> Well, no. I am not ready to share my changes. They are buggy, and may not
> be fully functionnal. I am going to break other peoples code, and get in
> trouble if I do.
>
>> Makes sense?
>
> Nope. Your explaination does make sens, but it simply shows me that I
> haven't solved my real problem. Or am I wrong?

I don't think you are supposed to merge into your local copy of trunk if you are
not ready to push to trunk.
Keep working in trunk-gael and merging trunk into it, instead of the
other direction.

That's my interpretation, keep local copies only as mirrors of the
corresponding
remote branch, and merge from other branches and push only to the
corresponding remote
never to others.

Josef

>
> Gaël
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

On Thu, Sep 3, 2009 at 1:24 PM, joep <email address hidden> wrote:
> I don't think you are supposed to merge into your local copy of trunk if you are
> not ready to push to trunk.
> Keep working in trunk-gael and merging trunk into it, instead of the
> other direction.
>
> That's my interpretation, keep local copies only as mirrors of the
> corresponding
> remote branch, and merge from other branches and push only to the
> corresponding remote
> never to others.
>

That's exactly how I see it: if you aren't ready to push, do not touch
your local mirror of public trunk with anything other than

bzr pull

to sync with other's changes. You *only* merge into your local mirror
of lp:nipy when you are ready to make those changes public. Basically
think of your local copy of trunk as if it were an svn-style checkout
of lp:nipy, where *any* commit will be immediately seen by others.
For changes you aren't ready to share yet, just work in your own
branch, pulling *from* the public trunk as much as you need/want to,
and you should be OK.

Does that make sense? That's how I work and so far I've had no problems.

This discussion actually makes me think it might even be useful to
*really* do a

bzr checkout lp:nipy my-local-trunk

instead of a 'bzr branch', so that the local copy of trunk is really,
really a mirror, and I can't even commit into it without pushing.
This would be a way to safely enforce that constraint.

Mmh, I hadn't thought of this before, but it makes sense now. Am I
missing something?

Cheers,

f

On Thu, Sep 3, 2009 at 8:39 PM, Fernando Perez<email address hidden> wrote:
> On Thu, Sep 3, 2009 at 1:24 PM, joep <email address hidden> wrote:
>> I don't think you are supposed to merge into your local copy of trunk if you are
>> not ready to push to trunk.
>> Keep working in trunk-gael and merging trunk into it, instead of the
>> other direction.
>>
>> That's my interpretation, keep local copies only as mirrors of the
>> corresponding
>> remote branch, and merge from other branches and push only to the
>> corresponding remote
>> never to others.
>>
>
> That's exactly how I see it: if you aren't ready to push, do not touch
> your local mirror of public trunk with anything other than
>
> bzr pull
>
> to sync with other's changes.  You *only* merge into your local mirror
> of lp:nipy when you are ready to make those changes public.  Basically
> think of your local copy of trunk as if it were an svn-style checkout
> of lp:nipy, where *any* commit will be immediately seen by others.
> For changes you aren't ready to share yet, just work in your own
> branch, pulling *from* the public trunk as much as you need/want to,
> and you should be OK.
>
> Does that make sense?  That's how I work and so far I've had no problems.
>
> This discussion actually makes me think it might even be useful to
> *really* do a
>
> bzr checkout lp:nipy my-local-trunk
>
> instead of a 'bzr branch', so that the local copy of trunk is really,
> really a mirror, and I can't even commit into it without pushing.
> This would be a way to safely enforce that constraint.
>
> Mmh, I hadn't thought of this before, but it makes sense now.  Am I
> missing something?

Very dangerous, when I got started with bzr, I managed to screw up
svn.scipy this way. I thought my local branch is isolated and I
force-pushed (not really knowing what I'm doing) and it pushed
straight through to svn.scipy (the bzr branch was a checkout of svn
scipy).
Now, I prefer to have an extra layer of protection for the remote
repository and make remote pushes a careful choice, I can always get a
new copy from the remote branch if I did something wrong with my local
trunk copy.

First rule I learned, if bzr tells me, I'm doing something stupid,
don't override it.

Josef

>
> Cheers,
>
> f
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

Actually, you don't even need a local copy of the trunk. All you need
is a local developer branch that you commit to, push to your local
launchpad account and merge lp:nipy into to keep in sync with changes
in the trunk.

Matthew's diagram explains it well:

http://neuroimaging.scipy.org/site/doc/manual/html/devel/guidelines/bzr_workflow.html#overview

That is, after doing a:
bzr branch lp:nipy ~/src/trunk-dev

On Thu, Sep 3, 2009 at 5:39 PM, Fernando Perez<email address hidden> wrote:
> This discussion actually makes me think it might even be useful to
> *really* do a
>
> bzr checkout lp:nipy my-local-trunk
>
> instead of a 'bzr branch', so that the local copy of trunk is really,
> really a mirror, and I can't even commit into it without pushing.
> This would be a way to safely enforce that constraint.
>
> Mmh, I hadn't thought of this before, but it makes sense now.  Am I
> missing something?

On Thu, Sep 3, 2009 at 7:00 PM, Christopher Burns <email address hidden> wrote:
> Actually, you don't even need a local copy of the trunk.  All you need
> is a local developer branch that you commit to, push to your local
> launchpad account and merge lp:nipy into to keep in sync with changes
> in the trunk.

No, I meant for pushing back upstream, for the case of a developer who
does eventually make direct commits to the trunk. If you want to push
back up, you do need a local copy of the trunk, where you can apply
changes and then push them back, right? I think Matthew's diagram is
for the general case of developers who are not taksed with making the
actual trunk merge/push operations.

My idea of using a checkout instead of a branch does have the problem
that Josef outlined, of lacking any layer of protection between your
local mistakes and the public server. Since one of the features of a
DVCS is that you tend to be more liberal with committing than with SVN
(where you're always very much on guard that a commit has significant
public consequences), I think Josef's point is a very valid one.

Cheers,

f

On Fri, Sep 4, 2009 at 1:27 AM, Fernando Perez<email address hidden> wrote:
> On Thu, Sep 3, 2009 at 7:00 PM, Christopher Burns <email address hidden> wrote:
>> Actually, you don't even need a local copy of the trunk.  All you need
>> is a local developer branch that you commit to, push to your local
>> launchpad account and merge lp:nipy into to keep in sync with changes
>> in the trunk.
>
> No, I meant for pushing back upstream, for the case of a developer who
> does eventually make direct commits to the trunk.  If you want to push
> back up, you do need a local copy of the trunk, where you can apply
> changes and then push them back, right?  I think Matthew's diagram is
> for the general case of developers who are not taksed with making the
> actual trunk merge/push operations.
>

Just to confirm this, here is the hidden part:
http://neuroimaging.scipy.org/site/doc/manual/html/devel/guidelines/bzr_administration.html#bzr-administration

I didn't see this before, even though it's linked from below the other diagram.
It would have been useful to see this when I was trying to figure out
how to handle "statsmodels trunk"

Josef

> My idea of using a checkout instead of a branch does have the problem
> that Josef outlined, of lacking any layer of protection between your
> local  mistakes and the public server.  Since one of the features of a
> DVCS is that you tend to be more liberal with committing than with SVN
> (where you're always  very much on guard that a commit has significant
> public consequences), I think Josef's point is a very valid one.
>
> Cheers,
>
> f
> --
> https://code.launchpad.net/~nipy-developers/nipy/trunk-neurospin/+merge/8472
> Your team nipy-developers is subscribed to branch lp:~nipy-developers/nipy/trunk-neurospin.
>

On Thu, Sep 3, 2009 at 10:39 PM, joep <email address hidden> wrote:
>> No, I meant for pushing back upstream, for the case of a developer who
>> does eventually make direct commits to the trunk.  If you want to push
>> back up, you do need a local copy of the trunk, where you can apply
>> changes and then push them back, right?  I think Matthew's diagram is
>> for the general case of developers who are not taksed with making the
>> actual trunk merge/push operations.
>>
>
> Just to confirm this, here is the hidden part:
> http://neuroimaging.scipy.org/site/doc/manual/html/devel/guidelines/bzr_administration.html#bzr-administration
>
> I didn't see this before, even though it's linked from below the other diagram.
> It would have been useful to see this when I was trying to figure out
> how to handle "statsmodels trunk"

Yup! I'd forgotten that diagram had also been made, that's indeed the
one that applies directly to Gael's original question. Thanks for
bringing it back up to my attention.

Cheers,

f

On Thu, Sep 03, 2009 at 08:24:09PM -0000, joep wrote:
> > Well, no. I am not ready to share my changes. They are buggy, and may not
> > be fully functionnal. I am going to break other peoples code, and get in
> > trouble if I do.

> >> Makes sense?

> > Nope. Your explaination does make sens, but it simply shows me that I
> > haven't solved my real problem. Or am I wrong?

> I don't think you are supposed to merge into your local copy of trunk
> if you are not ready to push to trunk. Keep working in trunk-gael and
> merging trunk into it, instead of the other direction.

So, sorry for being thick, but I am a bit lost with regards to my use
case:

1) I branch out trunk to trunk-gael

2) I work on trunk-gael, 'bzr ci' as I work

3) I bug fix is committed to trunk

4) I need the bug fix, in trunk-gael

5) I work on trunk-gael, 'bzr ci' as I work

6) I am ready for merging in trunk, but how do I avoid history folding?

The question is how to implement 4), without creating history folding and
without affecting trunk.

From the sentence above, I have the impression that I am supposed to do a
'cd trunk-gael ; bzr merge lp:nipy'. I believe this is what I had been
doing, and it creates history folding.

I am sorry, I am really thick.

Gaël

Hey,

On Thu, Sep 3, 2009 at 11:18 PM, Gael Varoquaux
<email address hidden> wrote:
> So, sorry for being thick, but I am a bit lost with regards to my use
> case:
>
> 1) I branch out trunk to trunk-gael
>
> 2) I work on trunk-gael, 'bzr ci' as I work
>
> 3) I bug fix is committed to trunk
>
> 4) I need the bug fix, in trunk-gael
>
> 5) I work on trunk-gael, 'bzr ci' as I work
>
> 6) I am ready for merging in trunk, but how do I avoid history folding?
>
> The question is how to implement 4), without creating history folding and
> without affecting trunk.
>
> >From the sentence above, I have the impression that I am supposed to do a
> 'cd trunk-gael ; bzr merge lp:nipy'. I believe this is what I had been
> doing, and it creates history folding.

Yes, and it does make the history a little messy with a lot of nested
merge messages, but it does not *drop revisions*. The major danger
with the other scenario (doing a local merge in a branch that's meant
to only track a public one and not committing) is that if you later
try to commit, you'll get a "histories have diverged" message from
bzr. If you then say "what the heck, do as I say, --force", you'll
get the real bad case Josef alluded to, where you seriously do mess up
the upstream repository. We did make that mistake last year early in
the transition to lp in ipython, and we learned as Josef says, "when
bzr tells you not to do something stupid, listen" :)

So I think you are only worrying about the convoluted history and
non-linear revision numbers that will result in *your* branch. That
one, there's really no fix for that I know of, but there is no loss
of information, just a nested log and multi-dot revision numbers.
For the trunk, this pattern keeps a fully clean, linearly increasing
revno, like you can see recently for ipython:

https://code.launchpad.net/~ipython-dev/ipython/trunk

Does this help clarify things?

Cheers,

f

On Fri, Sep 04, 2009 at 06:27:08AM -0000, Fernando Perez wrote:
> Does this help clarify things?

Yes, it does. So it seems that my working pattern was the right, but it
is just giving results that I am not terribly happy with.

Sorry for the long discussion.

Gaël

On Thu, Sep 3, 2009 at 11:36 PM, Gael Varoquaux
<email address hidden> wrote:
>> Does this help clarify things?
>
> Yes, it does. So it seems that my working pattern was the right, but it
> is just giving results that I am not terribly happy with.
>
> Sorry for the long discussion.

No worries, no need to apologize. It took us a while to figure this
out and I was myself very frustrated and somewhat angry last year
after a series of mis-steps of my own in ipython, so I totally
understand. And I have to admit I don't really like the shape this
leaves the history in either, but I don't know of a solution for that
part (I'd love to hear if there is one).

From my initial, but still limited, experiences with git, I get the
impression that it produces a 'flatter' history by default due to how
it handles merging. That is in addition to the fact that it has
extensive rebase/rewrite capabilities. But for bzr use, I've just
accepted the above 'messy' history in local branches as a fact of life
for now. I'll be happy to learn of better approaches if an expert
shows me one...

Best,

f

On Fri, Sep 04, 2009 at 07:55:23PM -0000, Matthew Brett wrote:
> The proposal to merge lp:~nipy-developers/nipy/trunk-neurospin into lp:nipy has been updated.

> Status: Needs review => Merged

Thank you Matthew,

Gaël

Hi,

I&#39;d like to add you to my professional network on LinkedIn.

- Alexis

Accept: http://www.linkedin.com/blink?simpleRedirect=3oPdPwSczAUczwVejAMcP4OejkZh4BKrSBQonhFtCVF9ABPkjhPoQpifnBBiShBsC5EsOpQsSlRpRZBt6BSrCAZqSkConhzbmlQqnpKqiRQsSlRpORIrmkZpSVFqSdxsDgCpnhFtCV9pSlipn9Mfm4Ct3kJd65RrT9Gc6AJcjgSqjgSbjRBfP9SbSkLrmZzbCVFp6lHrCBIbDtTtOYLeDdMt7hE&msgID=I7984211734_1&markAsRead=

You received an invitation to connect. LinkedIn will use your email address to make suggestions to our members in features like People You May Know. Unsubscribe here: http://www.linkedin.com/blink?simpleRedirect=t6lKhj8Bp65Mq6dKtm5Ihj8BpmhLoP0Q9j8Td3x2cylMrjRIqm5JpipEqSZLon1Mr71ysn1GoPgZp6Bx9z4Sc30OfmhF9z4JfmhFripQdiQQonlLsCEMqiQNd3pFd3oJfmhFpipQsSlRpRZBt6BSrCAZqSkCkjoPp4l7q5p6sCR6kk4ZrClHrRhAqmQCsSVRfngCt3kJd65RrT9Gc6AJcjgSqjgSbjRBfP9SbSkLrmZzbCVFp6lHrCBIbDtTtOYLeDdMt7hE&amp;msgID=I7984211734_1&amp;markAsRead= Learn why we included this at the following link: http://www.linkedin.com/blink?simpleRedirect=e3wTd3RAimlIoSBQsC4Ct7dBtmtvpnhFtCVFfmJB9CNOlmlzqnpOpldOpmRLt7dRoPRx9DgRbjhxtmZOqz1Fbj4QdCAQdyQZpjYOtyZBbSRLoOVKqmhBqSVFr2VTtTsLbPFMt7hE&msgID=I7984211734_1&markAsRead=
&copy; 2014, LinkedIn Corporation. 2029 Stierlin Ct. Mountain View, CA 94043, USA