Merge lp://qastaging/~verterok/ubuntuone-client/fix-987376 into lp://qastaging/ubuntuone-client

Proposed by Guillermo Gonzalez
Status: Work in progress
Proposed branch: lp://qastaging/~verterok/ubuntuone-client/fix-987376
Merge into: lp://qastaging/ubuntuone-client
Diff against target: 250 lines (+76/-39)
2 files modified
tests/syncdaemon/test_tritcask.py (+40/-13)
ubuntuone/syncdaemon/tritcask.py (+36/-26)
To merge this branch: bzr merge lp://qastaging/~verterok/ubuntuone-client/fix-987376
Reviewer Review Type Date Requested Status
Facundo Batista (community) Approve
Manuel de la Peña (community) Approve
Review via email: mp+103273@code.qastaging.launchpad.net

Commit message

Only use mmap in files of size < 2**31, in order to avoid hitting address space limits, and fallback to standard IO.

Description of the change

Only use mmap in files of size < 2**31, in order to avoid hitting address space limits, and fallback to standard IO.
I'll be working on limiting the file growth in a following branch, to fix Bug #987382.

To post a comment you must log in.
Revision history for this message
Facundo Batista (facundo) wrote :

Thanks!

review: Approve
Revision history for this message
Manuel de la Peña (mandel) wrote :

The following happens on widows:

[ERROR]
Traceback (most recent call last):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\tests\syn
cdaemon\test_tritcask.py", line 214, in test_iter_entries_bad_crc
    for i, entry in enumerate(db.live_file.iter_entries()):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 257, in iter_entries
    for entry in self._iter_entries(fmmap):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 266, in _iter_entries
    entry, new_pos = self.read(fd)
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 323, in read
    data = fd.read(key_sz+value_sz)
exceptions.OverflowError: Python int too large to convert to C long

tests.syncdaemon.test_tritcask.DataFileTest.test_iter_entries_bad_crc
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\tests\syn
cdaemon\test_tritcask.py", line 214, in test_iter_entries_bad_crc
    for i, entry in enumerate(db.live_file.iter_entries()):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 252, in iter_entries
    for entry in self._iter_entries(self.fd):
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 266, in _iter_entries
    entry, new_pos = self.read(fd)
  File "C:\Users\Mandel\Projects\Canonical\ubuntuone-client\fix-987376\ubuntuone
\syncdaemon\tritcask.py", line 323, in read
    data = fd.read(key_sz+value_sz)
exceptions.MemoryError:

tests.syncdaemon.test_tritcask.NoMmapDataFileTest.test_iter_entries_bad_crc

review: Needs Fixing
Revision history for this message
Guillermo Gonzalez (verterok) wrote :

Thanks for spotting that one.
It's fixed and pushed.

Revision history for this message
Manuel de la Peña (mandel) wrote :

Everything works in all currently supported platforms!

review: Approve
Revision history for this message
Ubuntu One Auto Pilot (otto-pilot) wrote :
Download full text (247.1 KiB)

The attempt to merge lp:~verterok/ubuntuone-client/fix-987376 into lp:ubuntuone-client failed. Below is the output from the failed tests.

/usr/bin/gnome-autogen.sh
checking for autoconf >= 2.53...
  testing autoconf2.50... not found.
  testing autoconf... found 2.68
checking for automake >= 1.10...
  testing automake-1.11... found 1.11.3
checking for libtool >= 1.5...
  testing libtoolize... found 2.4.2
checking for intltool >= 0.30...
  testing intltoolize... found 0.50.2
checking for pkg-config >= 0.14.0...
  testing pkg-config... found 0.26
checking for gtk-doc >= 1.0...
  testing gtkdocize... found 1.18
Checking for required M4 macros...
Checking for forbidden M4 macros...
Processing ./configure.ac
Running libtoolize...
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
Running intltoolize...
Running gtkdocize...
Running aclocal-1.11...
Running autoconf...
Running autoheader...
Running automake-1.11...
Running ./configure --enable-gtk-doc --enable-debug ...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for library containing strerror... none required
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to con...

Revision history for this message
Facundo Batista (facundo) wrote :

Tested again latest changes.

review: Approve
Revision history for this message
Ubuntu One Auto Pilot (otto-pilot) wrote :
Download full text (246.2 KiB)

The attempt to merge lp:~verterok/ubuntuone-client/fix-987376 into lp:ubuntuone-client failed. Below is the output from the failed tests.

/usr/bin/gnome-autogen.sh
checking for autoconf >= 2.53...
  testing autoconf2.50... not found.
  testing autoconf... found 2.68
checking for automake >= 1.10...
  testing automake-1.11... found 1.11.3
checking for libtool >= 1.5...
  testing libtoolize... found 2.4.2
checking for intltool >= 0.30...
  testing intltoolize... found 0.50.2
checking for pkg-config >= 0.14.0...
  testing pkg-config... found 0.26
checking for gtk-doc >= 1.0...
  testing gtkdocize... found 1.18
Checking for required M4 macros...
Checking for forbidden M4 macros...
Processing ./configure.ac
Running libtoolize...
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
Running intltoolize...
Running gtkdocize...
Running aclocal-1.11...
Running autoconf...
Running autoheader...
Running automake-1.11...
Running ./configure --enable-gtk-doc --enable-debug ...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for library containing strerror... none required
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to con...

Unmerged revisions

1234. By Guillermo Gonzalez

fix tests

1233. By Guillermo Gonzalez

fix bad_crc tests to run on windows.

1232. By Guillermo Gonzalez

don't iterate the generator in DataFile.iter_entries, just return it.

1231. By Guillermo Gonzalez

fix max mmaped file limit

1230. By Guillermo Gonzalez

only use mmap in data files <= 2**32, to avoid hitting address space limits, and fallback to standard IO.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches