Percona XtraBackup moved to https://jira.percona.com/projects/PXB

Merge lp://qastaging/~stewart/percona-xtrabackup/test-combinations into lp://qastaging/percona-xtrabackup/2.1

test-combinations
Merge into 2.1

Proposed by Stewart Smith on 2013-06-24

Status:	Rejected
Rejected by:	Alexey Kopytov on 2013-08-06
Proposed branch:	lp://qastaging/~stewart/percona-xtrabackup/test-combinations
Merge into:	lp://qastaging/percona-xtrabackup/2.1
Prerequisite:	lp://qastaging/~stewart/percona-xtrabackup/source-dist-refactor
Diff against target:	722 lines (+309/-95) 4 files modified test/bootstrap.sh (+39/-62) test/inc/common.sh (+39/-1) test/testrun.c (+153/-14) test/testrun.sh (+78/-18)
To merge this branch:	bzr merge lp://qastaging/~stewart/percona-xtrabackup/test-combinations
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Alexey Kopytov (community)		2013-06-24	Disapprove on 2013-08-06
Review via email: mp+171012@code.qastaging.launchpad.net

Description of the change

This branch enables the Percona XtraBackup test suite to be run against multiple servers in one run - instead of having to have many build+test jobs in Jenkins per platform, you can have just one. I also fix the problem with trying to run the test suite from a binary tarball (produced by build-binary.sh). With this branch, you can build a source tarball, a binary tarball and then run the test suite against that binary tarball with all server versions we currently support.

I also improve testrun.sh execution time by about 5 seconds per invocation by caching the results of get_version_info() (using md5sum of mysqld as the key for the cache).

I also fix a long standing problem of testrun leaking files and reserved port numbers, finding a good solution that appears to work really quite well in my testing.

Most of the 5.6 related tests currently fail due to "libssl.so.0.9.8: cannot open shared object file: No such file or directory"

An example run is at:
http://jenkins.percona.com/job/percona-xtrabackup-2.1-stewart-test/6/

This branch depends on George's recent fixes and my source-dist-refactor branch. Hopefully the diff comes out okay on LP.

Revision history for this message

Alexey Kopytov (akopytov) wrote on 2013-07-08:

Stewart,

What's the benefit of testing multiple servers in a single run?

review: Needs Information

Revision history for this message

Stewart Smith (stewart) wrote on 2013-07-08:

Alexey Kopytov <email address hidden> writes:
> What's the benefit of testing multiple servers in a single run?

Smaller matrix (easier to see failures) as well as improved parallelism
as you always use all the workers right up until the very end of all the
runs rather than having to wait for that last test to complete for each
server.

The other benefit is that it makes it a little easier to test the
binaries we ship in one simple command rather than several commands.

--
Stewart Smith

Revision history for this message

Alexey Kopytov (akopytov) wrote on 2013-08-06:

Looking at the Jenkins links, I think this change makes it rather difficult to interpret the results. Which IMO outweighs the benefits. We can also provide a script to make it easier to test all binaries with one simple command.

review: Disapprove

Revision history for this message

Stewart Smith (stewart) wrote on 2013-08-06:

Alexey Kopytov <email address hidden> writes:
> Review: Disapprove
>
> Looking at the Jenkins links, I think this change makes it rather
> difficult to interpret the results. Which IMO outweighs the
> benefits. We can also provide a script to make it easier to test all
> binaries with one simple command.

(it probably no longer applies too).

We can likely accomplish the speed improvement through the following
instead:
1) one job builds on each platform
2) second job grabs build for platform and in its matrix tests each
server type we want (similar to current job)

--
Stewart Smith

Unmerged revisions

623. By Stewart Smith on 2013-06-20: remove incorrect free of servers.path in testrun.c
622. By Stewart Smith on 2013-06-20: protect against any concurrent runs of testrun.sh clobbering each other badly in xb_version_info_cache
621. By Stewart Smith on 2013-06-20: cache get_version_info in testrun.sh in a xb_version_info_cache/ directory with the md5sum of the mysqld binary being the filename with the cache. Even though md5 sucks, it should be okay for this purpose.
620. By Stewart Smith on 2013-06-20: replace . with _ in server name in testrun.c as . is used as test suite delimiter by jenkins junit.xml ui
619. By Stewart Smith on 2013-06-20: redo how we free leftover sockets and ports in testrun. We now keep file (for port) and a symlink (for socket) in the var/ directory for each mysqld. In kill_leftovers we then properly clean these up.
618. By Stewart Smith on 2013-06-20: in get_version_info() stop the server earlier before there are any exit points so we don't leak running servers
617. By Stewart Smith on 2013-06-20: If running against MySQL 5.1, ensure we're using the plugin
616. By Stewart Smith on 2013-06-20: only use xtrabackup_innodb55 for innodb55 target if that binary exists.
615. By Stewart Smith on 2013-06-20: implement a cleanup_leftovers() function in common.sh that is called at end of a test rather than kill_leftovers so that we free the reserved TCP port for each still running mysqld.
614. By Stewart Smith on 2013-06-14: add an 'all' target to test/bootstrap.sh that will pull all the current server binaries for current architecture into servers/