Merge lp://qastaging/~gl-az/percona-xtrabackup/BT23557-2.1-lp1160788 into lp://qastaging/percona-xtrabackup/2.1

23557

jenkins
http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/325/

Some metrics from HP Clout instance:

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 2
Stepping: 3
CPU MHz: 2666.760
BogoMIPS: 5333.52
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0,1

Database was prepped with sysbench parallel-prepare, 16 tables and 5000000 rows for about ~20GB.

Backup was performed with --compress --encrypt=AES256

Serial, no parallel decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress ./backup

  real 10m50.492s
  user 9m51.688s
  sys 0m53.628s

Parallel with 2 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=2 ./backup

  real 5m38.778s
  user 9m50.381s
  sys 0m56.072s

Parallel with 4 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=4 ./backup

  real 5m38.861s
  user 9m50.900s
  sys 0m56.473s

So we see a linear improvement here with CPU availability as long as i/o can keep up

Some metrics from some monster machine provided by Vadim:

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 2100.000
BogoMIPS: 4199.43
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-31

Database was prepped with sysbench parallel-prepare, 32 tables and 2500000 rows for about ~20GB.

Backup was performed with --compress --encrypt=AES256

Serial, no parallel decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress ./backup

  real 7m57.206s
  user 8m14.755s
  sys 0m33.522s

Parallel with 16 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=16 ./backup

  real 2m30.460s
  user 9m50.874s
  sys 11m32.019s

Parallel with 24 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=24 ./backup

  real 2m1.898s
  user 9m58.801s
  sys 14m12.701s

Parallel with 32 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=32 ./backup

  real 2m4.559s
  user 10m20.132s
  sys 21m29.176s

Again, we see a nice improvement up to the point we hit i/o limits...

George Ormond Lorch III <email address hidden> writes:
> Some metrics from some monster machine provided by Vadim:

Blog post coming in 5..4..3..2..1 ? :)

--
Stewart Smith

Looks good. Two minor remarks:

1) I think there should be test which uses --decrypt and --decompress options in one innobackupex call.

2) Lines 498-500: Why there is "DECRYPT AND DECOMPRESS" log output if only decompression is used?

Parallel decompression/decryption are nice. However, are there plans to
add this for streaming? From the tests I see, the decompress/decrypt
are being run on non-streamed backups and since it looks for files with
extensions (qp/xbcrypt), it is not possible to use this on streaming
backups (or earlier streamed backups in case of --encrypt since
decryption needs to be done before xbstream extraction there)

No because qpress is not stored in a streaming compatible format.
xbcrypt can be handled via stream but not qp.
On 7/12/2013 2:09 AM, Raghavendra D Prabhu wrote:
> Parallel decompression/decryption are nice. However, are there plans to
> add this for streaming? From the tests I see, the decompress/decrypt
> are being run on non-streamed backups and since it looks for files with
> extensions (qp/xbcrypt), it is not possible to use this on streaming
> backups (or earlier streamed backups in case of --encrypt since
> decryption needs to be done before xbstream extraction there)
>

--
George O. Lorch III
Software Engineer, Percona
+1-888-401-3401 x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

I shouldn't respond until I have had my full coffee...

There are really two problems in doing this right now. One is that
inobackupex --compress only passes the --compress option(s) down into
xtrabackup where only InnoDB files get compressed, all other files
currently go into the archive uncompressed so doing something like
"... xbcrypt -d ... | xbstream -x | qpress -di ..." can't work in a
pure stream since not all files in the stream are .qp. The other is that
qpress does not store the filename/directory structure within the .qp
and as such can't recreate the same output structure as say xbcrypt and
xbstream can, it basically just a block compress/decompresser.

Maybe one day when all of the decryption | decompression | destreaming
is included in a single executable that is aware of the full structure
and format it would be possible. Alexey might have some ideas on an easy
way to pull this off within innobackupex using some of his mad linux skills.

On 7/12/2013 7:37 AM, George Ormond Lorch III wrote:
> No because qpress is not stored in a streaming compatible format.
> xbcrypt can be handled via stream but not qp.
> On 7/12/2013 2:09 AM, Raghavendra D Prabhu wrote:
>> Parallel decompression/decryption are nice. However, are there plans to
>> add this for streaming? From the tests I see, the decompress/decrypt
>> are being run on non-streamed backups and since it looks for files with
>> extensions (qp/xbcrypt), it is not possible to use this on streaming
>> backups (or earlier streamed backups in case of --encrypt since
>> decryption needs to be done before xbstream extraction there)
>>
>

--
George O. Lorch III
Software Engineer, Percona
+1-888-401-3401 x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

1) Misprint in 477: "oarallel" instead of "parallel";

2) "DECRYPTING AND/OR DECOMPRESSING" in 524;

3) I have not found checking $option_parallel on positive value.

'parallel=i' => \$option_parallel,

It is supposed $option_parallel is integer. In the case if $option_parallel is less then 0 decrypt_decompress() will not work correctly. It seems there will be infinite loop in:

while ($freepidindex >= $option_parallel)

It should be checked somewhere.