Merge lp://qastaging/~julian-edwards/launchpad/log-parser-bug-680463 into lp://qastaging/launchpad
| Status: | Merged |
|---|---|
| Approved by: | Gavin Panella |
| Approved revision: | no longer in the source branch. |
| Merged at revision: | 11988 |
| Proposed branch: | lp://qastaging/~julian-edwards/launchpad/log-parser-bug-680463 |
| Merge into: | lp://qastaging/launchpad |
| Diff against target: |
24 lines (+7/-6) 1 file modified
lib/lp/services/apachelogparser/base.py (+7/-6) |
| To merge this branch: | bzr merge lp://qastaging/~julian-edwards/launchpad/log-parser-bug-680463 |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Gavin Panella (community) | Approve | ||
|
Review via email:
|
|||
Commit message
When parsing Apache logs, figure out the length of gzip log files without having to read them in to memory in their entirety which can cause the process to crash when it eats all the available memory.
Description of the change
Figure out the length of gzip log files without having to read them in to memory.
The existing code tries to read the uncompressed contents of a gzip file into memory in their entirety. This makes the PPA log parser blow up quite horribly as the log files are very large.
Use existing test with:
bin/test -cvv test_apachelogp
QA Plan
-------
I have got a copy of the production log files that cause the crash on dogfood. Running with the fix allows the processing to continue with no increased memory usage as observed in "top".
| Gavin Panella (allenap) | # |
