Merge lp://qastaging/~stewart/drizzle/bug646898-functionengine-mem-usage-position into lp://qastaging/drizzle/7.0
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Brian Aker | ||||
Approved revision: | 2189 | ||||
Merged at revision: | 2196 | ||||
Proposed branch: | lp://qastaging/~stewart/drizzle/bug646898-functionengine-mem-usage-position | ||||
Merge into: | lp://qastaging/drizzle/7.0 | ||||
Diff against target: |
137 lines (+68/-11) 2 files modified
plugin/function_engine/cursor.cc (+62/-9) plugin/function_engine/cursor.h (+6/-2) |
||||
To merge this branch: | bzr merge lp://qastaging/~stewart/drizzle/bug646898-functionengine-mem-usage-position | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Drizzle Developers | Pending | ||
Review via email: mp+50845@code.qastaging.launchpad.net |
Description of the change
greatly reduce the memory usage of function_engine tables when ::position() is used.
by storing packed rows instead of the full row image, we can decrease memory usage by more than a order of magnitude in some situations.
Replace FunctionCursor row_cache of full rows with one that is the
packed row (using similar routines as ARHCIVE does before
compression). This means we use minimal memory for each row. The ref
for ::position() is now a offset into the row_cache buffer instead of
an incrementing index into an array.
This change GREATLY reduces the memory required for various queries on
table function tables.
e.g. (measured using valgrind --tool=massif) running the
information_
BEFORE: peaked at 77.5MB heap usage
AFTER: peaked at 33.7MB heap usage
for schema_dictionary suite:
BEFORE: peaked at 782.6MB heap usage
AFTER: peaked at 31.05MB heap usage
(i.e. memory requirements can be reduced by moer than an order of
magnitude)
A future patch may want to also overflow to disk if needed.
http://
Thanks, in general I think we need to find a generic solution to this problem at some point.