Merge lp://qastaging/~openerp-dev/openobject-addons/7.0-pos-performance-cbi into lp://qastaging/openobject-addons/7.0

Proposed by Olivier Dony (Odoo)
Status: Needs review
Proposed branch: lp://qastaging/~openerp-dev/openobject-addons/7.0-pos-performance-cbi
Merge into: lp://qastaging/openobject-addons/7.0
Diff against target: 730 lines (+334/-194)
4 files modified
point_of_sale/static/src/js/models.js (+0/-17)
product/pricelist.py (+299/-153)
product/product.py (+33/-22)
product/product_pricelist_demo.yml (+2/-2)
To merge this branch: bzr merge lp://qastaging/~openerp-dev/openobject-addons/7.0-pos-performance-cbi
Reviewer Review Type Date Requested Status
Eneldo Serrata (community) Needs Fixing
Fabien (Open ERP) Disapprove
Olivier Dony (Odoo) surface Needs Fixing
Review via email: mp+148396@code.qastaging.launchpad.net

Description of the change

Fixes the algorithm for pricelist calculation to drastically improve performance.
This is still a work-in-progress (API needs to be corrected not to break the stable API), but currently brings down the time to load 28k products in the POS interfaces (that caches the prices) from 15min to 12sec.

Any feedback appreciated.

To post a comment you must log in.
Revision history for this message
Olivier Dony (Odoo) (odo-openerp) wrote :

The WITH RECURSIVE query requires postgres 8.4, so we might keep the original query for OpenERP 6.1 or slightly improve it. Without it the POS can still load 28k products in 40s.

Revision history for this message
qdp (OpenERP) (qdp) wrote :

let me be a reviewer, this piece of code is soooo exciting o_O

Revision history for this message
Olivier Dony (Odoo) (odo-openerp) wrote :
Download full text (4.9 KiB)

Here are my comments after a surface scan (I did not test):

1/ First, thanks for restoring the compatibility with the original 7.0 API. However the way price_get_multi now works is not equivalent to the original code. As I understand it, price_get_multi() was working as follows:
  - IN: `pricelist_ids`: list of pricelists to compute
  - IN: `products_by_qty_by_partner`: list of (product,qty,partner) to compute, with each product supposed to be provided only once
  - OUT: { prod_id1: {pricelist_id1: price, pricelist_id2: price},
           prod_id2: {pricelist_id1: price, pricelist_id2: price},
           (...)
           'item_id': id_of_last_pricelist_line_matched }

where the returned prices will match the qty and partner that was provided *in the same tuple*.
Yes this API is nonsense (especially the 'item_id') and due to debatable past refactorings, but that's how it is :-(

With the current state of this merge proposal it seems that `products_by_qty_by_partner` is only used to get the list of products to compute, while the qty and partner are forced to be the ones provided as keyword arguments. So it will work only if partner and qty are the same for all products being computed.

I understand that it is hard to optimize the query for fetching the `product.pricelist.item`s if the qty and partner can be different for each product, so maybe we should not do it that way.

Could we instead do the following:
 - the price_get_batch() method becomes responsible for computing the price for several products for a single `qty` and `partner`, and all the code moves in it. The signature would be:
     def price_get_batch(self, cr, uid, ids, prod_ids, qty, partner=None, context=None)
 - the price_get_multi() method keeps the original 7.0 signature (no extra parameter), but it groups all entries from `products_by_qty_by_partner` with the same `qty` and `partner` and passes all the groups to price_get_batch().
 - this way the API still works exactly as before and the optimization works too because the POS will call it with qty=1 and partner=None for all products, so there will only be one group to pass to price_get_batch().

2/ what is the point of adding the extra argument "pricelist_version_ids" to price_get_batch() and price_get_multi()? It forces you to copy the search() for versions in product._product_price() and in pricelist.price_get(). On the other hand it seems to be only computed once, so it could still be done at the beginning of price_get_multi(). Why does price_get() now ignore the `date` in context?

4/ the big `if` with the recursive query should probably be moved to a separate private method that abstracts the logic of computing res_per_prod. i.e.:
   res_per_prod = self._pricelist_table(self, cr, uid, pricelist_ids, products, partner, qty, ...)
The _pricelist_table() method could then be split into 2 methods _pricelist_table_pg83() and _pricelist_table_pg84() that implement each variant of the query.

5/ (not in your code, was already present) l.433-449: the logic to find the proper supplier for a given product should not be hardcoded, the product.product model has a `seller_id` function field that provides this info, so it should be...

Read more...

review: Needs Fixing (surface)
Revision history for this message
Kevin Shenk (batonac) wrote :

If this code is fixed and production ready, when will it be merged into the main edition?

Revision history for this message
van der Essen Frédéric (OpenERP) (fva-openerp) wrote :

Any estimation on when this is going to be merged ? This bug is a showstopper for a _lot_ of users.

Revision history for this message
Fabien (Open ERP) (fp-tinyerp) wrote :

This was an important improvement, but a better fix has been made in trunk-website-al.

review: Disapprove
Revision history for this message
Olivier Dony (Odoo) (odo-openerp) wrote :

The optimized WITH RECURSIVE query and the workaround for seller_info_id field are still quite useful on top of the future trunk improvement [1], so setting back to Needs Review so that at least those changes are reviewed at some point for trunk.
The branch is also usable as is for 7.0 in the mean time (and probably 6.1), for customers affected by slow performance during POS initial load.

[1] http://bazaar.launchpad.net/~<email address hidden>

Revision history for this message
Eneldo Serrata (eneldoserrata) wrote :

Hello Oliver i try to patch but i get conflict i'm using Versión 7.0-20131224-002412 any recommendation thanks

review: Needs Fixing

Unmerged revisions

8605. By Chris Biersbach (OpenERP)

[FIX] Now the fall-back to 8.3 is also functional again

8604. By Chris Biersbach (OpenERP)

[IMP] Added a comment

8603. By Chris Biersbach (OpenERP)

[IMP] Confusing read and search is a surefire sign you do not drink enough coffee...

8602. By Chris Biersbach (OpenERP)

[IMP] Removed some remaining debug code

8601. By Chris Biersbach (OpenERP)

[IMP] This takes into account the comments by ODO... various refactorings and small fixes

8600. By Chris Biersbach (OpenERP)

[IMP] Changed some method signatures and call to be in sync with the situation before my fixes

8599. By Chris Biersbach (OpenERP)

[FIX] small bug in the fall-back code

8598. By Chris Biersbach (OpenERP)

[IMP] Refactored the codeto include the fallback gor PostgreSQL prior to 8.4

8597. By Chris Biersbach (OpenERP)

[IMP] last small glitch, now all tests should go through. Also removed the debug code in the JS

8596. By Chris Biersbach (OpenERP)

[IMP] Another small improvement

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.