Merge lp://qastaging/~drawks/graphite/graphite-msgpack into lp://qastaging/~graphite-dev/graphite/main

Proposed by Dave Rawks
Status: Work in progress
Proposed branch: lp://qastaging/~drawks/graphite/graphite-msgpack
Merge into: lp://qastaging/~graphite-dev/graphite/main
Diff against target: 173 lines (+65/-9)
6 files modified
carbon/bin/validate-storage-schemas.py (+1/-1)
carbon/conf/carbon.conf.example (+6/-0)
carbon/lib/carbon/conf.py (+5/-0)
carbon/lib/carbon/protocols.py (+30/-1)
carbon/lib/carbon/service.py (+15/-7)
check-dependencies.py (+8/-0)
To merge this branch: bzr merge lp://qastaging/~drawks/graphite/graphite-msgpack
Reviewer Review Type Date Requested Status
Sidnei da Silva (community) Approve
Review via email: mp+92921@code.qastaging.launchpad.net

Description of the change

Added msgpack as a receiver protocol. msgpack is super fast at deserializing and much safer than pickle since it doesn't support serializing things like modules, functions, or class objects. Also added support to enable/disable individual listeners in carbon.conf

New code path can be tested with a simple metric writer like:

import msgpack, socket, time:
s=socket.create_connection(('localhost',2005))
fs=s.makefile()
for second in xrange(int(time.time()) - 86400,int(time.time())):
    msgpack.pack(("foo.bar",(second,1)),fs)
fs.flush()
fs.close()
s.shutdown(socket.SHUT_RDWR)

To post a comment you must log in.
Revision history for this message
Sidnei da Silva (sidnei) wrote :

1. Please move the import of Unpacker from msgpack from module-level to inside connectionMade().
2. (as said on IRC), note that setting the port to 0 effectively disables it, so the 3 new extra settings are not needed. If you do think it's better/more explicit to keep them, then might make sense to remove the 'if port:' from service.py and require setting the ENABLE_* settings instead.

review: Approve
Revision history for this message
Dave Rawks (drawks) wrote :

1. Delayed imports are generally frowned upon as you can have a situation where someone setups up the listener and then it doesn't break until the first connection to it is made. Also, if the import is inside connectionMade then this will result in reimport on every new connection to the listener, this seems less than optimal AND doesn't follow the same code conventions used in the pickle listener.

2. I do this that making enabling/disabling explicit is more clear from a managment point of view. Also I can imagine a world where we have more/modular listeners and not all of them will follow the convention of listening on a tcp/udp port. Just for argument sake, say that there is a listener that eats metrics from a serial port... OTOH removing the check for sane non-zero port numbers in the current code would lead to a situation where we'd have an exception deeper in the code when the bogus value is passed into the actual twisted protocol code.

688. By Dave Rawks

Disable msgpack receiver by default

689. By Dave Rawks

changed shebang line to use python from env instead of harcoded interpreter

Unmerged revisions

689. By Dave Rawks

changed shebang line to use python from env instead of harcoded interpreter

688. By Dave Rawks

Disable msgpack receiver by default

687. By Dave Rawks <email address hidden>

Added config options to enable/disable line,pickle, and msgpack receivers.
made msgpack import conditional on the msgpack receiver being enabled
updated check_dependencies to test for msgpack

686. By Dave Rawks <email address hidden>

Added msgpack protocol linereceiver

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
The diff is not available at this time. You can reload the page or download it.

Subscribers

People subscribed via source and target branches