OH: "async won't take hold until it sucks less." It cannot suck less because we are writing async code in languages designed to be procedural, given that you read them top-to-bottom.
I am starting to get more familiar with Twisted and it is indeed very powerful. Even if I don't like it, I will never do any network or async stuff using threads any more. Other libraries may be nicer for web but when you want a variety of protocols Twisted is the only game in town.
The thing is, Python is a procedural language. You write commands top-to-bottom, and they are executed as such. When you want to encapsulate things running in parallel, or at a later time, you put them in a function and register that. There are more things you want to do - block until all results have come in, or fail early if one result fails and so on.
The best way to visualise this is to use some kind of visual support - think arrows flowing around and whatnot - but it will be clunky and hard to implement, and mind you us Vim users will hate it (of course there's going to be an emacs plugin soon ;).
So I don't think async will start to suck less soon, because the basic way we get to interact with it is just the wrong approach. I've no idea what a better approach would be though.
Comments
Comment by David W. , 6 months, 3 weeks ago :
Hi there,
I'm not sure syntax is the correct place for this, perhaps the runtime makes more sense. Coincidentally, have you seen Twisted's inlineCallbacks support? IMHO it is a little hacky, but the code it permits you to write is incredibly concise:
http://blogs.gnome.org/jamesh/2009/01...
As for runtime support (and ignoring Stackless Python for a moment), Google's Go is one shining example of how this can work out well. Here the runtime schedules lightweight threads, which are the only kind available to the Go programmer, on real OS threads as needed.
Comment by Orestis Markou , 6 months, 3 weeks ago :
Doug: perhaps they have their minds inside-out already? Or they didn't have the chance to get fixed to a specific way of programming?
Today I wrote a small web service that given a list of hosts invokes a bunch of rsyncs and couchdb sync commands on them. When all the results are back it goes back to the web client with either 'success' or a list of errors.
This is easy to do with Twisted, but still you need to handle collecting all the errors, putting all the deferreds into a DeferredList and waiting on that... So not sure inline callbacks could be used here.
Not sure about Go/lightweight threads. I don't really do CPU-intensive calculations that would require have thread usage, and I only use threads to interface with 3rd-party blocking code, as little as possible.
A lot of uncharted territory here, I guess.
Comment by Roger , 6 months, 3 weeks ago :
Erlang is written "procedurally" - ie top to bottom straightforward code. The difference is that they have lots of "threads" (known as processes in the Erlang world) and that the "processes" can only communicate by sending messages - you cannot share data. (Since it is functional you can't modify data either which means that the interpreter can share the underlying data without having to worry about locks etc.)
Erlang processes are very lightweight - you can have tens of thousands in 32 bit and virtually unlimited in 64 bit. They are not visible to the operating system (or C libraries for that matter) which is why they are lightweight.
Stackless most closely matches the Erlang model from the Python point of view.
Comment by Ryan Williams , 6 months, 3 weeks ago :
This is exactly the reason that Eventlet was written.
It uses coroutines to implement extremely lightweight greenthreads that run on top of an event-driven hub. Therefore you write your code in a procedural, blocking, manner, and under the hood the library handles waiting for events, resuming the appropriate greenthread when it's ready. It's quite efficient, memory- and speed-wise.
For your use case of hitting a bunch of hosts with rsync, it would probably be a matter of
writing a procedural function that looks a bit like:
def do_rsync_stuff(host):
rsync(host)
couchdb_sync(host, arg1)
couchdb_sync(host, arg2)
....
Then you'd call pool.imap(rsync_function, list_of_hosts), and Eventlet would handle the work of making all those requests go concurrently.
Check out the examples page for some more readable code:
http://eventlet.net/doc/examples.html
Comment by rgz , 6 months, 3 weeks ago :
io seems like the perfect language to me from the way it
handles messages and array maps to concurrency, take a look:
Comment by Pete , 6 months, 3 weeks ago :
Yes. inlineCallbacks is the syntactic support you're looking for. Some may have a problem with the "yield" leaking through the abstraction making it less-than-composable, but conceptually, this delineation is necessary and a Good Thing - the programmer should know whenever any call chain makes async calls.
Concurrency is (mostly) an already-solved problem. I/O bound programs (99.999999% of what you're doing) is solved with Twisted + inlineCallbacks. For CPU bound programs, just use coarse-grained task parallelism when you can with the multiprocessing module. Sharing state will always be hard, so you should try not to, but if you need to, us Pythonistas have ZODB, relational databases, and noSQL transactional datastores.
Comments are not allowed in this post
Comment by Doug , 6 months, 3 weeks ago :
It seems as if Erlang has solved this problem and those who use it don't seem to have problems with turning their heads inside-out the way that some who use Twisted complain about...
Erlang is also procedural (one step after another), so it's not clear that is the core problem.