- Simple, modern networking broker
- Built-in support to libzrt
- http server
- nginx
- fastcgi
- libzrt
- 1 http frontend + 1 daemon backend
- factor some
object-query
functionality into this - need good multiplexing file/stream transfer
- no discovery needed
- in the ZeroCloud case, we already know the cluster topology
- two main functions:
- register
- register(channel_id)
- unregister(chanel_id)
- transfer
- send(ip:port, channel_id_src, channel_id_dest, data, size)
- recv(ip:port, channel_id_src, channel_id_dest, size) -> data
- channel_ids are just opaque identifiers, or a "label" for messages
- register
- NOT a queue
- only contains 0..1 messages at a given time
- block receivers until message is available
- use case: MapReduce
- Mapper should not be able to fill memory with all mapped data if the reducer is not ready to consume it
- use case: MapReduce
- Nothing should happen until somebody wants to consume something
- Channels are UNIdirectional
- we are limited to this because of pipe semantics
- CSP supports bidirectional channels
- but it's not deterministic
- "select" operator
- "read from any channel in the list"
- data = select(list)
- not supported now by ZeroVM, but could be
- recv_any([ip_list], [channel_list])
- broker-to-broker communication
- http: PUT/GET /broker/channel_id
register
lets brokers learn about each other
proxy-query
sends list of brokers and channels in each execution request- proxy knows it because it always has the complete cluster topology
- broker also passes the list of other brokers ip:port tuples to the ZeroVM process/thread
- Iterface for broker <--> ZeroVM communication
- register
- unregister
- send
- recv
- No routing
- ip:port for each remote end of channel is known on job start
- broker connects there directly, or reuses existing connection to that ip:port
- Might need multiple redundant connections
- Not for each ZeroVM instance, though
- (This would be a good way to run out of file descriptors.)
- For QoS
- Possible one connection for each message-size class, e.g.:
- 64 bytes
- 128 bytes
- 512 bytes
- 1024 bytes
-
1024 bytes
- needed so that long data transfers don't stall short ones
- Or, just "< 1024 bytes" and ">= 1024 bytes"
- 2 of each connection
- round-robin between them
- this is the typical strategy people implement to speed up page loads in HTTP
- Not for each ZeroVM instance, though
- Push vs. Pull
- We can decide based on message size
- Push:
- good for latency
- application calls
write(fd, data)
- fd translated into channel_id
- channel_id translated to remote IP
- broker issues
send
- when
send
comes to remote broker:- reads channel_id
- translates it into ZeroVM process/thread ID
- if
recv
was issued, the send is accepted - if not, tell the other party (sender) to wait/buffer
- Pull:
- good for long transfers
- only need a buffer with size "small_message" (see message sizes above)
- accepts first
send()
unconditionally- or else a deadlock will happen often
- broker: I got a
send()
- broker: Did I get a
recv()
for that? - If not, buffer it.
- If no buffer, block it.
Other notes:
- ZeroVM must support sending channels over channels. This would fix David Holland's inter-instance communication problem without breaking determinism.
- Have a look at process calculi