Queues

Beam includes a concurrency-safe distributed queue, accessible both locally and within remote containers.

Serialization is done using cloudpickle, so any object that supported by that should work here. The interface is that of a standard python queue.

Because this is backed by a distributed queue, it will persist between runs.

In the example below, we run one function remotely on Beam and another locally. The remote function puts a value in the queue, and the local function pops it out and prints it. The output will be beam me up.

from beam import Queue, function


@function()
def first():
    q = Queue(name="q")
    q.put("beam me up")
    return


@function()
def second():
    q = Queue(name="q")
    print(q.pop())
    return


if __name__ == '__main__':
    first.remote()
    second.local()