Maps
Using Beam’s distributed Map
Beam includes a concurrency-safe distributed map, accessible both locally and within remote containers. Serialization is done using cloudpickle, so any pickleable object will work. The interface is that of a standard python dictionary, but unlike a typical dicitonary it will persist between runs.
Example: Accessing a map locally and remotely
In the following example, we create a distributed map. Our first function is invoked remotely using .remote()
, and it sets the value a key in our map. The second function is invoked locally using .local()
, and it sets another value. Finally, we print the result of our third, remotely invoked function, which retrieves the values we just set.
from beam import Map, function
@function()
def first():
m = Map(name="m")
m["beam"] = "me up"
return
@function()
def second():
m = Map(name="m")
m["speed"] = "of light"
return
@function()
def third():
m = Map(name="m")
return [m["beam"], m["speed"]]
if __name__ == '__main__':
first.remote()
second.local()
print(third.remote())
You can run the example above with python app.py
. The output will be:
['me up', 'of light']
Was this page helpful?