Running Async Tasks
What Are Task Queues?
Task Queues are great for deploying resource-intensive functions on Beam.
Instead of processing tasks immediately, the task queue enables you to add tasks to a queue and process them later, either sequentially or concurrently.
Creating a Task Queue
You can run any function as a task queue by using the task_queue
decorator:
You’ll be able to access the result.txt
file when the task completes.
Endpoints vs. Task Queues
Endpoints are RESTful APIs, designed for synchronous tasks that can complete in 180 seconds or less. For longer running tasks, you’ll want to use an async task_queue
instead.
Sending Async Requests
Because task queues run asynchronously, the API will return a Task ID.
Example Request
Example Response
Viewing Task Responses
Because task_queue
is async, you will need to make a separate API call to retrieve the task output.
Saving and Returning Output Files
You can save files using Beam’s Output class.
The code below saves a file, wraps it in an Output
, and generates a URL that can be retrieved later:
Retrieving Results
There are two ways to retrieve response payloads:
- Beam makes a webhook request to your server, based on the
callback_url
in your endpoint - Saving an
Output
and calling the/task
API
Webhooks
If you’ve added a callback_url
to your decorator, Beam will fire a webhook to your server with the task response when it completes:
For testing purposes, you can setup a temporary webhook URL using https://webhook.site
Polling for Results
Output
payloads can be retrieved by polling the task
API:
Your Output will be available in the outputs
list in the response:
Retry Behavior
Task Queues include a built-in retry system. If a task fails for any reason, such as out-of-memory error or an application exception, your task will be retried three times before automatically moving to a failed state.
Programmatically Enqueuing Tasks
You can interact with the task queue either through an API (when deployed), or directly in Python through the .put()
method.
This is useful for queueing tasks programmatically without exposing an endpoint.
If invoked directly from your local computer, the code above will produce this output: