Generating 3D Point Clouds

In this example, we’ll demonstrate using Beam to deploy a cloud endpoint for OpenAI’s Point-E, a state-of-the-art model for generating 3D objects.

Setting up the environment

First, you’ll setup your compute environment. You’ll specify:

Compute requirements
Python packages to install in the runtime

app.py

from beam import App, Runtime, Image, Output, Volume

app = App(
    name="pointe",
    runtime=Runtime(
        cpu=1,
        memory="8Gi",
        gpu="A10G",
        image=Image(
            python_packages=[
                "filelock",
                "Pillow",
                "torch",
                "fire",
                "humanize",
                "requests",
                "tqdm",
                "matplotlib",
                "scikit-image",
                "scipy",
                "numpy",
                "clip@git+https://github.com/openai/CLIP.git",
            ],
        ),
    ),
)

Generating point clouds

Make sure you’ve cloned the point-e repo into your working directory — otherwise, the point_e imports won’t resolve.

This code is copied directly from this example notebook in the Point-E repo.

We’ve only made two modifications:

Accept by adding **inputs to the function parameters
Replacing the hardcoded prompt with a dynamic prompt you’ll pass down from the API, as inputs["prompt"]

run.py

import torch
from tqdm.auto import tqdm

from point_e.diffusion.configs import DIFFUSION_CONFIGS, diffusion_from_config
from point_e.diffusion.sampler import PointCloudSampler
from point_e.models.download import load_checkpoint
from point_e.models.configs import MODEL_CONFIGS, model_from_config
from point_e.util.plotting import plot_point_cloud


def generate_mesh(**inputs):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(device)
    print("creating base model...")
    base_name = "base40M-textvec"
    base_model = model_from_config(MODEL_CONFIGS[base_name], device)
    base_model.eval()
    base_diffusion = diffusion_from_config(DIFFUSION_CONFIGS[base_name])

    print("creating upsample model...")
    upsampler_model = model_from_config(MODEL_CONFIGS["upsample"], device)
    upsampler_model.eval()
    upsampler_diffusion = diffusion_from_config(DIFFUSION_CONFIGS["upsample"])

    print("downloading base checkpoint...")
    base_model.load_state_dict(load_checkpoint(base_name, device))

    print("downloading upsampler checkpoint...")
    upsampler_model.load_state_dict(load_checkpoint("upsample", device))

    sampler = PointCloudSampler(
        device=device,
        models=[base_model, upsampler_model],
        diffusions=[base_diffusion, upsampler_diffusion],
        num_points=[1024, 4096 - 1024],
        aux_channels=["R", "G", "B"],
        guidance_scale=[3.0, 0.0],
        model_kwargs_key_filter=("texts", ""),  # Do not condition the upsampler at all
    )

    prompt = inputs["prompt"]
    print(f"generating image for prompt: {prompt}")

    # Produce a sample from the model.
    samples = None
    for x in tqdm(
        sampler.sample_batch_progressive(
            batch_size=1, model_kwargs=dict(texts=[prompt])
        )
    ):
        samples = x

    pc = sampler.output_to_point_clouds(samples)[0]
    with open("mesh.ply", "wb") as f:
        pc.write_ply(f)
        print(pc)

    print("mesh generated")


if __name__ == "__main__":
    prompt = "a bicycle"
    generate_mesh(prompt=prompt)

Deployment

You’ll deploy the app by entering your shell, and running:

beam deploy app.py

Calling the API

If you navigate to the link in the last line of the shell output, you’ll be able to login to your Beam web dashboard and view the details of your deployment.

You’ll call the API by pasting in the cURL command displayed in the browser window.

  curl -X POST --compressed "https://apps.beam.cloud/77n0o" \
   -H 'Authorization: [YOUR_AUTH_TOKEN]' \
   -H 'Content-Type: application/json' \
   -d '{"prompt": "a bicycle"}'

Since you’ve deployed a task queue, the API will return a Task ID.

{"task_id":"774cb94c-05ff-488e-80ab-58a239ce347d"}

Retrieving outputs

Because this function generates a file output, you’ll need to retrieve it. There are two ways to retrieve outputs:

Logging into the dashboard, and viewing the tasks
In the API, by querying the /task endpoint.

In the dashboard, you’ll see your output file appear in the Outputs table when the task is completed.

Click the Download icon, and open the file in a 3D viewing software. You’ll see a 3D point cloud generated from your prompt:

​Setting up the environment

​Generating point clouds

​Deployment

​Calling the API

​Retrieving outputs

Setting up the environment

Generating point clouds

Deployment

Calling the API

Retrieving outputs