Pushing and Pulling from S3
Changelog
- February 10, 2025
- February 8, 2025
- February 7, 2025
- February 6, 2025
- February 5, 2025
- February 4, 2025
- February 3, 2025
- February 2, 2025
- February 1, 2025
- January 30, 2025
- January 29, 2025
- January 28, 2025
- January 27, 2025
- January 26, 2025
- January 25, 2025
- January 24, 2025
- January 21, 2025
- January 20, 2025
- January 17, 2025
- January 16, 2025
- January 15, 2025
- January 14, 2025
- January 13, 2025
- January 11, 2025
- January 10, 2025
- January 9, 2025
- January 8, 2025
- January 7, 2025
- January 3, 2025
- January 2, 2025
- December 27, 2024
- December 20, 2024
- December 19, 2024
- December 18, 2024
- December 16, 2024
- December 12, 2024
- December 11, 2024
- December 10, 2024
- December 6, 2024
- December 4, 2024
- December 3, 2024
- November 30, 2024
- November 27, 2024
- November 25, 2024
- November 23, 2024
- November 22, 2024
- November 21, 2024
- November 19, 2024
- November 18, 2024
- November 14, 2024
- November 13, 2024
- November 12, 2024
- November 11, 2024
- November 7, 2024
- November 4, 2024
- November 3, 2024
- November 1, 2024
- October 31, 2024
- October 30, 2024
- October 29, 2024
- October 28, 2024
- October 24, 2024
- October 22, 2024
- October 21, 2024
- October 18, 2024
- October 17, 2024
- October 16, 2024
- October 15, 2024
- October 12, 2024
- October 11, 2024
- October 9, 2024
- October 8, 2024
- October 7, 2024
- September 23, 2024
- September 4, 2024
- August 8, 2024
- July 22, 2024
- July 11, 2024
- July 2, 2024
- June 24, 2024
- June 14, 2024
- Beam V2 Guide & FAQ
Pushing and Pulling from S3
In this example, we’ll retrieve model weights from S3 and push outputs back to a bucket.
'''
Add the following credentials to your Beam secrets store:
AWS_ACCESS_KEY
AWS_SECRET_ACCESS_KEY
AWS_REGION_NAME
'''
class S3Client:
def __init__(self, bucket_name=None, key=None, file_path=None):
self.bucket_name = bucket_name
self.key = key
self.file_path = file_path
self.session = boto3.Session(
region_name=os.environ["AWS_REGION_NAME"],
aws_access_key_id=os.environ["AWS_ACCESS_KEY"],
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
)
self.s3 = self.session.client("s3")
def upload_object(self, file_path):
with open(file_path, "rb") as data:
return self.s3.upload_fileobj(data, "mybucket", "mykey")
def download_object(self, file_path, bucket_name, key):
object = self.s3.get_object(Bucket=bucket_name, Key=key)
with open(file_path, "wb") as f:
self.s3.download_fileobj(bucket_name, key, f)
return object
Was this page helpful?