​
Prerequisites

  • Amazon EKS
  • Karpenter
  • Helm and kubectl
  • Beta9 CLI

​
Dependencies

Beta9 uses an S3-compatible object storage system for its file system. In this example, we’ll deploy localstack.
Without a Localstack license, its data is temporary. If its pod is deleted, the data will be lost. We recommend that you use AWS S3 or something similar.
helm repo add localstack https://localstack.github.io/helm-charts
helm install localstack localstack/localstack --values=- <<EOF
extraEnvVars:
- name: SERVICES
  value: "s3"
enableStartupScripts: true
startupScriptContent: |
  #!/bin/bash
  awslocal s3 mb s3://juicefs
  awslocal s3 mb s3://logs
persistence:
  enabled: true
  storageClass: local-path
  accessModes:
  - ReadWriteOnce
  size: 50Gi
EOF

​
Install Helm Chart

Install the helm chart and open connections to the service. 
# Step 1: Install the chart
helm install beta9 oci://public.ecr.aws/n4e0e1y0/beta9-chart --version 0.1.166

# Step 2: Confirm the pods are running
kubectl get pods -w

# Step 3: Open ports to the http and grpc services
kubectl port-forward svc/beta9-gateway 1993 1994

​
Configure the CLI

Create a new config. 
./beta9
=> Welcome to Beta9! Let's get started 📡

           ,#@@&&&&&&&&&@&/
        @&&&&&&&&&&&&&&&&&&&&@#
         *@&&&&&&&&&&&&&&&&&&&&&@/
   ##      /&&&&&&&&&&&&&@&&&&&&&&@,
  @&&&&&.    (&&&&&&@/    &&&&&&&&&&/
 &&&&&&&&&@*   %&@.      @& ,@&&&&&&&,
.@&&&&&&&&&&&&#        &&*  ,@&&&&&&&&
*&&&&&&&&&&&@,   %&@/@&*    @&&&&&&&&@
.@&&&&&&&&&*      *&@     .@&&&&&&&&&&
 %&&&&&&&&     /@@*     .@&&&&&&&&&&@,
  &&&&&&&/.#@&&.     .&&&    %&&&&&@,
   /&&&&&&&@%*,,*#@&&(         ,@&&
     /&&&&&&&&&&&&&&,
        #@&&&&&&&&&&,
            ,(&@@&&&,

Context Name [default]:
Gateway Host [0.0.0.0]:
Gateway Port [1993]:
Token:
Added new context 🎉!
Confirm the config was created and has a token set. 
cat ~/.beta9/config.ini
[default]
token = <token should be here>
gateway_host = localhost
gateway_port = 1993

​
Setting Configuration Values

Setup your config file. You will need to set a few values in here and create a secret in your cluster, under the beta9 namespace. 
gateway:
  externalURL: https://app.example.com

imageService:
  registryStore: s3
  registryCredentialProvider: aws
  registries:
    s3:
      bucketName: <your-image-bucket-name>
      region: <your-aws-region>
      # keys not needed if using iam with k8s service account (irsa)
      accessKey:
      secretKey:
  runner:
    baseImageTag: 0.1.10
    baseImageName: beta9-runner
    baseImageRegistry: public.ecr.aws/n4e0e1y0

worker:
  imageTag: 0.1.143
  imageName: beta9-worker
  imageRegistry: public.ecr.aws/n4e0e1y0
  serviceAccountName: <k8s service account to use - should be able to access juicefs s3 bucket>

storage:
  mode: juicefs
  juicefs:
    awsS3Bucket: <your-juicefs-bucket>
    # keys not needed if using iam with k8s service account (irsa)
    awsAccessKey:
    awsSecretKey:

​
Mounting Secrets

Once you’ve configured the config and created a secret in K8s, you’ll need to do two more things:
  1. Mount the secret to the gateway by modifying the persistence value in the values.yaml file.
  2. Add an env var to the gateway called CONFIG_PATH that points to where you are mounting the secret.

​
IAM Policies

To access the S3 bucket that you need to setup and configure in the config/secret, you’ll need to also setup an IAM role that a K8s service account can authenticate with. This is called EKS IRSA. Once you figure this out, you’ll need to add an annotation to the K8s service account that points to their IAM role. Here is an example in the values.yaml file: 
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/beta9-role
  name: beta9-role
We recommend saving secrets with the External Secrets Operator, but you can also create secrets manually in the cluster.To create a secret manually, create your secrets file on disk and run kubectl apply like you would normally.

​
Gotchas

  • Make sure your ingress supports GRPC and HTTP
  • Your IAM permissions need to be set correctly. You will need to create S3 buckets manually or in Terraform.
  • If you are using Karpenter for your autoscaler, you’ll need to add a label to the nodes which you want the Beta9 scheduler to pick up.