BEAM clustering

Cluster your Elixir/Erlang services with a single ops.json key so nodes discover each other over distributed Erlang.

Elixir and Erlang applications often need their nodes to find each other — to hand work to a peer, to run a distributed registry, or to broadcast over Phoenix PubSub. On LocalOps, you get there by adding one key to ops.json in each service that should be part of the cluster.

This page assumes your service is already deploying on LocalOps. If you’re still packaging your app, start with the Elixir/Phoenix guide.

Enable it

ops.json

{
  "beam_cluster_id": "myapp"
}

Use the same value in every service you want clustered together — your web service, your worker, your internal RPC service. LocalOps then labels all their pods alike and creates a shared headless service whose DNS returns every member pod’s IP.

What LocalOps injects

Two environment variables are injected into your pods:

Variable	Value
LOPS_BEAM_CLUSTER_NAME	myapp
LOPS_BEAM_CLUSTER_DNS	myapp-cluster.app-services.svc.cluster.local

app-services is the Kubernetes namespace all your services run in within the environment.

Wire up libcluster

Read the DNS name from the environment — don’t hardcode it.

config/runtime.exs

config :libcluster,
  topologies: [
    k8s: [
      strategy: Cluster.Strategy.Kubernetes.DNS,
      config: [
        service: System.get_env("LOPS_BEAM_CLUSTER_DNS"),
        application_name: "myapp",
        polling_interval: 5_000
      ]
    ]
  ]

Always read LOPS_BEAM_CLUSTER_DNS at run time. The DNS name is derived from your beam_cluster_id and differs between your production environment and each preview environment, so a hardcoded value will silently fail to find peers somewhere.

Your release config stays yours

LocalOps does not set RELEASE_DISTRIBUTION, RELEASE_NODE, or RELEASE_COOKIE. Set these yourself — typically RELEASE_DISTRIBUTION=name with RELEASE_NODE=myapp@${POD_IP}. POD_IP is already injected into every service, so you can reference it directly.

Use the same RELEASE_COOKIE across every service in the cluster, or nodes will refuse each other. Add it as a secret with the same value in each service.

Cluster across regions

The headless service above only resolves inside one environment’s Kubernetes cluster, so nodes in your Mumbai environment can’t see nodes in Ireland. To form a single mesh across regional environments, give each environment a DNS name in a private zone that every regional VPC can resolve, and let LocalOps keep that name pointed at the environment’s pods.

The steps below use Mumbai (ap-south-1) and Ireland (eu-west-1) as example regions. Substitute the regions, CIDRs, and names that fit your setup.

1. Spin up an environment in each region

   In LocalOps, create an environment in every region the cluster should span — say one in ap-south-1 (Mumbai) and one in eu-west-1 (Ireland) — and deploy the same services to each.

   Every environment gets its own VPC, and by default every VPC uses 10.0.0.0/16. Peered VPCs can’t have overlapping CIDRs, so open Advanced options on the Create environment page and give each environment its own block — for example 10.10.0.0/16 for Mumbai and 10.20.0.0/16 for Ireland.

   The CIDR is fixed once the environment is created. An existing environment on the default 10.0.0.0/16 can’t be peered with another one on the same range — create a new environment with a distinct CIDR instead.

2. Set up VPC peering between the regions (AWS)

   DNS gets nodes to each other’s addresses; peering is what gets packets there. For every pair of regions:

   1. Request the peering connection. In the VPC console of the first region (Mumbai), go to Peering connections → Create peering connection. Pick the Mumbai environment’s VPC as the requester, choose Another Region (and Another account, if the environments live in different AWS accounts), and enter the Ireland environment’s VPC ID.

   2. Accept it. Switch the console to the other region (Ireland), open Peering connections, select the pending request, and choose Actions → Accept request.

   3. Update the private subnets’ route tables, on both sides. Your pods run in the environment’s private subnets, so their route tables need a route to the peer VPC. In Route tables, filter by each environment’s VPC and edit every route table associated with a private subnet — LocalOps creates one per availability zone — adding:

      Environment	Destination	Target
      Mumbai	10.20.0.0/16 (Ireland)	the peering connection (pcx-...)
      Ireland	10.10.0.0/16 (Mumbai)	the peering connection (pcx-...)

      Leave the public subnets’ route tables alone — nodes never talk across regions from there.

   Peering isn’t transitive — with three regions you need three peering connections (Mumbai↔Ireland, Mumbai↔Virginia, Ireland↔Virginia), each with routes on both sides. Past a handful of regions, a Transit Gateway is simpler to manage than a full mesh of peerings.

   You can do the same from the CLI:

   # In Mumbai: request
   aws ec2 create-vpc-peering-connection --region ap-south-1 \
     --vpc-id vpc-mumbai --peer-vpc-id vpc-ireland --peer-region eu-west-1
   
   # In Ireland: accept
   aws ec2 accept-vpc-peering-connection --region eu-west-1 --vpc-peering-connection-id pcx-0abc123
   
   # Repeat for every private route table in each VPC
   aws ec2 create-route --region ap-south-1 --route-table-id rtb-mumbai-private-a \
     --destination-cidr-block 10.20.0.0/16 --vpc-peering-connection-id pcx-0abc123
   aws ec2 create-route --region eu-west-1 --route-table-id rtb-ireland-private-a \
     --destination-cidr-block 10.10.0.0/16 --vpc-peering-connection-id pcx-0abc123
   
3. Open BEAM ports in the EKS cluster security groups

   With peering in place, packets reach the other VPC but are dropped at the pods unless the security group allows them. Nodes need EPMD (TCP 4369) plus the distribution ports open to the peer region.

   Erlang picks a random distribution port by default, so first pin it to a fixed range in your release config:

   config/runtime.exs

   config :kernel,
     inet_dist_listen_min: 9000,
     inet_dist_listen_max: 9010
   

   Then, in each region’s VPC console, open Security groups and filter by the environment’s VPC. LocalOps creates a security group along with the EKS cluster, and its name contains eks-cluster — that’s the group attached to your pods. Add these inbound rules, with the peer region’s CIDR as the source:

   Security group	Type	Port range	Source
   Mumbai eks-cluster	Custom TCP	4369	10.20.0.0/16 (Ireland)
   Mumbai eks-cluster	Custom TCP	9000-9010	10.20.0.0/16 (Ireland)
   Ireland eks-cluster	Custom TCP	4369	10.10.0.0/16 (Mumbai)
   Ireland eks-cluster	Custom TCP	9000-9010	10.10.0.0/16 (Mumbai)

   With more regions, add the same pair of rules for each peer region’s CIDR.

   If nodes resolve each other’s IPs but Node.list() stays empty, check these rules first — a missing rule shows up as a connection timeout, not a DNS failure.

4. Create one private zone shared by every regional VPC

   On AWS, create a Route 53 private hosted zone — say mesh.acmecorp.internal — and associate it with the VPC of every regional environment you spin up. One zone, many VPC associations. If your environments live in different AWS accounts, use cross-account zone association. On GCP, the equivalent is a Cloud DNS private zone with each regional VPC added to its visibility list.

   The zone has to be associated with a region’s VPC before nodes there can resolve peers.

5. Give each environment a name in ops.json

   Alongside beam_cluster_id, set beam_dns_name to the record this environment should own inside that zone:

   ops.json

   {
     "beam_cluster_id": "b2bapp",
     "beam_dns_name": "mumbai.mesh.acmecorp.internal"
   }
   

   Keep beam_cluster_id identical everywhere — it’s still what makes these nodes one cluster. beam_dns_name is per environment: mumbai.mesh.acmecorp.internal in Mumbai, ireland.mesh.acmecorp.internal in Ireland, and so on.

   LocalOps creates that record and keeps it in sync — it publishes the IPs of every pod in the environment carrying that beam_cluster_id, and updates the record as pods scale, restart, or get rescheduled. You don’t manage the record set yourself.

6. List every region in your topologies

   Add one topology per region so libcluster polls all of them. LOPS_BEAM_CLUSTER_DNS still points at the local headless service, so keep using it for this environment’s own nodes and name the peer regions by their beam_dns_name:

   config/runtime.exs

   config :libcluster,
     topologies: [
       local: [
         strategy: Cluster.Strategy.Kubernetes.DNS,
         config: [
           service: System.get_env("LOPS_BEAM_CLUSTER_DNS"),
           application_name: "b2bapp",
           polling_interval: 5_000
         ]
       ],
       mum: [
         strategy: Cluster.Strategy.Kubernetes.DNS,
         config: [
           service: "mumbai.mesh.acmecorp.internal",
           application_name: "b2bapp",
           polling_interval: 5_000
         ]
       ],
       ire: [
         strategy: Cluster.Strategy.Kubernetes.DNS,
         config: [
           service: "ireland.mesh.acmecorp.internal",
           application_name: "b2bapp",
           polling_interval: 5_000
         ]
       ]
     ]
   

   The same config ships to every region — each node finds its local peers through LOPS_BEAM_CLUSTER_DNS and every remote region through that region’s record. Nodes connect from both ends, and the mesh forms.

7. Verify the mesh from one region

   Once the new build is deployed in every region, use ops connect to open a shell on a pod in one region’s environment:

   ops connect --env mumbai --svc web
   

   From that shell, attach a remote console to the running release and list the connected nodes:

   bin/b2bapp remote
   

   Node.list()
   

   You should see nodes from every region, not just this one — for example b2bapp@10.10.x.x alongside b2bapp@10.20.x.x. If only local nodes show up, recheck the peering routes and the security group rules above.

The RELEASE_COOKIE has to match across regions, not just across services in one region.

Preview environments ignore beam_dns_name. They keep forming their own single-region cluster, for the same reason they stay out of your production cluster.

Things to know

It takes effect on your next build

LocalOps snapshots ops.json when it builds your service, so redeploying your current image won’t pick up a newly added beam_cluster_id. Push the change and let it build.

Preview environments get their own cluster

A preview environment forms its own separate cluster and never joins your production cluster. This is deliberate — otherwise a preview would share your cookie and be able to call into live nodes. See ephemeral databases and services for pull request previews.

Job and cron services join too

Job and cron services join the cluster as well, so scheduled and one-off tasks can call into the running cluster. If you’d rather they stayed out, just leave beam_cluster_id out of those services’ ops.json.

Two caveats when they are in. Each run is a join-then-leave, which is a cluster-wide event — :global locks the cluster on every nodeup, and Horde/Swarm rebalance their registries — so a frequent schedule means constant churn. And those pods join as full peers, able to run code on any other node.

Processes with their own image stay out

A processes entry that specifies its own image stays out of the cluster, since it isn’t running your app.

Verify it worked

Get a shell on a running pod with ops connect, open a remote console, and run:

ops connect --env production --svc web
bin/myapp remote

Node.list()

It should return your peers. The cluster’s name is in LOPS_BEAM_CLUSTER_NAME.

To watch inter-node distribution traffic over time, see the Erlang & Elixir instrumentation guide — the Erlang-Distribution Grafana dashboard it links is built for exactly this.