> ## Documentation Index
> Fetch the complete documentation index at: https://docs.localops.co/llms.txt
> Use this file to discover all available pages before exploring further.

# BEAM clustering

> Cluster your Elixir/Erlang services with a single ops.json key so nodes discover each other over distributed Erlang.

Elixir and Erlang applications often need their nodes to find each other — to hand work to a peer, to run a distributed
registry, or to broadcast over Phoenix PubSub. On LocalOps, you get there by adding one key to `ops.json` in each
service that should be part of the cluster.

<Note>
  This page assumes your service is already deploying on LocalOps. If you're still packaging your app, start with the
  [Elixir/Phoenix guide](/guides/languages/elixir).
</Note>

## Enable it

```json ops.json theme={null}
{
  "beam_cluster_id": "myapp"
}
```

Use the same value in every service you want clustered together — your web service, your worker, your internal RPC
service. LocalOps then labels all their pods alike and creates a shared headless service whose DNS returns every member
pod's IP.

## What LocalOps injects

Two environment variables are injected into your pods:

| Variable | Value |
| - | - |
| `LOPS_BEAM_CLUSTER_NAME` | `myapp` |
| `LOPS_BEAM_CLUSTER_DNS` | `myapp-cluster.app-services.svc.cluster.local` |

`app-services` is the Kubernetes namespace all your services run in within the environment.

## Wire up libcluster

Read the DNS name from the environment — don't hardcode it.

```elixir config/runtime.exs theme={null}
config :libcluster,
  topologies: [
    k8s: [
      strategy: Cluster.Strategy.Kubernetes.DNS,
      config: [
        service: System.get_env("LOPS_BEAM_CLUSTER_DNS"),
        application_name: "myapp",
        polling_interval: 5_000
      ]
    ]
  ]
```

<Warning>
  Always read `LOPS_BEAM_CLUSTER_DNS` at run time. The DNS name is derived from your `beam_cluster_id` and differs
  between your production environment and each preview environment, so a hardcoded value will silently fail to find
  peers somewhere.
</Warning>

## Your release config stays yours

LocalOps does not set `RELEASE_DISTRIBUTION`, `RELEASE_NODE`, or `RELEASE_COOKIE`. Set these yourself — typically
`RELEASE_DISTRIBUTION=name` with `RELEASE_NODE=myapp@${POD_IP}`. `POD_IP` is already injected into every service, so you
can reference it directly.

<Warning>
  Use the same `RELEASE_COOKIE` across every service in the cluster, or nodes will refuse each other. Add it as a
  [secret](/environment/services/secrets) with the same value in each service.
</Warning>

## Cluster across regions

The headless service above only resolves inside one environment's Kubernetes cluster, so nodes in your Mumbai
environment can't see nodes in Ireland. To form a single mesh across regional environments, give each environment a DNS
name in a private zone that every regional VPC can resolve, and let LocalOps keep that name pointed at the environment's
pods.

<Note>
  The steps below use Mumbai (`ap-south-1`) and Ireland (`eu-west-1`) as example regions. Substitute the regions,
  CIDRs, and names that fit your setup.
</Note>

<Steps>
  <Step title="Spin up an environment in each region" icon="server">
    In LocalOps, [create an environment](/environment/create) in every region the cluster should span — say one in
    `ap-south-1` (Mumbai) and one in `eu-west-1` (Ireland) — and deploy the same services to each.

    Every environment gets its own VPC, and by default every VPC uses `10.0.0.0/16`. Peered VPCs can't have overlapping
    CIDRs, so open **Advanced options** on the Create environment page and give each environment its own block — for
    example `10.10.0.0/16` for Mumbai and `10.20.0.0/16` for Ireland.

    <Warning>
      The CIDR is fixed once the environment is created. An existing environment on the default `10.0.0.0/16` can't be
      peered with another one on the same range — create a new environment with a distinct CIDR instead.
    </Warning>
  </Step>

  <Step title="Set up VPC peering between the regions (AWS)" icon="link">
    DNS gets nodes to each other's addresses; peering is what gets packets there. For every pair of regions:

    1. **Request the peering connection.** In the VPC console of the first region (Mumbai), go to **Peering
       connections → Create peering connection**. Pick the Mumbai environment's VPC as the requester, choose **Another
       Region** (and **Another account**, if the environments live in different AWS accounts), and enter the Ireland
       environment's VPC ID.
    2. **Accept it.** Switch the console to the other region (Ireland), open **Peering connections**, select the pending
       request, and choose **Actions → Accept request**.
    3. **Update the private subnets' route tables, on both sides.** Your pods run in the environment's private subnets,
       so their route tables need a route to the peer VPC. In **Route tables**, filter by each environment's VPC and edit
       every route table associated with a private subnet — LocalOps creates one per availability zone — adding:

       | Environment | Destination | Target |
       | - | - | - |
       | Mumbai | `10.20.0.0/16` (Ireland) | the peering connection (`pcx-...`) |
       | Ireland | `10.10.0.0/16` (Mumbai) | the peering connection (`pcx-...`) |

       Leave the public subnets' route tables alone — nodes never talk across regions from there.

    Peering isn't transitive — with three regions you need three peering connections (Mumbai↔Ireland,
    Mumbai↔Virginia, Ireland↔Virginia), each with routes on both sides. Past a handful of regions, a Transit Gateway
    is simpler to manage than a full mesh of peerings.

    You can do the same from the CLI:

    ```bash theme={null}
    # In Mumbai: request
    aws ec2 create-vpc-peering-connection --region ap-south-1 \
      --vpc-id vpc-mumbai --peer-vpc-id vpc-ireland --peer-region eu-west-1

    # In Ireland: accept
    aws ec2 accept-vpc-peering-connection --region eu-west-1 --vpc-peering-connection-id pcx-0abc123

    # Repeat for every private route table in each VPC
    aws ec2 create-route --region ap-south-1 --route-table-id rtb-mumbai-private-a \
      --destination-cidr-block 10.20.0.0/16 --vpc-peering-connection-id pcx-0abc123
    aws ec2 create-route --region eu-west-1 --route-table-id rtb-ireland-private-a \
      --destination-cidr-block 10.10.0.0/16 --vpc-peering-connection-id pcx-0abc123
    ```
  </Step>

  <Step title="Open BEAM ports in the EKS cluster security groups" icon="shield">
    With peering in place, packets reach the other VPC but are dropped at the pods unless the security group allows
    them. Nodes need EPMD (TCP `4369`) plus the distribution ports open to the peer region.

    Erlang picks a random distribution port by default, so first pin it to a fixed range in your release config:

    ```elixir config/runtime.exs theme={null}
    config :kernel,
      inet_dist_listen_min: 9000,
      inet_dist_listen_max: 9010
    ```

    Then, in each region's VPC console, open **Security groups** and filter by the environment's VPC. LocalOps creates
    a security group along with the EKS cluster, and its name contains `eks-cluster` — that's the group attached to
    your pods. Add these inbound rules, with the **peer** region's CIDR as the source:

    | Security group | Type | Port range | Source |
    | - | - | - | - |
    | Mumbai `eks-cluster` | Custom TCP | `4369` | `10.20.0.0/16` (Ireland) |
    | Mumbai `eks-cluster` | Custom TCP | `9000-9010` | `10.20.0.0/16` (Ireland) |
    | Ireland `eks-cluster` | Custom TCP | `4369` | `10.10.0.0/16` (Mumbai) |
    | Ireland `eks-cluster` | Custom TCP | `9000-9010` | `10.10.0.0/16` (Mumbai) |

    With more regions, add the same pair of rules for each peer region's CIDR.

    <Tip>
      If nodes resolve each other's IPs but `Node.list()` stays empty, check these rules first — a missing rule shows
      up as a connection timeout, not a DNS failure.
    </Tip>
  </Step>

  <Step title="Create one private zone shared by every regional VPC" icon="globe">
    On AWS, create a Route 53 **private hosted zone** — say `mesh.acmecorp.internal` — and associate it with the VPC of
    every regional environment you spin up. One zone, many VPC associations. If your environments live in different AWS
    accounts, use cross-account zone association. On GCP, the equivalent is a Cloud DNS private zone with each regional
    VPC added to its visibility list.

    The zone has to be associated with a region's VPC *before* nodes there can resolve peers.
  </Step>

  <Step title="Give each environment a name in ops.json" icon="file-pen">
    Alongside `beam_cluster_id`, set `beam_dns_name` to the record this environment should own inside that zone:

    ```json ops.json theme={null}
    {
      "beam_cluster_id": "b2bapp",
      "beam_dns_name": "mumbai.mesh.acmecorp.internal"
    }
    ```

    Keep `beam_cluster_id` identical everywhere — it's still what makes these nodes one cluster. `beam_dns_name` is
    per environment: `mumbai.mesh.acmecorp.internal` in Mumbai, `ireland.mesh.acmecorp.internal` in Ireland, and so on.

    LocalOps creates that record and keeps it in sync — it publishes the IPs of every pod in the environment carrying
    that `beam_cluster_id`, and updates the record as pods scale, restart, or get rescheduled. You don't manage the
    record set yourself.
  </Step>

  <Step title="List every region in your topologies" icon="network-wired">
    Add one topology per region so libcluster polls all of them. `LOPS_BEAM_CLUSTER_DNS` still points at the local
    headless service, so keep using it for this environment's own nodes and name the peer regions by their
    `beam_dns_name`:

    ```elixir config/runtime.exs theme={null}
    config :libcluster,
      topologies: [
        local: [
          strategy: Cluster.Strategy.Kubernetes.DNS,
          config: [
            service: System.get_env("LOPS_BEAM_CLUSTER_DNS"),
            application_name: "b2bapp",
            polling_interval: 5_000
          ]
        ],
        mum: [
          strategy: Cluster.Strategy.Kubernetes.DNS,
          config: [
            service: "mumbai.mesh.acmecorp.internal",
            application_name: "b2bapp",
            polling_interval: 5_000
          ]
        ],
        ire: [
          strategy: Cluster.Strategy.Kubernetes.DNS,
          config: [
            service: "ireland.mesh.acmecorp.internal",
            application_name: "b2bapp",
            polling_interval: 5_000
          ]
        ]
      ]
    ```

    The same config ships to every region — each node finds its local peers through `LOPS_BEAM_CLUSTER_DNS` and every
    remote region through that region's record. Nodes connect from both ends, and the mesh forms.
  </Step>

  <Step title="Verify the mesh from one region" icon="circle-check">
    Once the new build is deployed in every region, use [`ops connect`](/cli/usage#shell-into-a-service) to open a
    shell on a pod in one region's environment:

    ```sh theme={null}
    ops connect --env mumbai --svc web
    ```

    From that shell, attach a remote console to the running release and list the connected nodes:

    ```sh theme={null}
    bin/b2bapp remote
    ```

    ```elixir theme={null}
    Node.list()
    ```

    You should see nodes from every region, not just this one — for example `b2bapp@10.10.x.x` alongside
    `b2bapp@10.20.x.x`. If only local nodes show up, recheck the peering routes and the security group rules above.
  </Step>
</Steps>

<Warning>
  The `RELEASE_COOKIE` has to match across regions, not just across services in one region.
</Warning>

<Note>
  Preview environments ignore `beam_dns_name`. They keep forming their own single-region cluster, for the same reason
  they stay out of your production cluster.
</Note>

## Things to know

### It takes effect on your next build

LocalOps snapshots `ops.json` when it builds your service, so redeploying your current image won't pick up a newly added
`beam_cluster_id`. Push the change and let it build.

### Preview environments get their own cluster

A preview environment forms its own separate cluster and never joins your production cluster. This is deliberate —
otherwise a preview would share your cookie and be able to call into live nodes. See
[ephemeral databases and services for pull request previews](/environment/services/ops-json#ephemeral-databases-and-services-for-pull-request-previews).

### Job and cron services join too

[Job](/environment/services/job) and [cron](/environment/services/cronjob) services join the cluster as well, so
scheduled and one-off tasks can call into the running cluster. If you'd rather they stayed out, just leave
`beam_cluster_id` out of those services' `ops.json`.

<Warning>
  Two caveats when they are in. Each run is a join-then-leave, which is a cluster-wide event — `:global` locks the
  cluster on every `nodeup`, and Horde/Swarm rebalance their registries — so a frequent schedule means constant churn.
  And those pods join as full peers, able to run code on any other node.
</Warning>

### Processes with their own image stay out

A [`processes`](/environment/services/ops-json#configuring-processes) entry that specifies its own `image` stays out of
the cluster, since it isn't running your app.

## Verify it worked

Get a shell on a running pod with [`ops connect`](/cli/usage#shell-into-a-service), open a remote console, and run:

```sh theme={null}
ops connect --env production --svc web
bin/myapp remote
```

```elixir theme={null}
Node.list()
```

It should return your peers. The cluster's name is in `LOPS_BEAM_CLUSTER_NAME`.

To watch inter-node distribution traffic over time, see the
[Erlang & Elixir instrumentation guide](/environment/services/instrument/erlang) — the Erlang-Distribution Grafana
dashboard it links is built for exactly this.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.