Deployment
To deploy an Infernet Node in production, we have put together a best-practices repository: ritual-net/infernet-deploy (opens in a new tab).
Start by cloning the infernet-deploy
repository locally:
git clone https://github.com/ritual-net/infernet-deploy
Deploying a single node locally
Infernet Nodes execute containerized workflows. As such, installing and running (opens in a new tab) a modern version of Docker is a prerequisite, regardless of your choice of how to run the node.
Running locally
You can run an Infernet Node locally via Docker Compose (opens in a new tab). First, you must create a configuration file (see: example configuration (opens in a new tab)).
# Create a configuration file
cd deploy
cp ../config.sample.json config.json
For filling in config.json
properly, see Configuration. After you have configured your node, you can run it with:
# Run node
cd deploy
docker compose up -d
You can stop the node and related services with:
# Stop node
cd deploy
docker compose down
For your local node's REST API to be accessible from the public internet, you
will have to expose node's local port (default: 4000
) to the internet. This
it NOT recommended. If planning to support off-chain workloads through the
REST API, we highly recommend deploying a node cluster on AWS or GCP instead,
as detailed in Deploying a node cluster.
Deploying a node cluster
Via infernet-deploy
, node operators can use Terraform (opens in a new tab) for infrastructure procurement and Docker Compose (opens in a new tab) for deployment to deploy Infernet Nodes in production on Amazon Web Services (AWS) (opens in a new tab) and Google Cloud Platform (GCP) (opens in a new tab).
To deploy a cluster, operators need to specify:
- Cluster-level configurations via
terraform.tfvars
. - Node-level configurations via a
config.json
, for each node being deployed.
General setup
Install Terraform
Follow the official instructions (opens in a new tab) to install Terraform. We use Terraform in the infernet-deploy
scripts for infrastructure procurement.
Configure nodes
Once ready, you must create a configuration file (see: example configuration (opens in a new tab)) for each node being deployed.
- These configuration files must have unique names
- A straightforward approach would be
0.json
,1.json
, etc.
- A straightforward approach would be
- These files must be placed under the top-level
configs/
directory - The number and name of the
.json
files must match the number and name of keys in thenodes
variable interraform.tfvars
- See terraform.tfvars.example (opens in a new tab)
- Each key should correspond to the name of a
.json
file, excluding the.json
postfix
- Each node strictly requires its own configuration
.json
file, even if identical
For more details around node configuration options, reference the configuration documentation.
Infernet Router
Along with your node cluster, you can opt to deploy an infernet-router (opens in a new tab), which is a lightweight REST server that helps route requests to Infernet nodes. The Infernet Router does not route requests directly. Instead, it returns an Infernet node IP to send requests to. As such, it can be used in a 2-step process, where the client:
- Requests an IP from the Infernet Router.
- Sends Infernet API request(s) directly to that IP.
The Infernet Router REST server is deployed automatically with Terraform, by configuring the router
object and setting deploy = true
in your .tfvars
file. However, if you plan to use it, you need to understand its implications.
IMPORTANT: When configuring a heterogeneous node cluster (i.e. 0.json, 1.json, etc. are not identical), container IDs should be reserved for a unique container setup at the cluster level, i.e. across nodes (and thus .json files). That is becuase the router uses container IDs to make routing decisions between services running across the cluster.
Example: Consider nodes A and B, each running a single LLM inference container; node A runs
image1
, and node B runsimage2
. If we setid: "llm-inference"
in both nodes (containers[0].id
attribute in0.json
,1.json
), the router will be unable to disambiguate between the two services, and will consider them interchangeable, which they are not. Any requests for"llm-inference"
will be routed to either container, which is an error.
Therefore, re-using IDs across configuration files must imply an identical container configuration, including image, environment variables, command, etc. This will explicitly tell the router which containers are interchangeable, and allow it to distribute requests for those containers across all nodes running that container.
Deploying on AWS
Authenticate with AWS CLI
Follow the official AWS instructions (opens in a new tab) to authenticate with the CLI locally.
Setup terraform.tfvars
First, make a copy of the example configuration file terraform.tfvars.example (opens in a new tab):
cd procure/aws
cp terraform.tfvars.example terraform.tfvars
Then, configure your terraform.tfvars
file. See variables.tf (opens in a new tab) for configuration descriptions.
Run Terraform
When ready to deploy, execute the Terraform scripts:
# Initialize
cd procure
make init provider=aws
# Print deployment plan
make plan provider=aws
# Deploy
make apply provider=aws
# WARNING: Destructive
# Destroy deployment
make destroy provider=aws
Deploying on GCP
Authenticate with gcloud CLI
Follow the official GCP instructions (opens in a new tab) to authenticate with the CLI locally.
Setup terraform.tfvars
First, make a copy of the example configuration file terraform.tfvars.example (opens in a new tab):
cd procure/gcp
cp terraform.tfvars.example terraform.tfvars
Then, configure your terraform.tfvars
file. See variables.tf (opens in a new tab) for configuration descriptions.
Run Terraform
When ready to deploy, execute the Terraform scripts:
# Initialize
cd procure
make init provider=gcp
# Print deployment plan
make plan provider=gcp
# Deploy
make apply provider=gcp
# WARNING: Destructive
# Destroy deployment
make destroy provider=gcp