A lot of new customers ask us where they can run RealityServer since they don’t have their own server or workstation with NVIDIA GPU hardware available. Starting up RealityServer on Nimbix is covered in another article where everything is pre-configured for you, on AWS however you need to do a bit more setup yourself. We are assuming here that you are already familiar with Amazon Web Services and starting instances on Amazon EC2, along with basic concepts like security groups. We won’t cover the basics of how to start an instance here however there is lots of good information about that online, including this guide from Amazon. So, let’s get started.
Before you dive in to launch some instances on AWS to run RealityServer, you should first think about where you want to run your server (which region) as well as which instance type you want to use. While RealityServer can work on pure CPU instances to access all features and get the best performance, you should really use AWS GPU instances. There are currently three types of GPU instances available on AWS that support RealityServer well:
All of the current generation instance types use Tesla GPU hardware and offer various level of performance and memory capacity. The g5 instances use the Tesla A10G GPU and at the time of writing was the fastest available. The g4 instances while not the fastest offer the good value for money. The g5 have the best price performance ratio currently of all of the instance types. Of course Amazon offerings might change in the future. You can see our benchmark data on how these instance types perform. When selecting which region to start your node in consider how far it is likely to be from you; closer servers mean lower latency. We do not recommend previous generation instance types for RealityServer and they may not be supported in some cases.
When you start your instances, ensure you open TCP ports 8080, 8081 and 1935 in your security group, in addition to the standard SSH port. We generally use the standard Amazon Linux 2 HVM AMI (usually first in the list which launching from the console); for GPU instances you must use HVM AMIs. These instructions assume you are using the standard Amazon Linux 2 AMI; other distributions can also be used with RealityServer, but you will need to sort out the installation process yourself. In general since Amazon Linux is Red Hat Enterprise based, CentOS and RHEL should behave in a similar way.
Once you have launched your instance and are ready to connect you can ssh into your instance and start setting up the GPU drivers and RealityServer. First let’s install some dependencies we will need:
sudo yum update -y
sudo yum install -y gcc make wget libX11 libGLU libSM
sudo yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
If the yum update command installs a new kernel version you should reboot prior to installing the kernel headers and development package. Next let’s get the NVIDIA drivers installed. You can check for the latest Linux version here. Here we used the versions that worked well at the time this was written (on the g4 instance types), substitute as needed:
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/470.82.01/NVIDIA-Linux-x86_64-470.82.01.run
sudo sh ./NVIDIA-Linux-x86_64-470.82.01.run -a
Follow all of the prompts, it’s pretty self-explanatory. When running RealityServer you may get warnings about ECC being enabled, unfortunately AWS do not offer the ability to disable this GPU feature, so you may get a slight performance hit and you will lose 12.5% of your GPU memory capacity. This is just due to the way AWS works and sadly not something we can fix with RealityServer. Future releases of RealityServer may require more recent CUDA versions and therefore ensure the driver you install supported the minimum level of CUDA needed.
NOTE: Amazon Linux 2 users with kernel version 5.1.0, you will need to use a slightly different command to install the NVIDIA drivers in order to pick up the correct compiler for the newer kernel version.
sudo CC=/usr/bin/gcc10-cc sh ./NVIDIA-Linux-x86_64-470.82.01.run -a
With the GPU drivers installed (if you want to confirm they detect the GPU just run nvidia-smi and check out the output) you can now download and install RealityServer:
sudo mkdir -p /usr/local/migenius
cd /usr/local/migenius
sudo wget http://download.migenius.com/releases/RealityServer/realityserver-62-3938.103.tgz
sudo tar zxvf realityserver-62-3938.103.tgz
sudo chown -R ec2-user:ec2-user realityserver-62-3938.103.tgz
sudo ln -s realityserver-62-3938.103.tgz realityserver
Obviously replace the RealityServer link with that to the release you want to use. The last line above creates a symlink to the version you have installed, we will use this later when setting up services so you can easily switch versions later if you need to.
To use RealityServer you will need a valid license. Node-locked licenses do not work on AWS since the hardware you will use changes each time you start a new node. If you have your own license server you can point your AWS instance at the license server however more likely you will be using our Cloud license service. Please refer to the Cloud section of our RealityServer Licensing Instructions for details on how to setup your Cloud license. Basically you will just copy the realityserver.lic file provided by migenius to your RealityServer directory above.
If all goes well you can now do a test run and get RealityServer going. We recommend doing this before trying to install RealityServer as a service since it will be more difficult to diagnose issues at that stage. Here are the commands to test run RealityServer, assuming you are still in the directory from above:
cd realityserver
./realityserver
You should see a lot of log output. You can then connect to RealityServer and explore the examples and documentation by going to the following address (where ec2-ip is the public IP of your AWS instance):
http://ec2-ip:8080/
If you want to start RealityServer so that it keeps running after you log out you can use nohup for that:
nohup ./realityserver > rs.log 2>&1 &
Here we are sending the log output to rs.log so you won’t see anything on screen. To see the log output you can just tail the file:
tail -f rs.log
When you’re done with your instance and don’t want to leave it running, you can just power it off:
sudo poweroff
On AWS this puts the instance into a stopped state and you can restart it at anytime. You don’t get charged for the instance while it’s stopped (only for the storage associated with it), so this can be a handy way to keep a RealityServer development node ready to run. Now, that things are working correctly we can install RealityServer as a system service so that it runs on startup and restarts if it fails for any reason.
The raging debate over init.d vs systemd is now pretty much in the past so we can now recommend using systemd to manage your RealityServer service. This isn’t a complete guide to systemd but we’ll run through the basics of how to use systemd to control RealityServer. First you will need to create the service file in /etc/systemd/system/realityserver.service. We can do this in one command with:
sudo tee -a /etc/systemd/system/realityserver.service > /dev/null << EOL
[Unit]
Description=RealityServer
After=network.target remote-fs.target nss-lookup.target
[Service]
ExecStart=/usr/local/migenius/realityserver/linux-x86-64/bin/realityserver --config_file /usr/local/migenius/realityserver/realityserver.conf $RS_ARGUMENTS
# Required on some systems
WorkingDirectory=/usr/local/migenius/realityserver
Restart=always
# Restart service after 5 seconds RealityServer crashes
RestartSec=5
# Output to syslog
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=realityserver
User=ec2-user
Group=ec2-user
Environment=LD_LIBRARY_PATH=/usr/local/migenius/realityserver/linux-x86-64/lib
[Install]
WantedBy=multi-user.target
EOL
This is a single multi-line command you can cut and paste the entire command at once. Next we need to make the configuration directory for our service:
sudo mkdir /etc/systemd/system/realityserver.service.d
This directory holds the override.conf file which allows us to setup environment variables that are used when starting RealityServer. We can add this with another multi-line command:
sudo tee -a /etc/systemd/system/realityserver.service.d/override.conf > /dev/null << EOL
[Service]
Environment="RS_ARGUMENTS='--network off' '-o iray_render_mode=cuda_dynamic'"
Environment=MDL_SYSTEM_PATH=/opt/nvidia/mdl
EOL
The last line of the configuration adds the MDL system path environment variable which you will need if you have vMaterials or the MDL Material Exchange libraries installed. If you don’t use either of these you can omit this line. You’ll also want to uncomment the lines in your realityserver.conf file that reference the MDL_SYSTEM_PATH environment variable.
At this point there is a service setup and you should be able to check this with the following command:
systemctl status realityserver
You should see something like this:
● realityserver.service - RealityServer
Loaded: loaded (/etc/systemd/system/realityserver.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/realityserver.service.d
└─override.conf
Active: inactive (dead)
So it’s there and everything has been found but it hasn’t yet been enabled and it isn’t running. Before we fix that we’ll setup one more thing, logging. The service is currently setup to log all of it’s output to syslog which is going to make it difficult to separate RealityServer messages from other system messages. To fix that we need to do a little rsyslogd configuration. We’ll use the following multi-line command to set this up:
sudo tee -a /etc/rsyslog.d/realityserver.conf > /dev/null << EOL
\$FileCreateMode 0644
\$template RealityServerFormat,"%rawmsg%\n"
\$template RealityServerFile,"/var/log/realityserver.log"
if \$programname == 'realityserver' then {
-?RealityServerFile;RealityServerFormat
~
}
\$FileCreateMode 0600
EOL
This will change the configuration so that all output is now logged to /var/log/realityserver.log. If you want to allow reading of the log files as the ec2-user then you will also need to modify /etc/rsyslog.conf by adding the following lines at the top:
# Reset umask so FileCreateMode can be used instead
$umask 0000
# By default logs should only be readable by process owner
$FileCreateMode 0600
We now need to force rsyslogd to reload our configuration before we start RealityServer:
sudo systemctl restart rsyslog
Finally we are ready to enable the service so that it starts up on system boot.
sudo systemctl enable realityserver
Which should output something similar to this:
Created symlink from /etc/systemd/system/multi-user.target.wants/realityserver.service to /etc/systemd/system/realityserver.service.
Since we haven’t rebooted the service is not yet running so we can now start it up with:
sudo systemctl start realityserver
If you tail the contents of /var/log/realityserver.log you should see the last couple of lines look something like this:
19/09/06 09:35:35 1.0 V8 main info : Started.
19/09/06 09:35:35 1.0 PLUGIN main info : Started RealityServer(R) Web Services.
Assuming you can connect to RealityServer in your browser as you did in the earlier steps then everything should be working now. You can test that the service runs on startup by rebooting the machine. Of course, don’t forget to shutdown the instance if you are not using it. You now have an instance which will automatically run RealityServer on startup using the standard systemd process. If you create an AMI of this instance you can then start more instances with the same configuration.
We’ve seen some issues recently were during startup the NVIDIA driver was not initialised and loaded. This could cause issues with RealityServer startup. To ensure the driver is always loaded we recommend installing and enabling the NVIDIA Persistence Daemon. This is installed by the driver however not as a system service. You can add it as a system service in a similar way to RealityServer with the following commands.
sudo tee -a /etc/systemd/system/nvidia-persistenced.service > /dev/null << EOL
[Unit]
Description=NVIDIA Persistence Daemon
Wants=syslog.target
[Service]
Type=forking
ExecStart=/usr/bin/nvidia-persistenced --user ec2-user
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced
[Install]
WantedBy=multi-user.target
EOL
sudo systemctl enable nvidia-persistenced
sudo systemctl start nvidia-persistenced
There is a lot we haven’t covered here and, in a production system, load balancing, clustering and many more topics. We don’t cover this here however don’t hesitate to contact us if you want more details on how to do these things. We also haven’t covered creating an AMI from your work here so you can start more nodes. This is essentially no different to any other Amazon instance type, so there is a lot of good information out there on that.
One note, if you want to try clustering multiple nodes on AWS, you will need to use TCP based clustering (without the UDP discovery option) because AWS does not support UDP multicast on its network.
Happy Rendering!
Paul Arden has worked in the Computer Graphics industry for over 20 years, co-founding the architectural visualisation practice Luminova out of university before moving to mental images and NVIDIA to manage the Cloud-based rendering solution, RealityServer, now managed by migenius where Paul serves as CEO.