{"id":810,"date":"2017-01-26T10:00:17","date_gmt":"2017-01-26T10:00:17","guid":{"rendered":"http:\/\/www.migenius.com\/?p=810"},"modified":"2021-12-05T21:31:16","modified_gmt":"2021-12-05T21:31:16","slug":"realityserver-on-aws","status":"publish","type":"post","link":"https:\/\/www.migenius.com\/articles\/realityserver-on-aws","title":{"rendered":"RealityServer on AWS"},"content":{"rendered":"\n<p>A lot of new customers ask us where they can run RealityServer since they don&#8217;t have their own server or workstation with NVIDIA GPU hardware available. Starting up RealityServer on <a rel=\"noreferrer noopener\" href=\"https:\/\/migenius.jarvice.com\/landing\" target=\"_blank\">Nimbix<\/a> is covered in <a href=\"http:\/\/www.migenius.com\/articles\/realityserver-on-nimbix\">another article<\/a> where everything is pre-configured for you, on AWS however you need to do a bit more setup yourself. We are assuming here that you are already familiar with <a rel=\"noreferrer noopener\" href=\"http:\/\/aws.amazon.com\/\" target=\"_blank\">Amazon Web Services<\/a> and starting instances on Amazon EC2, along with basic concepts like security groups.&nbsp;We won&#8217;t cover the basics of how to start an instance here however there is lots of good information about that online, including this <a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/launching-instance.html\">guide from Amazon<\/a>. So, let&#8217;s get started.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h3>Instance Selection and Starting<\/h3>\n\n\n\n<p>Before you dive in to launch some instances on AWS to run RealityServer, you should first think about where you want to run your server (<a href=\"http:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/using-regions-availability-zones.html\">which region<\/a>) as well as which <a href=\"http:\/\/aws.amazon.com\/ec2\/instance-types\/\">instance type<\/a> you want to use. While RealityServer can work on pure CPU instances to access all features and get the best performance, you should really use <a href=\"http:\/\/aws.amazon.com\/ec2\/instance-types\/#GPU_\">AWS GPU instances<\/a>. There are currently three types of GPU instances available on AWS that support RealityServer well:<\/p>\n\n\n\n<ul><li>g3 &#8211; NVIDIA Tesla M60 (g3.4xlarge, g3.8xlarge, g3.16xlarge)<\/li><li>p3 &#8211; NVIDIA Tesla V100 (p3.2xlarge, p3.8xlarge, p3.16xlarge)<\/li><li>g4 &#8211; NVIDIA Tesla T4 (g4dn.xlarge, g4dn.2xlarge, g4dn.4xlarge, g4dn.8xlarge, g4dn.16xlarge, g4dn.12xlarge, g4dn.metal)<\/li><li>g5 &#8211; NVIDIA A10G (g5.xlarge, g5.2xlarge, g5.4xlarge, g5.8xlarge, g5.12xlarge, g5.16xlarge, g5.24xlarge, g5.48xlarge)<\/li><\/ul>\n\n\n\n<p>All of the current generation instance types use Tesla GPU hardware and offer various level of performance and memory capacity. The g5 instances use the Tesla A10G GPU and at the time of writing was the fastest available. The g4 instances while not the fastest offer the good value for money. The g5 have the best price performance ratio currently of all of the instance types. Of course Amazon offerings might change in the future. You can see our <a href=\"https:\/\/www.migenius.com\/benchmarks\">benchmark data<\/a> on how these instance types perform. When selecting which region to start your node in consider how far it is likely to be from you; closer servers mean lower latency. We do not recommend previous generation instance types for RealityServer and they may not be supported in some cases.<\/p>\n\n\n\n<p>When you start your instances, ensure you open TCP ports <em>8080<\/em>, <em>8081<\/em> and <em>1935<\/em> in your security group, in addition to the standard SSH port. We generally use the standard Amazon Linux 2 HVM AMI (usually first in the list which launching from the console); for GPU instances you must use HVM AMIs. These instructions assume you are using the standard Amazon Linux 2 AMI; other distributions can also be used with RealityServer, but you will need to sort out the installation process yourself. In general since Amazon Linux is Red Hat Enterprise based, CentOS and RHEL should behave in a similar way.<\/p>\n\n\n\n<h3>Dependencies and Configuration<\/h3>\n\n\n\n<p>Once you have launched your instance and are ready to connect you can <em>ssh<\/em> into your instance and start setting up the GPU drivers and RealityServer. First let&#8217;s install some dependencies we will need:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo yum update -y\nsudo yum install -y gcc make wget libX11 libGLU libSM\nsudo yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)\n<\/pre><\/div>\n\n\n<p>If the <em>yum update<\/em> command installs a new kernel version you should reboot prior to installing the kernel headers and development package. Next let&#8217;s get the NVIDIA drivers installed. You can check for the latest Linux version <a rel=\"noreferrer noopener\" href=\"http:\/\/www.nvidia.com\/object\/unix.html\" target=\"_blank\">here<\/a>. Here we used the versions that worked well at the time this was written (on the g4 instance types), substitute as needed:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nwget http:\/\/us.download.nvidia.com\/XFree86\/Linux-x86_64\/470.82.01\/NVIDIA-Linux-x86_64-470.82.01.run\nsudo sh .\/NVIDIA-Linux-x86_64-470.82.01.run -a\n<\/pre><\/div>\n\n\n<p>Follow all of the prompts, it&#8217;s pretty self-explanatory. When running RealityServer you may get warnings about ECC being enabled, unfortunately AWS do not offer the ability to disable this GPU feature, so you may get a slight performance hit and you will lose 12.5% of your GPU memory capacity. This is just due to the way AWS works and sadly not something we can fix with RealityServer. Future releases of RealityServer may require more recent CUDA versions and therefore ensure the driver you install supported the minimum level of CUDA needed.<\/p>\n\n\n\n<p><strong>NOTE:<\/strong> Amazon Linux 2 users with kernel version 5.1.0, you will need to use a slightly different command to install the NVIDIA drivers in order to pick up the correct compiler for the newer kernel version.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nsudo CC=\/usr\/bin\/gcc10-cc sh .\/NVIDIA-Linux-x86_64-470.82.01.run -a\n<\/pre><\/div>\n\n\n<h3>Installing RealityServer<\/h3>\n\n\n\n<p>With the GPU drivers installed (if you want to confirm they detect the GPU just run <em>nvidia-smi<\/em> and check out the output) you can now download and install RealityServer:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo mkdir -p \/usr\/local\/migenius\ncd \/usr\/local\/migenius\nsudo wget http:\/\/download.migenius.com\/releases\/RealityServer\/realityserver-62-3938.103.tgz\nsudo tar zxvf realityserver-62-3938.103.tgz\nsudo chown -R ec2-user:ec2-user realityserver-62-3938.103.tgz\nsudo ln -s realityserver-62-3938.103.tgz realityserver\n<\/pre><\/div>\n\n\n<p>Obviously replace the RealityServer link with that to the release you want to use. The last line above creates a symlink to the version you have installed, we will use this later when setting up services so you can easily switch versions later if you need to.<\/p>\n\n\n\n<h3>Installing Your License<\/h3>\n\n\n\n<p>To use RealityServer you will need a valid license. Node-locked licenses do not work on AWS since the hardware you will use changes each time you start a new node. If you have your own license server you can point your AWS instance at the license server however more likely you will be using our Cloud license service. Please refer to the Cloud section of our <a href=\"https:\/\/www.migenius.com\/realityserver-licensing\" target=\"_blank\" rel=\"noreferrer noopener\">RealityServer Licensing Instructions<\/a> for details on how to setup your Cloud license. Basically you will just copy the <em>realityserver.lic<\/em> file provided by migenius to your RealityServer directory above.<\/p>\n\n\n\n<h3>Test Run RealityServer<\/h3>\n\n\n\n<p>If all goes well you can now do a test run and get RealityServer going. We recommend doing this before trying to install RealityServer as a service since it will be more difficult to diagnose issues at that stage. Here are the commands to test run RealityServer, assuming you are still in the directory from above:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\ncd realityserver\n.\/realityserver\n<\/pre><\/div>\n\n\n<p>You should see a lot of log output. You can then connect to RealityServer and explore the examples and documentation by going to&nbsp;the following address (where <em>ec2-ip<\/em> is the public IP of your AWS instance):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nhttp:\/\/ec2-ip:8080\/\n<\/pre><\/div>\n\n\n<p>If you want to start RealityServer so that it keeps running after you log out you can use <em>nohup<\/em> for that:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nnohup .\/realityserver &gt; rs.log 2&gt;&1 &amp;\n<\/pre><\/div>\n\n\n<p>Here we are sending the log output to <em>rs.log<\/em> so you won&#8217;t see anything on screen. To see the log output you can just tail the file:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\ntail -f rs.log\n<\/pre><\/div>\n\n\n<p>When you&#8217;re done with your instance and don&#8217;t want to leave it running, you can just power it off:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nsudo poweroff\n<\/pre><\/div>\n\n\n<p>On AWS this puts the instance into a stopped state and you can restart it at anytime. You don&#8217;t get charged for the instance while it&#8217;s stopped (only for the storage associated with it), so this can be a handy way to keep a RealityServer development node ready to run. Now, that things are working correctly we can install RealityServer as a system service so that it runs on startup and restarts if it fails for any reason.<\/p>\n\n\n\n<h3>Installing as a System Service<\/h3>\n\n\n\n<p>The raging debate over init.d vs systemd is now pretty much in the past so we can now recommend using <a href=\"https:\/\/en.wikipedia.org\/wiki\/Systemd\">systemd<\/a> to manage your RealityServer service. This isn&#8217;t a complete guide to systemd but we&#8217;ll run through the basics of how to use systemd to control RealityServer. First you will need to create the service file in <em>\/etc\/systemd\/system\/realityserver.service<\/em>. We can do this in one command with:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo tee -a \/etc\/systemd\/system\/realityserver.service &gt; \/dev\/null &lt;&lt; EOL\n&#91;Unit]\nDescription=RealityServer\nAfter=network.target remote-fs.target nss-lookup.target\n \n&#91;Service]\nExecStart=\/usr\/local\/migenius\/realityserver\/linux-x86-64\/bin\/realityserver --config_file \/usr\/local\/migenius\/realityserver\/realityserver.conf $RS_ARGUMENTS\n# Required on some systems\nWorkingDirectory=\/usr\/local\/migenius\/realityserver\nRestart=always\n# Restart service after 5 seconds RealityServer crashes\nRestartSec=5\n# Output to syslog\nStandardOutput=syslog\nStandardError=syslog\nSyslogIdentifier=realityserver\nUser=ec2-user\nGroup=ec2-user\nEnvironment=LD_LIBRARY_PATH=\/usr\/local\/migenius\/realityserver\/linux-x86-64\/lib\n \n&#91;Install]\nWantedBy=multi-user.target\nEOL\n<\/pre><\/div>\n\n\n<p>This is a single multi-line command you can cut and paste the entire command at once. Next we need to make the configuration directory for our service:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo mkdir \/etc\/systemd\/system\/realityserver.service.d\n<\/pre><\/div>\n\n\n<p>This directory holds the <em>override.conf<\/em> file which allows us to setup environment variables that are used when starting RealityServer. We can add this with another multi-line command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo tee -a \/etc\/systemd\/system\/realityserver.service.d\/override.conf &gt; \/dev\/null &lt;&lt; EOL\n&#91;Service]\nEnvironment=&quot;RS_ARGUMENTS='--network off' '-o iray_render_mode=cuda_dynamic'&quot;\nEnvironment=MDL_SYSTEM_PATH=\/opt\/nvidia\/mdl\nEOL\n<\/pre><\/div>\n\n\n<p>The last line of the configuration adds the MDL system path environment variable which you will need if you have <a href=\"https:\/\/developer.nvidia.com\/vmaterials\">vMaterials<\/a> or the <a href=\"https:\/\/www.nvidia.com\/en-us\/design-visualization\/technologies\/material-definition-language\/\">MDL Material Exchange<\/a> libraries installed. If you don&#8217;t use either of these you can omit this line. You&#8217;ll also want to uncomment the lines in your <em>realityserver.conf<\/em> file that reference the <em>MDL_SYSTEM_PATH<\/em> environment variable.<\/p>\n\n\n\n<p>At this point there is a service setup and you should be able to check this with the following command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsystemctl status realityserver\n<\/pre><\/div>\n\n\n<p>You should see something like this:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n\u25cf realityserver.service - RealityServer\n   Loaded: loaded (\/etc\/systemd\/system\/realityserver.service; disabled; vendor preset: disabled)\n  Drop-In: \/etc\/systemd\/system\/realityserver.service.d\n           \u2514\u2500override.conf\n   Active: inactive (dead)\n<\/pre><\/div>\n\n\n<p>So it&#8217;s there and everything has been found but it hasn&#8217;t yet been enabled and it isn&#8217;t running. Before we fix that we&#8217;ll setup one more thing, logging. The service is currently setup to log all of it&#8217;s output to <em>syslog<\/em> which is going to make it difficult to separate RealityServer messages from other system messages. To fix that we need to do a little <em>rsyslogd<\/em> configuration. We&#8217;ll use the following multi-line command to set this up:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo tee -a \/etc\/rsyslog.d\/realityserver.conf &gt; \/dev\/null &lt;&lt; EOL\n\\$FileCreateMode 0644\n\\$template RealityServerFormat,&quot;%rawmsg%\\n&quot;\n\\$template RealityServerFile,&quot;\/var\/log\/realityserver.log&quot;\nif \\$programname == 'realityserver' then {\n        -?RealityServerFile;RealityServerFormat\n        ~\n}\n\\$FileCreateMode 0600\nEOL\n<\/pre><\/div>\n\n\n<p>This will change the configuration so that all output is now logged to <em>\/var\/log\/realityserver.log<\/em>. If you want to allow reading of the log files as the <em>ec2-user<\/em> then you will also need to modify <em>\/etc\/rsyslog.conf<\/em> by adding the following lines at the top:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n# Reset umask so FileCreateMode can be used instead\n$umask 0000\n \n# By default logs should only be readable by process owner\n$FileCreateMode 0600\n<\/pre><\/div>\n\n\n<p>We now need to force rsyslogd to reload our configuration before we start RealityServer:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nsudo systemctl restart rsyslog\n<\/pre><\/div>\n\n\n<p>Finally we are ready to enable the service so that it starts up on system boot.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo systemctl enable realityserver\n<\/pre><\/div>\n\n\n<p>Which should output something similar to this:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nCreated symlink from \/etc\/systemd\/system\/multi-user.target.wants\/realityserver.service to \/etc\/systemd\/system\/realityserver.service.\n<\/pre><\/div>\n\n\n<p>Since we haven&#8217;t rebooted the service is not yet running so we can now start it up with:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo systemctl start realityserver\n<\/pre><\/div>\n\n\n<p>If you tail the contents of <em>\/var\/log\/realityserver.log<\/em> you should see the last couple of lines look something like this:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n19\/09\/06 09:35:35   1.0   V8     main info : Started.\n19\/09\/06 09:35:35   1.0   PLUGIN main info : Started RealityServer(R) Web Services.\n<\/pre><\/div>\n\n\n<p>Assuming you can connect to RealityServer in your browser as you did in the earlier steps then everything should be working now. You can test that the service runs on startup by rebooting the machine. Of course, don&#8217;t forget to shutdown the instance if you are not using it. You now have an instance which will automatically run RealityServer on startup using the standard systemd process. If you create an AMI of this instance you can then start more instances with the same configuration.<\/p>\n\n\n\n<h3>NVIDIA Persistence Daemon<\/h3>\n\n\n\n<p>We&#8217;ve seen some issues recently were during startup the NVIDIA driver was not initialised and loaded. This could cause issues with RealityServer startup. To ensure the driver is always loaded we recommend installing and enabling the <a href=\"https:\/\/docs.nvidia.com\/deploy\/driver-persistence\/index.html#persistence-daemon\">NVIDIA Persistence Daemon<\/a>. This is installed by the driver however not as a system service. You can add it as a system service in a similar way to RealityServer with the following commands.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nsudo tee -a \/etc\/systemd\/system\/nvidia-persistenced.service &gt; \/dev\/null &lt;&lt; EOL\n&#91;Unit]\nDescription=NVIDIA Persistence Daemon\nWants=syslog.target\n\n&#91;Service]\nType=forking\nExecStart=\/usr\/bin\/nvidia-persistenced --user ec2-user\nExecStopPost=\/bin\/rm -rf \/var\/run\/nvidia-persistenced\n\n&#91;Install]\nWantedBy=multi-user.target\nEOL\nsudo systemctl enable nvidia-persistenced\nsudo systemctl start nvidia-persistenced\n<\/pre><\/div>\n\n\n<h3>Taking it Further<\/h3>\n\n\n\n<p>There is a lot we haven&#8217;t covered here and, in a production system, load balancing, clustering and many more topics. We don&#8217;t cover this here however don&#8217;t hesitate to <a href=\"http:\/\/www.migenius.com\/contact\">contact us<\/a> if you want more details on how to do these things. We also haven&#8217;t covered creating an AMI from your work here so you can start more nodes. This is essentially no different to&nbsp;any other Amazon instance type, so there is a lot of good information out there on that.<\/p>\n\n\n\n<p>One note, if you want to try clustering multiple nodes on AWS, you will need to use TCP based clustering (without the UDP discovery option) because AWS does not support UDP multicast on its network.<\/p>\n\n\n\n<p>Happy Rendering!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A lot of new customers ask us where they can run RealityServer since they don&#8217;t have their own server or workstation with NVIDIA GPU hardware available. Starting up RealityServer on Nimbix is covered in another article where everything is pre-configured for you, on AWS however you need to do a bit more setup yourself. We [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":824,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[15],"tags":[19,14,27],"jetpack_featured_media_url":"https:\/\/www.migenius.com\/migenius\/wp-content\/uploads\/2015\/01\/aws1.jpg","_links":{"self":[{"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/posts\/810"}],"collection":[{"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/comments?post=810"}],"version-history":[{"count":47,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/posts\/810\/revisions"}],"predecessor-version":[{"id":3923,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/posts\/810\/revisions\/3923"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/media\/824"}],"wp:attachment":[{"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/media?parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/categories?post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.migenius.com\/wp-json\/wp\/v2\/tags?post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}