RealityServer is a platform for integrating photorealistic 3D rendering into your application. It is based on a web services methodology and can be used both for rendering automation and fully interactive rendering. In this introduction we will cover the core concepts of RealityServer development and what you need to understand in order to get started.
Development with RealityServer is easy! However it helps enormously to be familiar with the overall architecture before diving in. Being a very extensive and generalised platform, it can be a little difficult understanding where to get started. We therefore strongly recommend anyone who is new to RealityServer thoroughly review the material presented here.
Before getting started, please follow our RealityServer Installation Guide for instructions on getting your server up and running. The installation guide also covers the directory structure of RealityServer and where to find things. There are several parts below where we will refer to resources available on the server for you to access, so it’s good to have one running.
RealityServer is a platform for software developers and is not intended to be an end-user product. If you are not comfortable with the command line, a text editor and coding then RealityServer is likely not well suited to your needs without assistance from an experienced developer. However migenius does offer software development services if you want us to build your application for you or help you with integration.
We’ll focus on web technologies here, however RealityServer is not limited to use with web applications. Anything that can talk to a web server can work with RealityServer, so virtually every modern programming language and tool. Here though, we’ll cover the most commonly used techniques for working with RealityServer.
What will not be covered here are the details of what RealityServer is from the product perspective. The assumption is you have already determined you are interested in RealityServer and want to get to work developing something. If you want to know more about whether RealityServer is the right product for you, please get in touch.
The following three key concepts are critical to understanding the architecture of RealityServer. We describe them here without reference to the specific technologies used, just the fundamental ideas. It is important that these concepts make sense to you before proceeding.
These are just like functions in your favourite programming language, they have names and parameters, except they run remotely on the server. Executing a command is what causes RealityServer to actually do something.
The client tells RealityServer which commands to run. Anything that can communicate with RealityServer can be a client. The client will typically also do something with the results sent back by the server after it executes the requested command (for example display an image).
This is where the commands are defined and actually execute. RealityServer hosts multiple programming APIs with which you can program your own commands (and it comes with many built in commands).
Now let’s talk about the specific technologies. RealityServer uses an RPC (Remote Procedure Call) client–server model. While RESTful tends to be a more popular model for client–server architectures today, its stateless nature makes it a poor choice for RealityServer. State data for 3D applications can easily reach into the gigabytes so would be expensive to recreate for each request.
To make RPC requests to RealityServer you use the JSON-RPC 2.0 protocol over HTTP. This is very lightweight and easy to understand. It is also possible to send commands over a WebSocket connection which has many advantages (e.g., lower latency and server-push). Either way you are performing the same basic operation, asking RealityServer to perform a task (run a command or commands) and give you the answer.
Spin up your RealityServer and let’s call a command. You can make a JSON-RPC request using anything that can send HTTP POST requests. For example using cURL on the command line (Postman is another great tool for making calls like this). Here is a really basic one using cURL.
curl -X POST -H "Content-Type: application/json" -d '[
{"jsonrpc": "2.0", "method": "echo", "params": {
"input" : "Hello world!"
}, "id": 1}
]' "http://127.0.0.1:8080/"
This is calling a very simple built in command called echo which takes a single parameter input. In the JSON-RPC structure above, you might notice the command is in an array (even though it’s just one command). You can put multiple commands in the array and they will execute as a batch in order. If you run the above you should get back something like this from the server.
[{"id":1,"jsonrpc":"2.0","result":"Hello world!"}]
Not very glamorous but you’ve just exercised all of the fundamental pieces of the RealityServer architecture, command (echo), client (cURL) and server (RealityServer). Everything else you will do builds on this basic process. While you may not ever construct commands manually like this, it is important to understand that they are there and that higher level abstractions are doing this under the hood.
We have another article on Exploring the JSON-RPC API which is worth a read and goes into some additional detail on calling this API from other languages which might be helpful if you cannot make use of our client libraries.
Calling the JSON-RPC API directly is great for testing simple sequences of commands and working through issues, however you will generally want to use something higher level to send your commands to RealityServer. For this purpose we provide the RealityServer Client.
The RealityServer client can be used both directly from a supported browser (all modern browsers should work) or alternatively from Node.js or both. The link above takes you directly to the documentation for the RealityServer Client which contains a detailed step-by-step tutorial for both the browser and Node.js cases where you build up a very simple application.
We strongly recommend using this new client library and ignoring the examples and deprecated JavaScript Client Library that are provided with RealityServer itself (they will be removed at some point). The step-by-step tutorials in the RealityServer Client will get you going quickly.
The purpose of the client library is to facilitate sending commands to RealityServer and managing the communication. Instead of using JSON-RPC over HTTP, the client library utilises a WebSocket based connection and custom protocol to improve performance. We’ve written an article going into more detail on the client library here.
While it would be possible to just use the built in commands and call them through the client library, this makes for a lot of back and forth between the client and server and isn’t particularly programmer friendly. To help make life easier RealityServer provides two methods for adding your own custom commands to the system.
Both of these allow you to create custom commands which can then be called exactly like built in commands. They even show up in the server-generated command documentation.
The V8 JavaScript API is built around the V8 JavaScript Engine, just like the one used in the Chrome browser. It allows you to use modern JavaScript code to program your custom RealityServer commands. A large number of helper classes are provided to make it easier to interact with RealityServer without having to explicitly call commands. We have a series of articles on using the V8 JavaScript API which is a great place to start learning.
For the vast majority of cases, implementing the application logic that needs to manipulate the scene data in RealityServer is best done from custom V8 commands. Your client will then only need to call a handful of simple commands.
The neuray Services API goes much further for C++ developers who want to access core functionality of our embedded Iray rendering engine or who need the absolute highest performance for a particular command. Just like V8 you can create custom commands however unlike V8 you can then directly access the underlying Iray API, giving full access to the rendering engine at a very low level. Our customers rarely need to take advantage of this but the ability is there if needed.
For a photorealistic rendering platform, we haven’t talked much about rendering yet. That is because the aspect most users have trouble getting their head around is the general architecture.
RealityServer includes the NVIDIA Iray rendering engine. Iray is a GPU accelerated renderer which includes the fully accurate Iray Photoreal mode and a more configurable Iray Interactive mode for cases where speed is more critical than accuracy. All features of Iray are available in RealityServer through the various APIs.
One aspect that can be a little tricky when starting out with RealityServer is the fact that you really need to understand both RealityServer and Iray. RealityServer provides a way to access the functionality of Iray but to use it you need to understand various Iray concepts, most importantly the concepts of the database, elements and attributes. This is described in the first section of our article Scopes in RealityServer.
The Scopes in RealityServer article is also important to read since unlike any other rendering product, RealityServer is inherently multi-user in nature. To allow for efficient data storage while also permitting users to be isolated from one another, RealityServer supports the concept of scope. Because this is so important and not usually needed in other environments we dedicated an entire article to it.
RealityServer lets you create all of the types of scene elements that Iray understands and the attributes on those elements which causes Iray to behave in the way you want. To understand how Iray works in more detail and what the available options are refer to the Iray Programmers Manual.
The Iray Programmers Manual is a detailed set of documentation describing all of the available options in Iray. You can safely ignore Section 2 of the manual as this relates to integrating Iray (which RealityServer already does so you don’t need to) as well as other parts which dive into the Iray C++ API.
Where you will find the most useful information is Sections 3, 4 and 5 which describe rendering in general, then specifically the Iray Photoreal mode and Iray Interactive mode. Sections 11, 12, 13 and 14 are also usually very relevant which talk about lighting, the environment dome, camera setup and how to create a physically correct scene.
All materials in RealityServer and Iray are represented using the Material Definition Language. This is an open source language for defining materials in a physically accurate, renderer independent fashion. RealityServer comes with a small set of core materials you can use but there are many ways to make more.
We offer a cloud based product called Medulr which allows you to create new MDL materials directly in the browser. There are also popular desktop tools like Substance Designer which can create MDL content. For pre-made materials NVIDIA offers the excellent vMaterials library which is fully compatible with RealityServer.
In addition to these tools you can also create your own custom MDL materials by writing MDL code yourself. The MDL Documentation is a good place to start however for an extremely in-depth and well written introduction check out the MDL Handbook by Andy Kopra. Even if you don’t plan to write many of your own materials, this is still a great read.
RealityServer also ships with a desktop application called Iray Viewer. This is a great tool for testing content and diagnosing issues with your scenes. One issue however is that NVIDIA do not include the same high quality importers for OBJ and glTF 2.0 that are present in RealityServer. We’ve written a quick guide to Using RealityServer Importers in Iray Viewer to get around this.
Dumping your current scene to a .ivm file which Iray Viewer can read is a great way to check that content you are creating programmatically is doing what you expect. Even if you don’t have materials or lighting setup yet, you can switch Iray Viewer to view the normal or texture coordinate buffers to check your geometry is being setup how you expect. This is extremely useful during early development.
You can dump your scene by adding an export_scene command to the requests from your client or in your custom V8 command using the export method on the scene, like this.
scene.export('filename.ivm');
That assumes you have a variable scene which is a Scene object either created in your script or loaded from an existing scene. See the Server-side V8 documentation for further details.
How do you get your 3D data into RealityServer? There are three main ways to load existing 3D content.
You can also create data from scratch in RealityServer using the generate_mesh command (or in V8 the Polygon_mesh.create static method). Of course you can combine all of these techniques together as well.
Out of the box RealityServer has direct support for the following file formats. We present them roughly in order of best ability to represent your content as authored to least.
The .mi file format is more or less the native format of RealityServer and Iray, if there is such a thing. This format provides the most direct way to express the creation of elements with minimal modification during import. These files are easy to create and manipulate and fairly human readable. The link above provides a good overview for the format. There is also a .mib binary format supported for rapid loading if needed (though the .mib files are specific to each version of RealityServer).
Note: that the .ivm format used by Iray Viewer is actually just a collection of .mi files which are loaded in a specific structure. So we don’t mention it as a separate format here but RealityServer can both import and export this format.
glTF 2.0 is a standard developed by the Khronos Group (of which migenius is a member). It was designed to be the JPEG of 3D and is heavily slanted for use with WebGL. However it now includes features such as PBR materials which makes it much more suitable for use with RealityServer (and great if you are combining WebGL and RealityServer together). RealityServer supports all of the relevant core features of glTF 2.0, including PBR materials.
Wavefront OBJ is a classic 3D file format that is still in wide spread use today, despite having significant limitations. These files have only very basic materials, no cameras, no lights and the format itself is not a standard so every application exports them slightly differently. In some applications this might be your only option and for transferring geometry they work reasonably well. You can then programmatically replace the low quality materials with better MDL materials if you like.
Of course you can programmatically change data imported from any format and it’s a great way to combine pre-generated content and dynamic programming.
The Open Asset Import Library (assimp) is an open source library which supports over 50 file formats for import and several for export. Some of these such as FBX and STL are quite popular while others are more obscure. The amount of useful information that comes through when using these formats varies considerably. Formats like STL for example do not even have materials, while FBX can bring basic materials and textures.
Iray has been integrated into many digital content creation tools. Most of these allow you to export either .mi or .ivm files for use in RealityServer. If available .ivm is preferred since it bundles up your textures and materials and removes the absolute paths. However if your favourite application doesn’t support .ivm you can export to .mi, load the file in Iray Viewer on the same machine and then save it out to .ivm to bundle everything up after the fact.
As of writing the following plugins and applications support direct export of .mi or .ivm files. Exporting from these tools has the advantage that artists can see exactly what their content will look like in RealityServer right inside their content creation tool before exporting since the same rendering engine is being used. The Iray plugins are commercial products available from the Iray Plugin Store. DAZ Studio Pro is a free application for character creation.
In Iray for Rhino you can export just by typing the IrayExport command. This will save an .mi file, so you’ll need to use Iray Viewer if you want to gather your dependencies. Please note, Iray for Rhino exports the last rendered state of the scene. So you have to of rendered at least once and if you make changes you need to render again to update the exported data.
Iray for 3ds Max adds the LW Iray+ (*.IVM, *.MI) option to the exportable file types shown when using the File → Export → Export… menu item. We recommend saving with the .ivm extension so that your resources are bundled together but you can also save as a raw .mi file if needed.
Iray for Maya has full documentation for its RealityServer export option which is found in section 19 of the User Guide. Basically you just select File → Export… or File → Export selection… as usual and use RealityServer [mi] as the file type. There are various options to control how elements will be named which are detailed in the documentation. This is a regular .mi export so Iray Viewer is needed if you want to bundle resources.
For DAZ Studio Pro just select File → Export… and then select the Mental Images Scene Format (*.mi) from the Save as type dropdown. Like Rhino and Maya, this is a regular .mi file so again, use Iray Viewer if you want to bundle resources.
Note: RealityServer ships with exporter (not rendering) plugins for 3ds Max and Maya. However we no longer recommend using them since they only support older versions of the software, do not work without mental ray (which Autodesk has now removed from these applications) and do not allow artists to render directly in the DCC application. Please use either a supported file format or the above plugins and applications instead.
If there is a format you’d really like to use and we don’t support it, or you have your own proprietary format, you can use the neuray API to write a custom importer plugin. These also work with Iray itself so can be used in Iray Viewer as well. If you are interested in writing a customer importer plugin we recommend contacting us for more details.
There is a lot of RealityServer documentation available, however it can be tricky to navigate and right now we still document older ways of doing things that we no longer recommend. All the information is there, however until we overhaul the documentation some hints are needed to find your around around.
We host the current version of all RealityServer documentation online at https://rsdoc.migenius.com/ which you can view at anytime. This is a copy of the same documentation that comes with RealityServer itself which you can access right on your server for improved performance (just navigate to http://server:8080/ or where ever you are running your RealityServer).
There are two main areas of documentation, the RealityServer Document Center and the Command Documentation. The Document Center is a static repository of documentation grouped into various categories while the Command Documentation is actually generated live by RealityServer at runtime. For the Command Documentation, if you have added custom commands you’ll want to view the documentation from your local RealityServer rather than our hosted version so that your commands are included.
If you navigate to the RealityServer Document Center the categories below are the most important when getting started. We’ve also linked some documentation outside the Document Center which is relevant. Unless you need a very specific feature or want to do C++ plugin development these should be the only places you really need to look.
Commands define all of the things you can do with RealityServer. While you will use the V8 API for the most part, you’ll still need to be familiar with what commands are available as well as use them for things which are not wrapped in classes in V8 yet.
The RealityServer Client is not supplied with RealityServer and the documentation is externally hosted. The link above will take you to the main documentation for the client with a great step-by-step tutorial for both Node.js and the browser. There is where you should start learning about how to call RealityServer commands.
The RealityServer Extras is extra functionality for the client which is not considered core to the process of communicating with RealityServer. Right now its main feature is a client side Camera class which you can use to help manage changes to the camera location for navigation when doing interactive rendering. The transformation class used for the camera can also be used for other objects if needed.
While you can use RealityServer in its default configuration you will very likely want to adjust some of the settings for your own purposes. This section describes all of the supported configuration directives that can be used in the realityserver.conf file found in your installation directory as well as the command line parameters RealityServer accepts. The Networking Directives and Logging Directives are the most commonly used ones.
This section contains documentation for things which did not fit into the other categories. It contains information on extra options some of our importers accept, the lightmap and wireframe renderers, HTTP upload features, server-side render loop, single pass stereo rendering, the smart batch command and the compositing system. There are some very cool features discussed here so it’s definitely worth reviewing.
The full Iray manual provided by NVIDIA is linked here. This is where you learn about what options Iray supports and how they work. There is also a great section called Physically plausible scene setup describing how to ensure your scene is correctly setup for physically based rendering. Basically everything you will want to set with the element_set_attribute command or the .attributes.set() method in V8 that has to do with rendering will be found here so you are likely to be referring to this set of documentation frequently.
This is likely to be the documentation you refer to most heavily. As discussed earlier, it is strongly recommended to use the V8 JavaScript API for most of your application logic which manipulates your scene. Here you’ll find details for all of the helper classes we provide in V8 so you don’t need to work with commands directly. There is also a great overview of the execution model of the V8 API which is important if you are coming from an environment such as Node.js since it is quite different (RealityServer is multi-threaded synchronous while Node.js is single threaded asynchronous).
If you will be writing your own MDL materials then you’ll need to refer to this documentation which contains the MDL specification as well as the description of the Base module which you will also likely want to utilise in your materials.
The migenius website contains many tutorials on various tasks you might want to perform with RealityServer. When you’re not sure about how to get something done it’s definitely worth checking out first. The blog is also valuable as it often contains descriptions of new features and functionality we add to RealityServer. The articles on 3D Transformations and SRT Transformations are particularly helpful when coming from other 3D environments and converting transformation details.
There is a lot of ground covered by this article but where should you get started after you’re done here and you want to create a RealityServer application? This is the process we use internally at migenius when starting a new RealityServer project.
You’ll find your custom V8 command will grow very quickly and that you can add functionality very easily using this process. You will likely get to the point where you want to have multiple custom commands instead of just one. You can also write your own JavaScript classes and put them in the v8/include directory to support greater code re-use if you find yourself writing the same code over and over again.
The RealityServer Client includes two basic example applications, one for the browser and the other for Node.js. While these give you all of the basics, we frequently get questions of the form “how do I do X?”. To help with that we have whipped up a custom V8 command which exercises many of the common tasks customers ask us about. While we ship quite a few V8 commands with RealityServer for you to look at, this example is much more extensive. Here is a list of things it does.
We will grow this over time but along with the tutorials on our website, this covers a lot of what customers usually need to get started. You can download this example below.
Extract the example to your RealityServer directory. It will add a small model to your content_root/scenes directory in the Tutorial folder and also a new custom V8 command in v8/commands called tutorial_scene_maker.js. If you open up the command in your favourite text editor you will see it is heavily commented and you should be able to follow what it is doing.
To actually call the command and have it generate a scene we can render, you could write a small RealityServer Client based application (which would be a good exercise), or alternatively just send the following JSON-RPC sequence using cURL or Postman (cURL command shown as an example).
curl -X POST -H "Content-Type: application/json" -d '[
{"jsonrpc": "2.0", "method": "create_scope", "params": {
"scope_name": "tutorial_scope"
}, "id": 1},
{"jsonrpc": "2.0", "method": "use_scope", "params": {
"scope_name": "tutorial_scope"
}, "id": 2},
{"jsonrpc": "2.0", "method": "tutorial_scene_maker", "params": {
"scene_name": "tutorial_scene",
"export_filename": "scenes/Tutorial/tutorial-debug.ivm"
}, "id": 3},
{"jsonrpc": "2.0", "method": "render", "params": {
"scene_name": "tutorial_scene",
"renderer": "iray",
"render_context_options": {
"scheduler_mode": {
"type": "String",
"value": "batch"
}
}
}, "id": 4},
{"jsonrpc": "2.0", "method": "delete_scope", "params": {
"scope_name": "tutorial_scope"
}, "id": 5}
]' -o render.jpg "http://127.0.0.1:8080/"
If all goes well you should get an image called render.jpg which looks something like the one shown above. It will also dump the scene to a .ivm file so you can open it in Iray Viewer. Converting the above sequence into a RealityServer Client program is also a great exercise to confirm what you have learned here.
Every new version of RealityServer includes a file relnotes.txt and neurayrelnotes.pdf. These contain the RealityServer and Iray release notes respectively. This is the best way (aside from our blog) to find out about new features that have been added, bugs that have been fixed and other changes in the software. If we ever make any breaking changes this is where you’ll be notified (this happens quite rarely).
If you’ve gotten this far you’re already half way to becoming a RealityServer expert, however we understand you might need more help or want to get started even faster. We’re here to help so reach out and get in touch.
Paul Arden has worked in the Computer Graphics industry for over 20 years, co-founding the architectural visualisation practice Luminova out of university before moving to mental images and NVIDIA to manage the Cloud-based rendering solution, RealityServer, now managed by migenius where Paul serves as CEO.