2023 Aug: Modules, Requires & Safety!

Package Require, Modules, Safety, Docs++ and more!

Tune in for a roundup of the latest updates from August!

Async Operations Improvs

In nanos world we have a bunch of operations and methods that run on background, including core/internal and scripting operations (like HTTP.RequestAsync, File.ReadAsync, SceneCapture.EncodeToBase64Async, etc), when executed, those methods start a new thread to be executed asynchronously.

The trickiest part is keeping control of them, i.e. we can't fully destroy a File entity until it's async operations are finished, this is why the server could freeze for some seconds if we attempt to destroy a File that has is performing async reads executing (or when reloading a package that has a bunch of them).

The problem with it is that until last update we had a global control of all threads running, i.e. if we wanted to destroy a File, we needed to wait for all async operations to finish to make sure no operation related to that File was still running.

But this changed with this new update! Now we have a new system controlling all operations that are running per-entity, meaning that one entity will not be affected (or affect) any other part of the server by needing to wait for something that is not required by it.

This improvement also fixed a bunch of deadlocks we were having and improved a lot the overall stutters when dealing with async methods.

Package.Require Caching

We got some improvements on how Package.Require works internally, and also it got a new parameter: force_load.

Now it caches the return value of a file to be returned if you try to load the file again. You can bypass this behavior by passing true to the new parameter added: Package.Require(file, force_load).

New Modules Behavior

Modules are external native code (.dll or .so) that you can run on the server.

Right now they work by looking for your binary files inside the Server/Modules/ folder, then you can load them by using the native method require("module_name").

This workflow is undergoing a great improvement in the upcoming updates. In the future, we have the goal to make modules to behave like Packages - i.e. creating a Package of type Module that will load binaries in a standalone way, this way we can have it available as Packages on our Store/Vault and they can be loaded in a more standard way (with package dependencies or Package.Require()).

So with that in mind, we did a first step towards that: we implemented our own binary loader, having control over loading .dlls or .so without needing the native require() anymore.

This way, now temporarily* Package.Require() can load .dlls/.so and require() is now just an alias to Package.Require(), making everything still backwards compatible.

tip

This aliasing thing also solves several bugs and problems related to the native require, and makes everything to stay under our sandboxed environment control.

note

*Temporarily as in the future we may change the way to load binary modules with the new Modules Packages type.

At the highest level, nothing changed and should keep working as expected, but under the hood everything works different now and allows us to expand and advance to newer implementations!

Scripting Safety

We are implementing a new layer of security on server side to prevent malicious code to run without the server creator be aware. Please let me know your opinion regarding those changes:

Unsafe Libraries

Now the methods os.execute, os.rename, os.remove, os.exit, os.getenv, os.tmpname, os.setlocale, dofile, loadfile and all io.* can only be executed if the server is started with the new command-line parameter: --enable_unsafe_libs.

We're studying the possibility to make this a per-package setting defined in the Package.toml instead of requiring that command line to be started, this way the server owner is alerted if a Package is "potentially unsafe" (as the Packages using those methods will not work if started through the New Game Menu right now).

File Access

This way, all file access is recommended to be made through the File class (instead of io.*), having a sandboxed and secure way of accessing files.

And in addition to that, now File can't access the Config.toml file directly anymore, to prevent malicious packages from stealing the server's token for example.

Recursive Scripting Tables

We did a big improvement in our internal serialization/deserialization implementation when passing data from Lua to our Native methods (or scripting events).

Before we had a system that suppressed recursive tables (tables with references to any parent table) from being passed if they were already referenced in that stack of deserialization (to avoid infinite loops), but now we improved our algorithm and we can pass recursive tables!

Internally they are cached in case we want to deserialize a same table, and it's value is just dereferenced back to lua stack!

New Methods and Params

Several new methods and parameters were added to classes, to mention a few:

• New parameter to File.GetDirectories() and File.GetFiles(): max_depth, to limit the folder depth of searching.
• New method Assets.GetAssetPath() to return the real asset path given an asset reference.
• New method Character.GetVehicleSeat() to return the current seat index of the current vehicle.
• New method Client.Disconnect() to disconnect myself.
• New method Viewport.GetViewportScale() to get the viewport size.
• New method CharacterSimple.CallAnimationBlueprintEvent() to call an event on the Animation Blueprint instance.

Docs Improvements

We added new Base Classes Pages and API to the docs: Base Vehicle and Base Pawn to unify all shared methods between VehicleWater & VehicleWheeled and Character & CharacterSimple.

This way we have less duplicated methods and events to maintain between those classes.

Conclusion

Safety is one of the main concerns in sandbox games like nanos world, as we are always loading and running third-party created scripts and packages, we consider it very important for servers creator to be able to fully trust and know what they are executing. This month we got a great improvement advance about this, and we will definitely continue to improve that in the next updates.

Also the Modules upgrade is a very important improvement for nanos world to become more robust, for the next updates we will move forward with the consolidation of the Module Package and tying everything together.

A small note of Unreal 5.3: we are still waiting it's fully release, until that we will not move forward to update the game.

As always, if you have suggestions or ideas, don't hesitate to post it on our Feedback Hub! Thanks for another month!