Not logged inOpenClonk Forum
Up Topic Development / Developer's Corner / Asynchronous Scripts
- - By Clonk-Karl [de] Date 2012-03-02 20:43 Edited 2012-03-02 21:17
Since we move more and more GUI into script, and also triggered by #723, I spent some thoughts on asynchronous scripts. The idea behind this is to run some scripts only at one host in the network and not for others (for example to create menus, HUD elements, etc.) but to still make it impossible for script to spoil the synchronization in network games. The basic idea is as follows:

We introduce an "Async" flag to C4Value. Engine functions specify for their parameter types whether they are allowed to be asynchronous and then on function call a corresponding typecheck is performed. If an engine function which requires a synchronous parameter is called with an async C4Value a runtime error occurs. However, in the other direction, async parameters are allowed to be called with synchronous C4Values. For example, RemoveObject() requires a synchronous this pointer, but GetName() only an asynchronous one. If GetName() is called with a synchronous C4Object though then it returns a synchronous string, otherwise an asynchronous one.

Next, objects (and, in the function pointer branch, also proplists I suppose) can be asynchronous. There should be an "Async" flag in C4Object. The this pointer returns a synchronous C4Value for both synchronous objects and asynchronous objects. There will be a new function CreateAsyncObject to create asynchronous objects. The FindObject* family of functions will either have a new Find_Async flag which allows them to return asynchronous objects, or there will be a new FindAsyncObject* family of functions.

array/proplist access with [] or . return an asynchrous object if the array or proplist is asynchronous itself. array/proplist setters are only allowed to set asynchronous values if the array/proplist is asynchronous itself. This means synchronous objects are not allowed to have local asynchronous variables!

Whenever a conditional jump (if/while/for) is performed on an asynchronous C4Value, then every C4Value used in that block becomes automatically asynchronous -- even if that block is not even executed! This might actually be the most tricky part to implement, but the "Async" flag as such must stay synchronous!

Here is an example:

var i = 5, j = 7; // here, i is synchronous
var x = [1,2,3] // x is synchronous and so are x[0], x[1], x[2].
var y = x;
if(FindObjectAsync(Find_ID(Clonk))) // this is a condition on an async value!
{
  i = 6;
  FindObject(Find_ID(Clonk))->RemoveObject(); // fails because FindObject() returns an async object within the if() block
  CreateObject(Clonk); // fails because the this pointer is async within the if() block.
  x[3] = 5; // fails, because x is asynchronous but points to a synchronous array. For the synchronous array, it is not allowed to set an asynchronous value (even "5" in an if(<async condition>) block is asynchronous).
}
bla(x, y, i, j); // here, i and x are asynchronous. j and y are not, and neither is the array pointed to by x and y.
bla2(x[2], i); // even though x[2] is synchronous, the array getter returns an asynchronous value because x became asynchronous
bla3(y[2], i); // works because y is synchronous.
x[3] = 2; // works, but x stays asynchronous
FindObject(Find_ID(Clonk))->RemoveObject(); // works

In addition, there can be asynchronous engine callbacks. Such callbacks will always be called in an asynchronous context, i.e. as if they were within a if(<async condition>) block. Examples for such callbacks include mouse
hovering or drag+drop feedback. There should also be a callback similar to InitializePlayer for every player at the local client.

For records, I suggest to record every asynchronous callback that was performed, plus its parameters. This allows in records to view local players including their HUD and so on, but not the HUD, menus etc. of remote players (simply because that information is no longer available at all).

Feedback by both scripters and engine developers is highly welcome. Would such an interface be suitable for creating even more awesome GUIs in script? Do you see any way how obtaining different non-Async C4Values on different machines if this feature were implemented this way?

Edit: Changed the semantics of the example.
Reply
Parent - - By Caesar [de] Date 2012-03-02 21:21
I've spent some thougts on it quite a while ago and I've never been quite sure if it was worth the hassles. Doing expensive AI things over network sounds awesome though.

>Next, objects (and, in the function pointer branch, also proplists I suppose) can be asynchronous. There should be an "Async" flag in C4Object. The this pointer returns a synchronous C4Value for synchronous objects or an asynchronous C4Value otherwise. There will be a new function CreateAsyncObject to create asynchronous objects. The FindObject* family of functions will either have a new Find_Async flag which allows them to return asynchronous objects, or there will be a new FindAsyncObject* family of functions.


That's something which is a bad idea in my eyes, e.g. The only purpose of asynchronous objects would be gui elements, because all OOP-Things should be doable with proplists as soon as Guenther merges his changes (okay, something is still missing..) For these gui-elements, I'd rather like to supply an advanced interface instead of hacking into C4Object.

What I don't like either is this mixing of synchronous and asynchronous things with dynamic interchange inside function. That's why:

>Whenever a conditional jump (if/while/for) is performed on an asynchronous C4Value, then every C4Value used in that block becomes automatically asynchronous -- even if that block is not even executed! This might actually be the most tricky part to implement, but the "Async" flag as such must stay synchronous!


Tell my if this system could cause fewer unforeseeable side-effects: Two different scopes and function modifiers. Functions or code parts that were declared async have rw access to variables in asynchronous and only read access to synchronous variables. Synchronous functions have rw access to normal variables and write access to asynchronous variables. Trying to call an async function from sync space would force the return value to 0. Trying to call a sync function from async space is illegal. In that regard, a third class of functions, which are const might be usefull. Variables would be synchronous or asynchronous by a declaration modifier.
Parent - - By Clonk-Karl [de] Date 2012-03-02 21:29

> That's something which is a bad idea in my eyes


It would be quite flexible though and might allow to do some things that we do not think of yet.

> Trying to call an async function from sync space would force the return value to 0. Trying to call a sync function from async space is illegal.


How would you handle functions which should work both for sync and async stuff? For example GetName(), Sqrt(), etc.? In the engine you could play tricks, but there will also be similar GetSomething() functions in script.
Reply
Parent - By Caesar [de] Date 2012-03-02 22:04

>How would you handle functions which should work both for sync and async stuff? For example GetName(), Sqrt(), etc.? In the engine you could play tricks, but there will also be similar GetSomething() functions in script.


Yep, you're right, the third function class which I talked about, const functions, are needed. Read access to synchronous values, call access only to functions declared const.
Parent - - By Clonk-Karl [de] Date 2012-03-02 22:54
What would you think about only allowing async (function-local) variables to be assigned in an asynchronous context, instead of making synchronous variables asynchronous automatically? This would maybe get away with the complicated "variables become asynchronous when inside an if(...)" thing. Maybe this means we need an explicit declaration for a variable to be asynchronous, but that might be OK.

In the example in the initial post this would mean that the "i = 6" would fail unless i was declared as async. Or maybe "i = EnsureAsync(6)".
Reply
Parent - By Caesar [de] Date 2012-03-03 17:45
Sounds a lot more sane. I don't like the syntax, something with that much magic included should not look like any arbitrary function, but that's detail.
Parent - - By Günther [de] Date 2012-03-02 21:29
A neat idea, but I think the required changes would be far, far too complicated, permeate the entire script engine, and probably have too many tricky corner cases that are hard to reason about.

I think a better source of inspiration is the separation that web browsers have between different origins or the page and browser UI. The various parts have all their own global object, and instead of direct references they only get proxy objects that only allow safe operations. (At least, I think Gecko works something like that.)

But designing a declarative solution for menus should be relatively less work.
Reply
Parent - - By Clonk-Karl [de] Date 2012-03-02 21:36

> I think a better source of inspiration is the separation that web browsers have between different origins or the page and browser UI. The various parts have all their own global object, and instead of direct references they only get proxy objects that only allow safe operations. (At least, I think Gecko works something like that.)


Hm. Does this mean that these asynchronous callbacks would have a completely different Scenario and object list and all, so that they would somewhat "live in their own world"?

> But designing a declarative solution for menus should be relatively less work.


Maybe on the short term, but then every week scripters will request some new feature ;). Right, boni?
Reply
Parent - By boni [at] Date 2012-03-02 22:27
What? Me requesting stuff? At least mine are (usually) reasonable and useful! ;P
Parent - By Clonk-Karl [de] Date 2012-03-02 21:39

> have too many tricky corner cases that are hard to reason about.


Hm, one of that corner cases might be network savegames... probably async objects should not be saved at all and scripts need to take care of restoring them when resuming the savegame.
Reply
Parent - - By PeterW [gb] Date 2012-03-03 22:02 Edited 2012-03-03 22:05
To add something to everything that has been said: Also think about that this is bound to create new desyncs. Probably over a longer time, as we are increasing the amount of code that's touching both synchronous as well as asynchronous data. Async objects sound like a nightmare - just consider cross checks (special layer?). This makes it imperative in my view that we get a more robust DebugRec solution at minimum.

As this isn't really a new proposal (well, at least I have heard it already), I have a pretty stable opinion that we don't want to do this, even if it's just for async callback functions that can only read variables. The reason is mainly that I have so far not come across a second example besides GUI.

And for a specific application, a dedicated sub-language might be best. Like the one we built for FindObjects... I think that has shown how well it can work out to just provide scripters with a few powerful combinators.
Parent - - By Clonk-Karl [de] Date 2012-03-04 20:06

> Also think about that this is bound to create new desyncs. Probably over a longer time, as we are increasing the amount of code that's touching both synchronous as well as asynchronous data. Async objects sound like a nightmare - just consider cross checks (special layer?). This makes it imperative in my view that we get a more robust DebugRec solution at minimum.


I fully agree with this. This surely isn't something that should be rushed and needs good debugging capabilities and testing. But we can start slowly, by allowing only C4Values to be asynchronous to start with, and not arrays, proplists or objects. And then do one step after the other.

> The reason is mainly that I have so far not come across a second example besides GUI.


I think this approach would be quite flexible. For example, we could have functions that return bone position/orientation/transformations and do things like a trajectory preview asynchronously. A new function "SynchronizeCall" could be used with async parameters to be called synchronously at the next control frame to get back from the "async" to the "sync" world. That would allow showing messages for a given number of seconds instead of a given number of frames. I am sure there is plenty more stuff that can be done.

I guess I'll try and get something very basic done and then see how much work it is to get it "production-ready" and decide whether this has a future or not. I appreciate all your arguments very much, but the question of allowing parts of scripts to run asynchronously is just more interesting and motivating to me than to work on an independent interface. It's the researcher within me :)
Reply
Parent - By Günther [de] Date 2012-03-05 15:54

> but the question of allowing parts of scripts to run asynchronously is just more interesting and motivating to me than to work on an independent interface


Can I at least convince you to aim far maximum separation between the two worlds? That is, not to separate them at the granularity of individual values, but to have asynchronous functions that only have access to asynchronous engine functions and a separate asynchronous global state. That should be doable with modifications to perhaps a dozen or so places in the parser and not require adding logic to every single piece of the C4Script interpreter. (Which I'd like to speed up instead of slow down, by the way.)
Reply
Parent - - By PeterW [gb] Date 2012-03-05 16:44

> For example, we could have functions that return bone position/orientation/transformations and do things like a trajectory preview asynchronously.


Hm, tempting, yes - but that's still nothing we really need full scripting flexibility for - that's basically a loop with some slightly more fancy math and hit checks. And it's probably about the most intelligent we ever want to make user interfaces - we should not allow the "view" part to become complex enough to effectively implement game logic.

Also remember that we have open access to the engine code now, so there's no reason that a scenario designer couldn't submit an engine patch to put in the specific GUI function he needs. We should probably start encouraging that sort of thing.

> the question of allowing parts of scripts to run asynchronously is just more interesting and motivating to me than to work on an independent interface.


You prefer water-proofing thousands of existing code lines to building a new clean implementation? Your standards of "interesting work" surely differ from mine... ;)
Parent - By Caesar [de] Date 2012-03-05 20:50

>so there's no reason that a scenario designer couldn't submit an engine patch to put in the specific GUI function he needs.


Basically, that's true, but I want to remind you that not everyone is able to do C++ on a level that is sufficient for CR...
Parent - - By Newton [de] Date 2012-03-05 20:56

> probably about the most intelligent we ever want to make user interfaces


I wouldnt count on that
Parent - - By PeterW [gb] Date 2012-03-06 00:23
Anything in mind? My reasoning is that the view should only be more or less direct representations of the game data... So unless we move game data into asynchronous parts (which we positively shouldn't), the code complexity has bounds.
Parent - - By Caesar [de] Date 2012-03-06 00:50
I've seen a custom Minimap as hud. It's a direct representation of the game data. And yet, it forced the way fps down.
Parent - - By PeterW [gb] Date 2012-03-06 13:44
That's probably due to it being implemented poorly.
Parent - - By Caesar [de] Date 2012-03-06 14:27
Hm, to clarify, it wasn't a minimap, but a map showing the solid areas about 2000 pixels around you. The map-data was cached and updated over time, the problem was updating the display. You had to control the placement of hundreds of objects per player, and to make it look good, at least all three frames. It worked with one or two players with 'reasonable' lag on a mediocre machine. Using a static map picture wasn't possible either, because there is no clipping.
Parent - - By PeterW [gb] Date 2012-03-06 15:47
Well, that's a great example of something that might benefit a lot from not being implemented in script, if you ask me ;)
Parent - By Caesar [de] Date 2012-03-06 15:49
It sure is, neither Mimmo, me, nor anyone else involved there would have been able to do it in the engine.
Parent - - By Newton [de] Date 2012-03-06 18:26
No, not really. Right now, all the extra-stuff that is displayed related to gamepad controls comes to my mind. Or the production menus.
Parent - By boni [at] Date 2012-03-06 20:02
Gamepads need a completely seperate UI anyway imo.
Parent - By Sven2 [de] Date 2012-03-04 20:12

> The reason is mainly that I have so far not come across a second example besides GUI.


Music/sound related callbacks would be possible.
- By Caesar [de] Date 2012-03-23 11:44
Just on a side-note: We already have asynchronous stuff... String tables. I already tripped into that one.
Up Topic Development / Developer's Corner / Asynchronous Scripts

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill