Not logged inOpenClonk Forum
Up Topic Development / Developer's Corner / Function pointers
1 2 Previous Next
- - By Günther [de] Date 2011-12-22 23:27 Edited 2011-12-22 23:36
With 5.2 done, I've resumed playing with the script engine and added the next big "missing feature": First class functions. The basic idea is that functions can be stored in C4Values, and -> looks up the function in the proplist instead of hidden function storage that's currently used. The rest is a ton of details to make it look like C4Script always had first class functions and uses them everywhere.

- Add "new p {}" syntactic sugar for "{ Prototype = p }"
- '->' also works with proplists
- Add a proplist representing the global scope, named Global
- Add a proplist representing the scenario, named Scenario
- this returns the Scenario proplist in Scenario calls
- this returns the definition in a definition call
- Add def and effect parameter types
- Remove unused PrivateCall and ProtectedCall
- Remove unused function "descs"
- Remove ScriptGo, ScriptCounter, goto and the ScriptN callbacks

For the details, check out the repository.

Now, the last remaining problem is the behavior of the inherited and _inherited keywords and of plain function calls.
- inherited in definition Foo points to different functions depending on what other definitions were included before Foo, which is the only thing that makes it still necessary to have multiple copies of functions instead of one for for every definition using it. (We're basically solving the Diamond Problem in an unconventional way.) I'd like to restructure the parser a bit to make local variable initialization work across script reloads, and these multiple copies of function makes that inconvenient.
- Is there a common use case for calling a function with this != the proplist it's in? Perhaps we could replace #including libraries with that and ban multiple inheritance?
- I think the main use case for inherited (besides #appendto scripts) are callback chains. But this basically relies on everyone writing _inherited(...) into callbacks in case one of the #included definitions needs the same callback later. Other languages/libraries are using callback lists instead, which can be dynamically manipulated. We're using Effects and GameCallEx, which are kind of awkward in comparison.
- Should the parser look up the function to be called when not using "->", or should the lookup be done at runtime, as if one had used "this->"?

I probably need to read some recently written scripts to get a feeling for the patterns that could be simplified with function pointers. Unfortunately, this isn't as obvious as the EffectVar stuff...
Reply
Parent - - By Newton [de] Date 2011-12-22 23:43 Edited 2011-12-22 23:45
Could you make a hello-world example how the script would look like then? @ Global/Scenario proplist, functions in proplists(?), this-usage in definitions?

I am also curious about your thoughts/plans to replace #include with another system. Currently I can't picture how that would look like. Any more details?
Parent - By Günther [de] Date 2011-12-23 19:31

> Could you make a hello-world example how the script would look like then? @ Global/Scenario proplist, functions in proplists(?), this-usage in definitions?


I haven't made any actual syntax changes beyond the "new foo {}" thingy.

Global and Scenario mostly work like definition proplists. So like you can do "Clonk->IsClonk()" now, you can do "Scenario->InitializePlayer()" instead of GameCall("InitializePlayer"). Except that the scenario proplist is also writable like an object, so you now can use local variables in scenario scripts.

this in DefinitionCalls like "Clonk->Foo()" means that in Foo, this==Clonk instead of this==nil.

Functions as values mean that you can do "obj.Hit = Flint.Hit" to make obj explode like a flint does.
Reply
Parent - - By Zapper [de] Date 2011-12-23 12:49 Edited 2011-12-23 12:54

>- Should the parser look up the function to be called when not using "->", or should the lookup be done at runtime, as if one had used "this->"?


Is there any disavantage of the parser already looking it up?

>(besides #appendto scripts)


I think #appendto scripts are an important part when you have something as modular as the Clonk package system. I think it would be a pity to lose that

Until now I really liked the way Clonk handled inheritance and overloading/appendtoes.

PS: On the other hand some sort of "scoping" would be nice - that is: library A has private function "Draw" and B has private function "Draw", C inherits from A and B and executes some code of B that uses "Draw" - A's "Draw" should not be called here because it was never ment to be called from somewhere else. ("private"/"public")

PPS: And the system should be easy to understand/work with to not scare off beginners :/
Parent - - By Isilkor Date 2011-12-23 13:04

> Is there any disavantage of the parser already looking it up?


If you have the parser look it up, you lose the possibility of switching the function out for another one at runtime.
Reply
Parent - By Günther [de] Date 2011-12-23 19:35
On the other hand, if you have "func Foo() { Bar(); } func Bar() {Message("Hello World"); }" and do "var p = { Bla = Foo }; p->Bla();", you will get a Hello World message instead of a function-not-found error message.
Reply
Parent - - By Sven2 [de] Date 2011-12-23 13:14
Both #appendto and Effects are kinda awkward, but their functionality is definitely required and useful. It would be great if we had better mechanisms for that. For instance, instead of an #appendto to the rock script to overload the Hit callback, it could be possible to write Rock->Hit = MyHitImplementation.

We would have to think of a method to call the previously overloaded function though. The overloaded function is often integrated into the logic of the overloader, for example as in

func MaxContents() { if (foo) return inherited()*2 else return inherited(); }.

So I don't think it's sufficient to just collect the functions as in Rock->Hit = [Rock->Hit, MyHitImplementation].
Parent - - By Günther [de] Date 2011-12-23 20:50
What I'd like to do is limit #include to one script per definition, and ban global functions from #appendto scripts. inherited would work as before, but you can always see exactly where it points by simply looking at the #include/#appendto script(s). For the remaining functionality, one could do "local IsPowerConsumer = Library_PowerConsumer.IsPowerConsumer". Another possibility would be to ban getting the same function from multiple #include-scripts.

Both would probably require that the including object calls LibFooInit() in Construction() instead of each Construction() calling _inherited. The total lines of code shouldn't grow too much from that. But I'm not sure how complicated and widespread other callback inherited-chains are.
Reply
Parent - - By Zapper [de] Date 2011-12-24 01:25

>one could do "local IsPowerConsumer = Library_PowerConsumer.IsPowerConsumer"


That doesn't really sound feasible with big libraries :/

What I liked about the old library system was that you really could get the ~whole functionality of an object/library just by saying, "hey, I want to use that object!" via #include.
And to be honest I would rather have a system that is usable and easy than a system that is beautiful from a code designer's perspective :/
Parent - - By Günther [de] Date 2011-12-24 15:16
I'd argue that the way inherited currently works isn't easy.
  #include Bar
  func Foo() { _inherited(...); }


Which function gets called by that _inherited? Answer is: You can't know without looking at every script. It could be from some script that appends to Bar, or from some entirely unrelated script that merely got included before the example script in case Bar doesn't have a Foo function. This only works in practice because we mostly use _inherited for callback functions which don't rely too much on exactly which function gets called. And these callback chains fall apart when one callback function forgets the _inherited.
So while the occasion to change all this is my desire to make the script parser more sane, we should use the opportunity to build something better.
Reply
Parent - By Isilkor Date 2011-12-24 22:52
We could disambiguate it the way that C++ does:
#include Foo
#include Bar
func Baz() { ++counter; return Foo::Baz(); }
Reply
Parent - By Zapper [de] Date 2011-12-24 23:06
I agree that the (current) system is partly quite awkward (especially the whole duplicate-names-in-included-scripts thing).
I am just a little bit afraid that either old well-used features (#appendto/the possibility to "freely" overload functionality in scripts) get thrown out because the don't fit with the new design or that "overloading" then means a lot of new bloat around what you actually want to accomplish
Parent - By Newton [de] Date 2011-12-24 19:51 Edited 2011-12-24 19:53
ack

The complete source code is based on multiple inheritance.
Parent - - By PeterW [de] Date 2011-12-24 01:08 Edited 2011-12-24 01:12
Can we get the "new p {} actually calls p.new()" feature? With the standard "new" implementation being some standard - possibly engine - function? That could open up a lot of very prototypic-y programming patterns.

Like, say, have effects getting created by doing
var effect = new Effect { period = 100, call = &MyFunc };
Parent - - By Günther [de] Date 2011-12-24 15:00
And what should the syntax for plain old inheritance be? Say you want to create one MegaClonk prototype that derives from the Clonk, and then create a bunch of instances of that. At the moment, the first part would be
  static const MegaClonk = new Clonk { MaxEnergy = 100000 };,
and the second
  CreateObject(MegaClonk);.
Also, Effects call more for something like
  local GlowEffect = { // or "new Effect {" if we want
    func Start() { [...] },
    Interval = 42
  };
[...]
    CreateEffect(GlowEffect, obj);

That way, not every effect instance has to carry redundant data.
Whether proplists have magic engine behavior depends on the proplist, not the prototypes of the proplist (except that objects must have a definition in their prototype chain). I like that the syntax reflects that - Magic engine behavior comes from calling an engine function. Hiding the difference between them and plain script constructs isn't a goal, because the rules differ a lot.
Reply
Parent - By Zapper [de] Date 2011-12-24 23:09

>Say you want to create one MegaClonk prototype that derives from the Clonk, and then create a bunch of instances of tha


Hopefully enough syntax sugar to make it possible that every beginner still starts with the SuperTeraXFlintBomb as their first object :)

>Also, Effects call more for something like[...]


Mh, that cries for more than one script per object definition!
ScriptGlowEffect.c would contain the properties for the glow effect then - less nesting of braces! (Especially when they contain long functions)
Parent - - By PeterW [de] Date 2011-12-25 17:12
Well, the prototypic way of thinking about it would be to see definition as "inactive" objects. Which, thinking about it, might be a useful status to have e.g. for Sven's sections - from the script's perspective, it might be a good idea to have the engine simply turn all objects into prop lists without engine magic?
Parent - - By Günther [de] Date 2011-12-26 19:56
So you'd replace the CreateObject function with the SetStatus function? I'm not sure that's an improvement - though we could keep a
global func CreateObject(d, x, y, o) {
    var obj = new d {};
    obj->SetStatus(1);
    obj->SetPosition(x, y);
    obj->SetOwner(o);
  }

around. (We can't have the constructor create an active object because that'd change the state of the game, and the constructors have to be side effect free since they would get called from the parser for local foo = new bar { baz = 42 };)
Reply
Parent - By PeterW [de] Date 2011-12-27 17:37
Currently I'm just throwing ideas around, maybe the whole thing fits together in some way. We could make "CreateObject" the function that assigns engine-magic to the object, and have "RemoveObject" just disassociate the object with a game object? That could solve some strangeness with deleted objects as well.
Parent - - By Günther [de] Date 2011-12-25 21:04
To ground the discussion a bit, I gathered some data about how inherited is actually used. And the only objects doing anything remotely interesting with it are Goal_DeathMatch and Goal_LastManStanding. In every other case, the call chain look like plain old single inheritance. Apparently we don't really need complex callback chains. So I think what I'll do is simply ban including the same function from multiple scripts and slightly rewrite those two objects. inherited will then simply point to a function from one of the script that are included or appended to, and never to a function that got inserted by a script from elsewhere.
Attachment: inheritedcalls.pdf - inherited/_inherited overview (without _inheriteds that do nothing) (33k)
Reply
Parent - - By Zapper [de] Date 2011-12-25 23:12

>So I think what I'll do is simply ban including the same function from multiple scripts


I am not a fan of this I guess.
What happens if you include two scripts that use the same function then? One of the functions is completely hidden and cannot be called at all?

Also the pdf doesn't work (looks empty to me - I can copy&paste text from it, but that is not exactly helpful). Can anyone confirm?
Parent - By Günther [de] Date 2011-12-26 02:32

> What happens if you include two scripts that use the same function then?


You get a helpful error message, of course. That has the nice aspect of catching accidental name collisions.

The pdf is probably simply too big for your viewer. I could try to make graphviz make it smaller, but it can be nicely summarized - almost all inherited chains are only two functions long.
Reply
Parent - - By Randrian [de] Date 2011-12-25 23:31
Well I don't know if the content we now have is a good source off data.
We don't have extension packs now, which a part of Clonk (and should be of OpenClonk too). And every pack my want to add new functionalitys to the clonk. I hope you system covers that too.
Reply
Parent - - By Günther [de] Date 2011-12-26 02:43
Keep in mind that the reason I'm doing this is that I'm adding function pointers. Expansion packs will be able to create exploding Clonks by doing CreateObject(Clonk).Hit = Flint.Hit. (Or continue using #appendto.)
Reply
Parent - By Newton [de] Date 2011-12-28 12:04
Hmm, this in-place definition of objects would be like anonymous classes... I never really used that feature (in Java) yet.
Parent - - By Caesar [de] Date 2011-12-26 00:32

>So I think what I'll do is simply ban including the same function from multiple scripts


Sec, you mean, I can't have 2 objects including a third with both calling inherited() in a function? What's with all the library design then?
Parent - - By Günther [de] Date 2011-12-26 02:36
Of course not. I mean having A and B both defining a function Foo and C including both A and B. At the moment you can make that work out to a call to C.Foo calling both A.Foo and B.Foo via inherited, but only two objects use this feature, so I'll take the enhanced protection against accidental name collisions instead.
Reply
Parent - - By Zapper [de] Date 2011-12-26 10:56 Edited 2011-12-26 10:58

>but only two objects use this feature, so I'll take the enhanced protection against accidental name collisions instead.


As Randrian said: I am not sure whether the current work state of OpenClonk is far enough to base the decision on it whether features are needed

PS: if I include A and B - do they current have an order? if so, what speaks against making include-chains work like C->B->A? As in "#include works like you copy&pasted the whole script"
Parent - - By Günther [de] Date 2011-12-26 19:47

> PS: if I include A and B - do they current have an order? if so, what speaks against making include-chains work like C->B->A? As in "#include works like you copy&pasted the whole script"


I answered that in the first post in this thread:

>> inherited in definition Foo points to different functions depending on what other definitions were included before Foo, which is the only thing that makes it still necessary to have multiple copies of functions instead of one for for every definition using it. (We're basically solving the Diamond Problem in an unconventional way.) I'd like to restructure the parser a bit to make local variable initialization work across script reloads, and these multiple copies of function makes that inconvenient.


The fact that you didn't know that it currently works that way convinces me even more that it is just too surprising or obscure a feature to keep :-) Especially since "inconvenient" is an euphemism for "I spent hours trying to find a solution that I liked and found none". Also, since the engine will get simpler as a result of this change, we can spend some code on a better solution.
Reply
Parent - By Zapper [de] Date 2011-12-26 20:28
mh, I believe I am still not convinced :/
Parent - - By Günther [de] Date 2011-12-27 18:49

> if so, what speaks against making include-chains work like C->B->A? As in "#include works like you copy&pasted the whole script"


Ignoring the implementation considerations for a moment, "as if copying the script" may be easy to explain, but it isn't the most useful tool. One normally wants some isolation against other scripts to avoid accidental name collisions. Splitting out a helper function in a library shouldn't risk breaking a random extension object that uses that library.
Reply
Parent - By Zapper [de] Date 2011-12-28 01:13

>Splitting out a helper function in a library shouldn't risk breaking a random extension object that uses that library.


To be fair, it shouldn't break another library either that is included alongside said library :)
Parent - By Newton [de] Date 2011-12-28 12:05
The PDF only displays as a white stripe, if I zoom in I still don't see anything.
Parent - By Günther [de] Date 2012-01-25 00:58
So discussion during the OCM didn't yield any solid conclusions, and I finally got fed up with not being able to decide what the code should do, so I opted to keep the inherited behavior largely as-is. The parser now searches for the function to put the code into when it reaches the point where the preparser created the function. This creates problems for functions that are duplicated because their script got included multiple times or that are actually duplicated. I think I'll modify the #include logic to skip scripts that are already included and ban multiple definitions of the same function in one script. Global functions now get multiple copies of their code if their script gets included or appended, but that's just a minor bit of wasted memory.

Next steps:
- the mentioned #include deduplication
- fix function lifetimes (either not deleting functions until game end or reference counting)
- test
- merge
- Add/replace lots of features that should be using function pointers and only didn't because those didn't exist
- Take another look at inherited using the lessons learned from that. Hopefully a pure "how can we make this better" will be more productive than the "how can we make this better while also solving the local variable initialization problem".
Reply
- - By Zapper [de] Date 2012-01-21 23:40
On the OCM Guenther asked me to make a list of what is not optimal about the current effect implementation.

- the "target" parameter
..is unneeded. The only parameter needed is "effect" with the target as a property. Maybe same for "time" (in Fx*Timer)?

- temporary callbacks / Initialize / Destruction
You probably see the line if(temp) return; more than Start-calls without it.
Effects should have additional callbacks Fx*Initialize and Fx*Destruction that are meant to f.e. initialize variables. Those callbacks are obviously only called once on creation/removal - not on temporary Start/Stop

- all callbacks should automatically be forwarded to all effects on the object

- too few seem to use the return-value constants for Fx*Timer
Including me... why? Are they too hard to remember? In the docs it even says If this function is not implemented or returns -1, the effect will be deleted after this call.

Something else that I needed once:
- when is Fx*Stop called? After the items are dropped but before the Death()-callback in the object?
There is no possibility to easily f.e. remove the items of a Clonk when he dies(?)

- constants for some effect priorities (minor)

Anything else?
Parent - - By Sven2 [de] Date 2012-01-22 13:32
I agree with all, except:

> - all callbacks should automatically be forwarded to all effects on the object


Didn't Guenther implement a more generic way to hook into object functions, like e.g. FindObject(Flint).Hit = MyEffect.Hit;? This would also work for callbacks from script, like OnShockwaveBlast, etc.
Parent - - By Matthias [de] Date 2012-01-22 17:21
How do you hook multiple effects into an object function, then?
Reply
Parent - By Günther [de] Date 2012-01-25 01:11 Edited 2012-01-25 01:14
static const HitEffect = {
NextHit = nil;
func Hit () {
  // this is the object
  Awesome();
  var fn = this.Hit;
  this.Hit = GetEffect(HitEffect).NextHit;
  var r = this->Hit();
  this.Hit = fn;
  return r;
}
func Start(obj) {
  // this is the effect
  if (obj->GetEffect(HitEffect))
    return obj->RemoveEffect(this);
  NextHit = obj.Hit;
  obj.Hit = Hit;
}
func Stop(obj) {
  // this is the effect
  obj.Hit = NextHit;
}
}

One could probably imagine some library solution that would reduce the boilerplate a bit, but having the engine do the job would be easier. And would work with multiple instances of an effect on the same object. (Though the Real Solution(tm) to that would be closures. With closures one could also reduce the boilerplate to a single function call, at the cost of one wrapper function per callback. Not really worth it...)
Reply
Parent - By Günther [de] Date 2012-01-25 01:00
Thanks. :-) It'll be a while before I get to this, though.
Reply
- - By Günther [de] Date 2012-02-14 01:41
And I'm almost done! I've put the branch into a new repository, so that anybody can pull it without getting stale extra commits. (Though be careful not to push the commits to openclonk.org yet.)

I'll add some documentation next. Until then, here's some silly test-scenarioscript, and the commits with "Script:" in the title are the ones that change the C4Script language or API.

func InitializePlayer(i)
{
  var p1 = { Name="p1" };
  p1.Bar = this.Foo;
  p1->Bar();
  local p2;
  p2 = { Name="p2" };
  p2.Bar = this.Foo;
  p2->Bar();
  Foo();
}

func Foo() {
  Log("Foo: %v", this);
}
Reply
Parent - - By Zapper [de] Date 2012-02-14 09:46
You have a local variable in your function there, mate!
Parent - - By Günther [de] Date 2012-02-14 15:42
Yes, and it just works.
Reply
Parent - - By Zapper [de] Date 2012-02-14 17:32
must be dark magic /o/
Parent - By Günther [de] Date 2012-02-14 20:43
Nah, it works just as if it had been declared outside of the function.
Reply
Parent - By Caesar [de] Date 2012-02-15 13:16
Yes.

func InitializePlayer(i)
{
  p1 = { Name="p1" };
  p1.Bar = this.Foo;
  p1->Bar();
  p2 = { Name="p2" };
  p2.Bar = this.Foo;
  p2->Bar();
  Foo();
  var p1;
  local p2;
}
Parent - By Mortimer [de] Date 2012-02-15 11:22
oh, I guess I should add Eclipse support for that.
Reply
Parent - By Günther [de] Date 2012-02-17 01:12
Thanks to JCaesar for pointing out that c4script standalone program was broken. I've pushed some fixes for that.
Reply
- By Günther [de] Date 2012-03-09 19:23 Edited 2012-03-09 23:16
And merged! This commit has a little bit of documentation.

Next steps:
- Calling bare functions. Probably simply like this: func foo(function bar) { bar(); }
- Declaring functions inside proplist literals. Probably only in proplist literals that initialize local or static const variables, to make the savegame code simpler.
- Creating effects with a prototype, and make effect callbacks work like "effect->Foo()". Combined with the above, this should reduce the boilerplate involved with writing effects considerably.
- Make all the things that accept function names also accept function pointers.
Reply
- - By Marky [de] Date 2017-06-21 13:03
Really late post, but I had the following issue several times lately: I developped a new functionality in a separate definition foo in my local scenario. Whenever I want to add this to an existing definition I cannot do it without getting at least a warning:

In Foo:
#appendto Bar

=> will produce an error: #appendto in definition

In System.ocg\AppendToBar.c:
#include Foo
#appendto Bar


=> will produce a warning #include in #appendto

Is there an official way to do this without a warning? Alternatively, I would be glad if I could just overload the definition somehow without copying it. For example, the possibility to just redefine the whole script part of a definition without the need for having a DefCore, graphics, etc. This is already partially possible by replacing functions from another definition, but it is a very dirty method, imo.
Parent - - By Luchs Date 2017-06-21 20:26
You can just ignore the warning if you know what you're doing. I just added that to make people aware that it probably doesn't work like they would expect.
Up Topic Development / Developer's Corner / Function pointers
1 2 Previous Next

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill