pComp->Value(mkNamingAdapt(s(UpdateEngine), "UpdateEngine", "www.clonkx.de/" C4ENGINENICK "/cr_%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(UpdateObjects), "UpdateObjects", "www.clonkx.de/" C4ENGINENICK "/cr_%d%d%d%d_%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(UpdateMajor), "UpdateMajor", "www.clonkx.de/" C4ENGINENICK "/cr_%d%d%d%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(PuncherAddress), "PuncherAddress", "clonk.de:11115"));
(Why is the PuncherAddress still clonk.de btw? Isn't it part of the network code?)
So, question: Who is the expert in the update system? Also: Who could do it if that person were away?
And no, I have never actually looked at the source, I just know the basics.
First of all, what does the masterserver have to do with the current version? Did I understand correctly:
The masterserver is asked for the current version number by the client, which in turn first checks the update-directory if there is a newer version available (or reads some curr_version.txt) and then gives the newest version number to the client. Then, if the clonk client doesn't have the most current version, it itself puts the current version number in the query string (and because of that we have these %d%d%d in the config for the update server) and executes (=downloads) it. The downloaded c4u will then be executed. Is this correct?
But what C4U files do have to be on the server exactly (What is UpdateEngine, UpdateObjects, UpdateMajor)? If I got the version 4.10.0.1 [023] and the newest is 4.10.1.2 [001], what package(s) will exactly be queried? Is it all the same from which version one client comes when it comes to updating to a version? In many games, the immediate updates need to be executed first before updating to the last. At least, many updates are specificly for "from version X to version Y". How is it in Clonk?
Also, on the creation of c4u packages, how does this work? (Not technically, but how to create it and from what to create it)
(And what is this puncheraddress?)
More details are easiest found in the redwolf design repository in the tools directory, if you have access to that. I think some of the scripts have longer comments.
C4U files are created from an exact copy of the old files to update from and the desired result.
Puncheraddress is used to punch a hole trough NAT with UDP.
> Some upgrades need intermediate upgrades, some just the latest cumulative one.
And how is that determined?
// Objects major update: we will update to the first minor of the next major version - we can not skip major versions or jump directly to a higher minor of the next major version.
if (rUpdateVersion.iVer[2] > C4XVER3)
strUpdateURL.Format(Config.General.UpdateMajor, (int)rUpdateVersion.iVer[0], (int)rUpdateVersion.iVer[1], C4XVER3 + 1, 0, C4_OS);
// Objects version match: engine update only
else if ((rUpdateVersion.iVer[2] == C4XVER3) && (rUpdateVersion.iVer[3] == C4XVER4))
strUpdateURL.Format(Config.General.UpdateEngine, (int)rUpdateVersion.iBuild, C4_OS);
// Objects version mismatch: full objects update
else
strUpdateURL.Format(Config.General.UpdateObjects, (int)rUpdateVersion.iVer[0], (int)rUpdateVersion.iVer[1], (int)rUpdateVersion.iVer[2], (int)rUpdateVersion.iVer[3], (int)rUpdateVersion.iBuild, C4_OS);
I think the three URLs are due to an optimization regarding the engine. The engine is always included in full, and thus "cumulative" to all older engine versions. But cumulative object updates grow bigger the more older versions have to be supported, so every few versions a cutoff is made: The update to x.0 applies to all (x-1).y, and updates to x.y1 apply to all x.y0 where 0<y0<y1. Example:
1.0 [71]
1.1 [72]
1.1 [73]
2.0 [74]
2.0 [75]
So if you have 71, you can update to 72 and 73 with an "object" update, to 74 with a major update, and not to 75.
If you have 72, you can update to 73 with an engine update, to 74 with a major update, and not to 75.
If you have 73, you can update to 74 with a major update.
If you have 74, you can update to 75 with an engine update.
So I'm not sure whether we should keep this scheme: The engine update optimization depends on engine updates being more frequent than object updates, but this isn't the case with OC at the moment, and I think that's a good thing. We might even want to look into pure object updates, which can be a lot smaller.
Generally, this is all a tradeoff between the number of update packages a player has to download, the download size, the number of update packages we have to produce, and the number of old versions we have to keep around. We could optimize for the first two by providing optimal update packages for every combination of old and new versions, or for the last two by providing only updates from one version to the immediately next one. But that has an additional downside: In case we produce a broken update, we can't supersede it by providing a cumulative fixed one that allows players to skip the broken one. So some form of cumulative updates should be provided.
I think we might want to move the "which update package to download" logic into the masterserver. That way, we can decide at update creation time how the packages should look like, and not when producing the old version.
I think you have a point regarding the masterserver. To have the logic in the masterserver is also better because this way, we are able to change the update infrastructure centrally, not individually in every client. The update request would look as easy as: Client sends version information, masterserver replies the link to the destined update package.
Of course, this means that some more initial work has to be done. Thats why I suggest that the update system will be kept very simple in the first versions:
client sends version information, masterserver always points to the newest complete installation. That saves the effort to implement any logic in the masterserver.
The maintenance burden is minimal, the main cost is time and space on the computer producing the updates. To create a cumulative update, you simply have to run c4group for every supported old version with the same update file name. That's easily automatable. (We should probably look into changing this so that one can create an update with source A and B and target C into an update with source A, B and C and target D. That way we wouldn't need to keep the old versions around.) The bigger problem with updates is that you have to compile the engines and move them to the computer producing the update, or move the packed object files to the computer compiling the engine. That's some coordination work. I think matthes manually started the scripts compiling the engine on windows and mac computers, and then the script producing the updates on a dedicated linux server.
So we already have people writing scripts to automatically compile and publish the game. Anyone of you willing to extend that to create .c4us?
> Well, I'd definitely want to be able to recall broken versions. Experience tells us that the update mechanism will get broken by a change eventually.
If always the newest build is loaded, this is fulfilled, right? A more intelligent update structure can be designed in the masterserver code after the initial change (without pressure of time because of an upcoming release) that takes this into account.
> Rest
Yeah, that sounds pretty intelligent/advanced. I want to stress that I am of the opinion that the development of such automatic scripts/update logic in the masterserver should definitely come after the change from update logic in the engine -> update logic in the masterserver and not be done at the same time. The update system is a critical system which should 100% work for every immediate development step.
Regarding the issue that the compiled engine/content needs to be moved to the computer producing the update: Why can't the script that outputs the different update packages not be on the same machine as it was compiled? I mean, these would probably only be some batch/bash scripts calling c4group.
> So we already have people writing scripts to automatically compile and publish the game
Do we? You mean CKs(+Isilkors?) nightly builds?
> If always the newest build is loaded, this is fulfilled, right?
Yes. But didn't you say that we shouldn't try to create cumulative updates? Forcing someone who skipped one update to apply every intermediate update is bad enough, we shouldn't break the update mechanism as well.
> Why can't the script that outputs the different update packages not be on the same machine as it was compiled? I mean, these would probably only be some batch/bash scripts calling c4group.
Because two computers packing the same data usually get different c4groups because c4groups contain timestamps. Having the game data bit-for-bit identical between platforms is desirable, even if it isn't required for network play. (I'm not sure whether it is.)
> Do we? You mean CKs(+Isilkors?) nightly builds?
Yes.
> Yes.
I could try to do that. Should this happen whenever the Version.txt is updated?
I think the steps are these:
mkdir Update.c4u
touch Update.c4u/AutoUpdate.txt
c4group Update.c4u/Objects.c4u -g old/Objects.c4d new/Objects.c4d
c4group Update.c4u/System.c4u -g old/System.c4g new/System.c4g
cp new/clonk Update.c4u
c4group Update.c4u -p
version
did.
version
file is increased. This is currently done in the nightly build cycle (that is, once a day) but we could add a trigger to do this right after each push to hg.openclonk.org. The results (updates and full packages) are currently available at http://londeroth.org/~ck/oc/. Is there a way to get these uploaded to the master server automatically?
This proposal is intentionally kept as simple as possible: The different pieces of code only need to communicate one version number to each other, apart from the build server uploading update packages and full packages. This is a little wasteful of disc space and bandwidth, because one update package for every old supported version needs to be uploaded and stored. If this is a problem, the build server could create one cumulative update package and copy that locally on the download server. Or we could pass around update URLs from the build script to the masterserver to the engine, so that different engine versions can download the same update URL. Newton and ck already wrote a spec for that that can be used if necessary, but maybe this simpler version suffices.
>a spec from Newtons
This is the specification from me and CK.
As said in the chat, I have no motivation to discuss new concepts any further. I am content with any spec that covers the requirements that were taken into account in the linked spec.
> (The two different architectures 32bit and 64bit are both included in the update package.)
I don't think that's a good way to go. This basically means that also our installer would need to install both versions so that the update can be applied. Apart from the apparent question on where to install it by default ("C:\Program Files" or "C:\Program Files (x86)"?) this would also require to ship all dependency DLLs in both 32 and 64 bit, and it would require to rename some of the DLLs.
Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill