![[de]](/mwf/flags/de.png "Germany")
Another thing. I got no idea how the c4u update system works. But we should definitely have it up and running for the release or at least have these values in the C4Config.cpp adjusted to where the updates will be: pComp->Value(mkNamingAdapt(s(UpdateEngine), "UpdateEngine", "www.clonkx.de/" C4ENGINENICK "/cr_%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(UpdateObjects), "UpdateObjects", "www.clonkx.de/" C4ENGINENICK "/cr_%d%d%d%d_%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(UpdateMajor), "UpdateMajor", "www.clonkx.de/" C4ENGINENICK "/cr_%d%d%d%d_%s.c4u"));
pComp->Value(mkNamingAdapt(s(PuncherAddress), "PuncherAddress", "clonk.de:11115"));(Why is the PuncherAddress still clonk.de btw? Isn't it part of the network code?)
So, question: Who is the expert in the update system? Also: Who could do it if that person were away?
![[gb]](/mwf/flags/gb.png "United Kingdom")
Well, in principle you create a c4u via C4Group and name it following the way you posted. The engine then contacts the Masterserver to get the current version. If it differs from the current engine Clonk offers to download the new version and uses it just as it would use any other c4u, updating all changed files. And that is it really, though I haven't included the version line into the current C4Masterserver yet.And no, I have never actually looked at the source, I just know the basics.
I think I know much less about the whole update infrastructure than you think I do.First of all, what does the masterserver have to do with the current version? Did I understand correctly:
The masterserver is asked for the current version number by the client, which in turn first checks the update-directory if there is a newer version available (or reads some curr_version.txt) and then gives the newest version number to the client. Then, if the clonk client doesn't have the most current version, it itself puts the current version number in the query string (and because of that we have these %d%d%d in the config for the update server) and executes (=downloads) it. The downloaded c4u will then be executed. Is this correct?
But what C4U files do have to be on the server exactly (What is UpdateEngine, UpdateObjects, UpdateMajor)? If I got the version 4.10.0.1 [023] and the newest is 4.10.1.2 [001], what package(s) will exactly be queried? Is it all the same from which version one client comes when it comes to updating to a version? In many games, the immediate updates need to be executed first before updating to the last. At least, many updates are specificly for "from version X to version Y". How is it in Clonk?
Also, on the creation of c4u packages, how does this work? (Not technically, but how to create it and from what to create it)
(And what is this puncheraddress?)
The basic idea is that the engine knows it's version, reads the current version from the master server response, and computes the update download link from that information and one of the URL templates in the configuration, depending on the version difference. Some upgrades need intermediate upgrades, some just the latest cumulative one. When the user clicks yes, the update is downloaded, the c4group executable extracted from the update package and executed to do the update.More details are easiest found in the redwolf design repository in the tools directory, if you have access to that. I think some of the scripts have longer comments.
C4U files are created from an exact copy of the old files to update from and the desired result.
Puncheraddress is used to punch a hole trough NAT with UDP.
> Some upgrades need intermediate upgrades, some just the latest cumulative one.
And how is that determined?
The version number difference. Sorry, I don't have the exact details memorized, but I think it has something to do with which version number has increased.
Do you know who has? We need someone to take care of this stuff or disable the update completely. And this is IMO very undesireable.
// Objects major update: we will update to the first minor of the next major version - we can not skip major versions or jump directly to a higher minor of the next major version.
if (rUpdateVersion.iVer[2] > C4XVER3)
strUpdateURL.Format(Config.General.UpdateMajor, (int)rUpdateVersion.iVer[0], (int)rUpdateVersion.iVer[1], C4XVER3 + 1, 0, C4_OS);
// Objects version match: engine update only
else if ((rUpdateVersion.iVer[2] == C4XVER3) && (rUpdateVersion.iVer[3] == C4XVER4))
strUpdateURL.Format(Config.General.UpdateEngine, (int)rUpdateVersion.iBuild, C4_OS);
// Objects version mismatch: full objects update
else
strUpdateURL.Format(Config.General.UpdateObjects, (int)rUpdateVersion.iVer[0], (int)rUpdateVersion.iVer[1], (int)rUpdateVersion.iVer[2], (int)rUpdateVersion.iVer[3], (int)rUpdateVersion.iBuild, C4_OS);I think the three URLs are due to an optimization regarding the engine. The engine is always included in full, and thus "cumulative" to all older engine versions. But cumulative object updates grow bigger the more older versions have to be supported, so every few versions a cutoff is made: The update to x.0 applies to all (x-1).y, and updates to x.y1 apply to all x.y0 where 0<y0<y1. Example:
1.0 [71]
1.1 [72]
1.1 [73]
2.0 [74]
2.0 [75]
So if you have 71, you can update to 72 and 73 with an "object" update, to 74 with a major update, and not to 75.
If you have 72, you can update to 73 with an engine update, to 74 with a major update, and not to 75.
If you have 73, you can update to 74 with a major update.
If you have 74, you can update to 75 with an engine update.
So I'm not sure whether we should keep this scheme: The engine update optimization depends on engine updates being more frequent than object updates, but this isn't the case with OC at the moment, and I think that's a good thing. We might even want to look into pure object updates, which can be a lot smaller.
Generally, this is all a tradeoff between the number of update packages a player has to download, the download size, the number of update packages we have to produce, and the number of old versions we have to keep around. We could optimize for the first two by providing optimal update packages for every combination of old and new versions, or for the last two by providing only updates from one version to the immediately next one. But that has an additional downside: In case we produce a broken update, we can't supersede it by providing a cumulative fixed one that allows players to skip the broken one. So some form of cumulative updates should be provided.
I think we might want to move the "which update package to download" logic into the masterserver. That way, we can decide at update creation time how the packages should look like, and not when producing the old version.
To optimize for the last one is probably the better way for OC because it hopefully means less maintenance for us.I think you have a point regarding the masterserver. To have the logic in the masterserver is also better because this way, we are able to change the update infrastructure centrally, not individually in every client. The update request would look as easy as: Client sends version information, masterserver replies the link to the destined update package.
Of course, this means that some more initial work has to be done. Thats why I suggest that the update system will be kept very simple in the first versions:
client sends version information, masterserver always points to the newest complete installation. That saves the effort to implement any logic in the masterserver.
Well, I'd definitely want to be able to recall broken versions. Experience tells us that the update mechanism will get broken by a change eventually.The maintenance burden is minimal, the main cost is time and space on the computer producing the updates. To create a cumulative update, you simply have to run c4group for every supported old version with the same update file name. That's easily automatable. (We should probably look into changing this so that one can create an update with source A and B and target C into an update with source A, B and C and target D. That way we wouldn't need to keep the old versions around.) The bigger problem with updates is that you have to compile the engines and move them to the computer producing the update, or move the packed object files to the computer compiling the engine. That's some coordination work. I think matthes manually started the scripts compiling the engine on windows and mac computers, and then the script producing the updates on a dedicated linux server.
So we already have people writing scripts to automatically compile and publish the game. Anyone of you willing to extend that to create .c4us?
> Well, I'd definitely want to be able to recall broken versions. Experience tells us that the update mechanism will get broken by a change eventually.
If always the newest build is loaded, this is fulfilled, right? A more intelligent update structure can be designed in the masterserver code after the initial change (without pressure of time because of an upcoming release) that takes this into account.
> Rest
Yeah, that sounds pretty intelligent/advanced. I want to stress that I am of the opinion that the development of such automatic scripts/update logic in the masterserver should definitely come after the change from update logic in the engine -> update logic in the masterserver and not be done at the same time. The update system is a critical system which should 100% work for every immediate development step.
Regarding the issue that the compiled engine/content needs to be moved to the computer producing the update: Why can't the script that outputs the different update packages not be on the same machine as it was compiled? I mean, these would probably only be some batch/bash scripts calling c4group.
> So we already have people writing scripts to automatically compile and publish the game
Do we? You mean CKs(+Isilkors?) nightly builds?
> If always the newest build is loaded, this is fulfilled, right?
Yes. But didn't you say that we shouldn't try to create cumulative updates? Forcing someone who skipped one update to apply every intermediate update is bad enough, we shouldn't break the update mechanism as well.
> Why can't the script that outputs the different update packages not be on the same machine as it was compiled? I mean, these would probably only be some batch/bash scripts calling c4group.
Because two computers packing the same data usually get different c4groups because c4groups contain timestamps. Having the game data bit-for-bit identical between platforms is desirable, even if it isn't required for network play. (I'm not sure whether it is.)
> Do we? You mean CKs(+Isilkors?) nightly builds?
Yes.
> Yes.
I could try to do that. Should this happen whenever the Version.txt is updated?
Heh, that'd be even more automation than CR had. Progress!I think the steps are these:
mkdir Update.c4u
touch Update.c4u/AutoUpdate.txt
c4group Update.c4u/Objects.c4u -g old/Objects.c4d new/Objects.c4d
c4group Update.c4u/System.c4u -g old/System.c4g new/System.c4g
cp new/clonk Update.c4u
c4group Update.c4u -p
Hm, maybe dreaming a bit here, but we can actually consider doing automatic releases this way: For each commit that updates the version file the build server could build at least Windows and Linux, create an update from the previous version(s), create an installer, source tarball, maybe also a Linux binary tarball, tag the release... :)
What if you want to change something else in the version file without doing a release? o_O
We just have to reduce the version file to contain only the version and nothing else. You know, like version did.
We could still add a check which makes sure that the version number actually increased.
OK, I have set up automatic updates for the two platforms I have a build setup for currently (win32, win64). They get generated every time the version in the
version file is increased. This is currently done in the nightly build cycle (that is, once a day) but we could add a trigger to do this right after each push to hg.openclonk.org. The results (updates and full packages) are currently available at http://londeroth.org/~ck/oc/. Is there a way to get these uploaded to the master server automatically?
I can code the option to get the c4u-Paths from a file into C4Masterserver. I'd need the request-structure though and how to answer. I just guess client sends Action=Update and the masterserver answers Location=example.com/OC014.c4u or something?
Anyway, does it still make sense to use c4us after we abandonded groups? Zips might be preferable...
I man last time I looked everything else was not in the c4-Groups anymore and I didn't really follow that discussion, so I assumed they were abandonded.
As long as there is no alternative implemented we'll stay with C4Group, for better or worse.
Regarding the interfaces between engine, buildscripts and server: how about the masterserver publishes the currently available version as it did for CR, and the engine simply compares that version with the engine version, but the download URL the engine uses to fetch an update is based on the old version? That way, the download server could either serve an individual update package for every supported old engine, serve the same cumulative update package under multiple names, respond with a 410 gone for too old engines when we run out of space, or continue to serve old update packages to old engines, requiring multiple update cycles for them. And we wouldn't have to decide on what we want to do until we actually produce the update, nor write complicated code in the masterserver or the engine.
Okay, I wrote a bit more about this in the wiki, based on a spec from Newtons for a different design. From the discussion paragraph:This proposal is intentionally kept as simple as possible: The different pieces of code only need to communicate one version number to each other, apart from the build server uploading update packages and full packages. This is a little wasteful of disc space and bandwidth, because one update package for every old supported version needs to be uploaded and stored. If this is a problem, the build server could create one cumulative update package and copy that locally on the download server. Or we could pass around update URLs from the build script to the masterserver to the engine, so that different engine versions can download the same update URL. Newton and ck already wrote a spec for that that can be used if necessary, but maybe this simpler version suffices.
Attachment: commit-1a16c5a - engine changes (5k)
>a spec from Newtons
This is the specification from me and CK.
As said in the chat, I have no motivation to discuss new concepts any further. I am content with any spec that covers the requirements that were taken into account in the linked spec.
Attachment: masterserver.pdf (131k)
> (The two different architectures 32bit and 64bit are both included in the update package.)
I don't think that's a good way to go. This basically means that also our installer would need to install both versions so that the update can be applied. Apart from the apparent question on where to install it by default ("C:\Program Files" or "C:\Program Files (x86)"?) this would also require to ship all dependency DLLs in both 32 and 64 bit, and it would require to rename some of the DLLs.
Yeah, for windows it might not be that great an idea. It worked well for CR on Linux, though. Actually, do we need 64 bit windows installers? I thought every 64 bit windows version contained the necessary libraries and kernel support to run 32 bit executables, and it's not as if Clonk is a highly demanding application that needs the big address space or the extra registers.
Some newish 64 bit Server Core doesn't bring a 32 bit compatibility layer anymore, although you can install it afterwards.
Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill
