Not logged inOpenClonk Forum
Up Topic Development / Developer's Corner / CCAN 2.0: package management
- - By Zapper [de] Date 2017-06-22 18:42
Assuming we will at some point have built-in downloading of mods from some sort of CCAN successor.

I am thinking about how this "package management" should work in practice. I want to be able to download and update stuff using the game UI, but I also want to be able to update/manage stuff that the player put into the game data folder themselves.
I think one "mod" can contain multiple files (e.g. Caedes.ocf and Caedes.ocg). How would one find out which files in the game data folder belong to which mod? How to show the user which mods are installed?

If I used some sort of database to manage installed mods, I could not fluently manage manually added (e.g. downloaded from network games) packages.

And yes, those questions are the current blocker in my implementation of the ingame client for Kanibal's Larry.
Parent - - By Maikel Date 2017-06-22 19:06

> And yes, those questions are the current blocker in my implementation of the ingame client for Kanibal's Larry.


No new pictures :(

> Rest of story...


I don't understand what is exactly the problem. What is wrong with putting everything in the appdata folder and then either updating in your UI or when you join a game that hosts a newer version (i.e. you get a query to update or to not join). All stuff that is not on Larry could be added to the appdata folder by hand.

I think it is essential that people use Version.txt in their .ocd's .ocf's etc. so that the engine can determine whether an update is there.
Parent - - By Zapper [de] Date 2017-06-22 21:15
Well, imagine I have the files A.ocd, B.ocf and C.ocs in my game data folder. How do I know that A and B belong to mod X? Should I download information about all mods and try all combinations of filenames to check if that combination possibly belongs to a mod?
Should I use the filenames? Or a checksum?

If I find matching filenames, how do I know which version of the mod they belong to? And ideally all that should be pretty fast

PS:

>No new pictures :(


That's because I didn't do anything yet!
Parent - By Marky [de] Date 2017-06-22 21:39

>Well, imagine I have the files A.ocd, B.ocf and C.ocs in my game data folder. How do I know that A and B belong to mod X?


I don't know what this information is necessary for exactly. Update mod X? The information could be read from Scenario.txt: Everything that is part of Scenario.txt in C.ocs if obviously part of that mod, somehow.

A problem that I can imagine is the following: B.ocf needs version 1 of A.ocd, but C.ocs needs version 2 of A.ocd. Which one do you download?
Parent - By Marky [de] Date 2017-06-22 21:35

>I think it is essential that people use Version.txt in their .ocd's .ocf's etc. so that the engine can determine whether an update is there.


I am already glad that I started versioning stuff :)
Parent - By K-Pone [de] Date 2017-06-22 22:16
I think there's almost no way to properly check scenarios saved from network games except when they were not put in a scenario folder. Single scenarios which are not contained in a scenario folder might be checked via MD5/CRC32/whatever checksum and/or Version.txt info. Those inside a scenario folder are saved into the root directory of the OC appdata folder when saving them from the lobby.

In case of mods downloaded from Larry/CCAN 2.0/... I could think of letting clients download from that platform instead of using the current way of downloading and saving scenarios. That would at least ensure clients have the same version and if clients need to install a mod they could do this the same way as they would normally do when installing mods.

Maybe I better give an example for that:
Lets assume the host is hosting "DM_Knueppeln.c4s.ocs" from Knueppeln 0.8 and it has been installed from the integrated mod list. So the masterserver will know the host uses Knueppeln 0.8 from the mod list and clients will be able to see this and check whether Knueppeln 0.8 is installed or not. Assuming now it's not installed, any client which still needs it can download and install Knueppeln 0.8 from Larry (let's use Larry as the successor to CCAN in this case). Having downloaded and installed the Knueppeln package, this can be noted by the client in its package management.

Now let's do another scenario, but this time the scenario is not from any of the mods from the mods list. In this case the scenario and all the stuff it requires will be shared between clients normally (the 'classic' way). This might also be an alternative in case of Larry being offline for whatever reason.

Seing this situation I would probably tend to make a subfolder in the OC appdata folder containing version and package management for all the mods downloaded from Larry. Could look like this:

OpenClonk
|-Mods
| |-Knueppeln
| | |-0.8 stable
| | | |-Knueppeln.c4d.ocd
| | | \-Knueppeln.c4f.ocf
| | \-0.9 beta
| |   |-Knueppeln.c4d.ocd
| |   \-Knueppeln.c4f.ocf
| \-Tower of despair
|   \-0.5 stable
|     \-OCTowerV0.5.ocs
...


That's just the way how I would solve the versioning problem. This would at least allow the players to have multiple versions of a package installed without having conflicts on a mod being installed/present in a wrong version. Though it might need the mod developers to update dependencies on their scenarios.

For those mods which were manually added (either by network game saving or by manually putting the files to the OC appdata folder) I would say they might be shown in the list as well. Maybe they could be marked as manually installed or "unidentifiable", so the player can see this file(s) don't belong to any of the installed mods from the list.

Hope this post didn't get too confusing
Parent - - By Luchs Date 2017-06-22 22:28
I think we should update the c4group format to make it work better with an automated mod manager.
Parent - - By Sven2 Date 2017-06-23 05:57
I don't think we should add dependency information directly into the groups somehow. We can easily just have either an additional dependency file (always sorted to the front of the group just like Version.txt) or extra lines in the Version.txt to mark dependencies. No change to c4group would be needed.

Having this information "hacked" into the group headers makes the c4group interface more complicated. We would also lose the information on unpack+pack. Plus you couldn't keep the information in the version control but would have to pass it through the release pipelines somehow.
Parent - - By Luchs Date 2017-06-25 21:26
I think a better package format for Clonk would have the following features:

1. Fast pack/unpack. Avoid double compression of images and music. (Unpack performance is fine as-is, but there is a noticeable delay when hosting from a development tree.)

2. Transparent delta updates from the lobby. During development, it's very common to iterate quickly on a couple of scripts. Having to wait every time for the unchanged images and music is annoying.

3. Transparent web downloads from the lobby. Avoid the limited upstream from home ISPs when possible.

4. Versioning and dependency information. Version.txt is fine as API, but we'll want to do more with that information. There is some value in a fixed-format header field for external tools compared to unpacking and parsing a free-form text file.

---

I already have some ideas on what a new package format could look like. I'm going to write it down in the next days...

---

Additional notes for 1:

Compared to standard compression tools, c4group is unfortunately very much on the slow side of things:

% time tar caf Objects.ocd.tar.gz Objects.ocd
bsdtar caf Objects.ocd.tar.gz Objects.ocd  1.38s user 0.02s system 99% cpu 1.400 total
% time tar caf Objects.ocd.zip Objects.ocd
bsdtar caf Objects.ocd.zip Objects.ocd  1.31s user 0.04s system 99% cpu 1.354 total
% time tar cf Objects.ocd.tar Objects.ocd
bsdtar cf Objects.ocd.tar Objects.ocd  0.01s user 0.03s system 99% cpu 0.041 total
% time c4group Objects.ocd -p
c4group Objects.ocd -p  5.01s user 0.14s system 99% cpu 5.144 total


Size comparison:

% ls -lS
total 140308
-rw-r--r--  1 lukas lukas 45582848 Jun 25 21:47 Objects.ocd.tar
-rw-r--r--  1 lukas lukas 33014223 Jun 25 21:54 Objects.ocd.zip
-rw-r--r--  1 lukas lukas 32753064 Jun 25 21:47 Objects.ocd
-rw-r--r--  1 lukas lukas 32316248 Jun 25 21:46 Objects.ocd.tar.gz


So c4group is more than four times slower than bsdtar while producing larger files. Compression saves roughly 12 MB.

With these stats, I'd argue that current c4group spends more time (de-)compressing the groups than is saved by having to load less data from disk. (We can still do compression for network transfers. An online archive like Larry could save gzipped archives and serve these directly with Content-Encoding: gzip.)
Parent - - By Sven2 Date 2017-06-26 00:03
I'm pretty sure that for the unpacking, the bottleneck is the way disk IO is handled by c4group (I think it's creating temp files and renaming them).

The more important data point would be how fast it is to load from the tar to memory versus the time to load from c4group to memory, because it determines the startup time for every single scenario. This would be disk IO time plus unpacking time. I'm pretty sure we can find a better format, but I also imagine that we would lose a lot if we can no longer pre-organize everything to be in the correct order for reading.
Parent - - By Zapper [de] Date 2017-06-26 06:44

>I'm pretty sure that for the unpacking, the bottleneck is the way disk IO is handled by c4group (I think it's creating temp files and renaming them).


I got SSDs, so I don't care about disk I/O, and I think in a few years everybody will (only) have SSDs. So I guess we can still optimize for CPU and win a lot in the long run
Parent - By Sven2 Date 2017-06-26 15:40
The two are not mutually exclusive. c4group isn't optimized well for either I/O or CPU usage in general. The only thing that's optimized and which I think should stay is sequential reading when starting a scenario, which I would like to keep. This should be possible with formats like .tar.gz as well.

I imagine we could be a lot faster if e.g. we pre-unpack groups into memory in a separate thread while the game is loading.
Parent - - By Fulgen [at] Date 2017-06-29 21:14

>I think in a few years everybody will (only) have SSDs.


Perhaps if they get less expensive and more space.
Parent - - By Luchs Date 2017-06-29 21:53
You can get 250 GB for less than 100 € - they're really not expensive anymore.
Parent - - By Fulgen [at] Date 2017-06-30 06:27
Hm, but compared to normal hard drives, they are still a lot more expensive. (Maybe I'm not persuaded because I never had one.)
Parent - - By Luchs Date 2017-06-30 08:00
But what are you comparing? Price per GB? The point of SSDs is that they're fast, which is great for software but doesn't matter for most data.

Do you own a USB flash drive? Are you aware they're so much more expensive than hard drives and do not have a lot of space?
Parent - By Fulgen [at] Date 2017-06-30 12:57

>Do you own a USB flash drive? Are you aware they're so much more expensive than hard drives and do not have a lot of space?


I am, but the point of USB flash drives is that they are small and tough, so I can take it wherever I want.
Up Topic Development / Developer's Corner / CCAN 2.0: package management

Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill