![[de]](/mwf/flags/de.png "Germany")
You can check whether you're using an out-of-tree build by starting cmake-gui and comparing the two directories at the top. If they're the same, you're not and not affected by this. If they are not the same, you need to create a symlink from the build directory named planet pointing to the planet directory in the source directory. For example, if you have $HOME/Clonk as the source directory and $HOME/Clonk/build ans the build directory, create $HOME/Clonk/build/planet->$HOME/Clonk/planet. If you had it, the old symlink from planet/clonk to build/clonk can be deleted. We probably should automatically create this symlink, but as usual the real problem is Windows. Is there anybody who's using multiple out-of-tree builds and can't create that symlink and can't switch to an in-tree build? In that case, I'll bring the BUILD_TO_PLANET option back, probably renamed to BUILD_TO_SOURCE.
This especially applies to those who put the CMakeCache.txt into planet in the source directory. While this will continue to work on windows with compilers that put the executable into the same directory as the CMakeCache.txt, it won't wotk with MSVC. The easiest solution is to recreate the CMakeCache.txt in the toplevel source directory, where the CMakeLists.txt is.
Wait, wait, wait. 90% of the windows developers DO have their build directory in planet.For me the question is: Why do you want to push a change to default that generates problems/work for anyone using MSVC? As long as the is no big gain for this, I don't see a reason for this change.
It is already pushed, and I guess less than 10% of windows devs use an out of tree build. The others can nearly go as usual, except having to start ./RelWithDebInfo/Clonk.exe instead of ./planet/ClonkRD.exe when launching manually. Someone who used out of tree builds with a multiconfig IDE like MSVC wasn't really able to use BUILD_TO_PLANET because the files would have overwritten themselves in planet anyway. For those, a simple symplink from ${CMAKE_BINARY_DIR}/planet to ${CMAKE_SOURCE_DIR}/planet suffices now.
> It is already pushed, and I guess less than 10% of windows devs use an out of tree build
if you followed the tutorial in the wiki, you will have an out of tree build - nearly everyone used the tutorial to set up his environment.
> like MSVC wasn't really able to use BUILD_TO_PLANET because the files would have overwritten themselves in planet anyway
I used it for years.
Also, symlink and windows?
![[gb]](/mwf/flags/gb.png "United Kingdom")
Hm, that requires administrator privileges. If it didn't, I could probably automate the process of creating the symlink. Any way to get around that?
Ah, I'll try that. Is a junction something more or less magical than a symbolic link? The documentation I can find on it isn't particularly clear... Only notes that symbolic links are more flexible.
Symlinks work on network shares, junctions don't, and you also can't junction single files (although you can use hard links if they're on the same volume).
Hm, having this kind of link in "planet" chokes TortoiseHg, as it attempts to follow it infinetely. I'd vote for it to be put into the build directory anyway, this CMAKE_INTDIR-thing looks hacky. TortoiseHg doesn't seem to mind junctions there either.
Also, isn't it about time that we stop calling it "planet"? Maybe "game" or "data"?
Also, isn't it about time that we stop calling it "planet"? Maybe "game" or "data"?
Let DWCC be the directory the CMakeCache.txt is in. The CMAKE_INTDIR is there to compensate for the fact that some CMake targets put the binary in DWCC/Debug/, DWCC/Release/, etc., and some in DWCC/. But I think MSVC may make the working directory be DWCC when starting from the debugger instead of DWCC/Debug or something, since I got some str.The setup I'm using is a symlink from DWCC/planet to planet/ in the source directory, and I'm starting the engine with ./clonk planet/Tests.ocf/Minimal.ocs or just ./clonk. Also, for occasional tests with MSVC, I simply made the DWCC be the repository directory. You don't need multiple DWCC for release and debug builds with MSVC, and the average casual windows developer won't need them for crosscompiling or other experiments.
I get what it tries to do. But I think that's the wrong way to approach it - do we really want to ship binaries inside a "$DIR\Release" directory? The good thing about symlinks is that we can get pretty close to what the binary sees in an actual installation. I think we should try that on Windows as well. Therefore, I'm arguing that it's the right thing to write the symlink as "$DIR\Release\planet". Once we have a script for it there's no reason to prefer the one over the other.
Plus, as I said, it also neatly works around TortoiseHg crashing, which it always does as long as the build directory happens to be a directory Mercurial looks at.
Plus, as I said, it also neatly works around TortoiseHg crashing, which it always does as long as the build directory happens to be a directory Mercurial looks at.
Not at all, the installed binary gets its data files from Config.General.SystemDataPath, which is the exepath on windows, a hardcoded path on linux (which probably needs to change for our binary tarball releases), and some magic bundle directory on macosx. The #ifndef __APPLE__ part of C4Reloc::Init is just to make case of CMakeCache.txt and CMakeLists.txt in the same directory and executing the binary without installing it work out of the box. No more, no less. How you set up your directory structure with a so-called out-of-tree build to make the binary find the files is another matter. I think I made it work on every non-apple platform when the directory structure is like this:/path/to/build/dir/CMakeCache.txt
/path/to/build/dir/planet -> /path/to/repos/planet
/path/to/repos/CMakeLists.txt
And either
/path/to/build/dir/clonk$(EXEEXT)
or
/path/to/build/dir/$(CONFIGURATION)/clonk$(EXEEXT)
But I haven't actually tested that on windows. And it doesn't involve any symlinks in the repository, so TortoiseHG doesn't have any reason at all to complain.
Well, okay, I'm fine with that as well. I have now comitted how I intended this to work - this will look at both the executable parent path as well as a "data" subdirectory, but only to check whether there's a full installation there. Otherwise it just ignores the directory. The nice thing about that is that it means we can get rid of the #ifdef for Macs.
I don't think that works on Windows XP. You also forgot to add the data file to .hgignore, but I think we should instead simply not use symlinks in the CMAKE_CURRENT_SOURCE_DIR==CMAKE_CURRENT_BINARY_DIR case. While it might work, why do we need two names for the same thing?
Well, the problem is that strictly speaking, we need to check CMAKE_CURRENT_SOURCE_DIR==CMAKE_CURRENT_BINARY_DIR/CMAKE_INTDIR. And there's no way to do that within CMake (especially as CMAKE_INTDIR actually expands to a variable that gets resolved by MSVC).
Really? Did you check why the code in C4Reloc which took CMAKE_INTDIR into account didn't work?Anyway, your patch breaks crosscompilation and presumably building with Windows XP. While I don't particularly care about the latter, I don't think we should invest too much time into making this work reliably. While out-of-tree builds are nice, in-tree builds work fine for most people and can be made to work without symlinks.
> Did you check why the code in C4Reloc which took CMAKE_INTDIR into account didn't work?
Didn't it work? I didn't check. I just think that no matter whether it worked or not, hard-coding properties of the build system into the binary is the wrong way to do it. Sooner or later we might get terribly confused by the MinGW build behaving slightly differently from the MSVC build.
> and presumably building with Windows XP.
Hm, that's trickier. Maybe we should simulate that by making "data" a text file that points to the correct location or something? Wouldn't be too hard.
> Anyway, your patch breaks crosscompilation
Hardly surprising - it is a pretty radical change. Any pointers on what goes wrong? Does it try to "mklink" under Linux or something?
> I just think that no matter whether it worked or not, hard-coding properties of the build system into the binary is the wrong way to do it. Sooner or later we might get terribly confused by the MinGW build behaving slightly differently from the MSVC build.
That's already the case, and it is very obvious that a change of the relative directory needs to result in a change somewhere. If you aren't confused that the msvc build puts your symlink into a different place you aren't going to be surprised that the binary looks in a different place, too. Remember, the installed binary finds the data files the same way, so we don't need to remember which compiler built the release binaries. If this is the only reason for your change, I'm for simply reverting it.
> Hardly surprising - it is a pretty radical change. Any pointers on what goes wrong? Does it try to "mklink" under Linux or something?
Yes, see the linked log.
So the idea is to settle for "Out-of-tree builds aren't supported, period"? That seems like a unjustified backwards step to me. Wasn't the whole original idea to sort this out?
I probably just react badly to yet more changes in this area just after I finally put in the time to do something about the problem of a simple "cmake . && make && ./clonk" (or the equivalent with MSVC) not working out-of-the-box. Out-of-tree builds required either BUILD_TO_PLANET or the build directory being planet, which only worked for one build, or symlinks in planet to the various build trees. So the only problem I introduced was that people using one out-of-tree build directory with one of the former methods had to change to in-tree-builds or start using symlinks, while people using multiple out-of-tree builds have to use symlinks like before. This is neither a step backwards nor unjustified.I don't have much against creating the symlink from cmake, but that should be a simple optional step. You only need it when you want to start the game without installing it first.
Well, the symlink solution has major problems, that's what I'm saying. So I feel we should either say "in-tree only" or make sure that it works properly. Which - for me - means that newbie engine builder won't have to make those junctions himself.
So I guess we could make that a CMAKE variable that you can activate, like BUILD_TO_PLANET? I could just parameterize the whole behaviour - with possibly different defaults per platform - then everybody's happy on some level.
So I guess we could make that a CMAKE variable that you can activate, like BUILD_TO_PLANET? I could just parameterize the whole behaviour - with possibly different defaults per platform - then everybody's happy on some level.
Just keeping track of the combination of all operating systems, compilers, crosscompilers, in- and out-of-tree builds makes any change here almost impossible to get right, so adding build options into the mix won't really help. The platforms already use different paths for the SystemDataPath, but the repository layout as well as well as the two cmake build tree layouts are the same on every operating system. I'm fine with "out-of-tree only if you can create symlinks yourself or just want to install or build an installer", as that was the case since we started using CMake, and I would like to keep the complexity of this part of the whole as low as possible, but I'm not opposed to making it better.
Slightly abridged extracts from #openclonk-dev just now:<Guenther> So, anybody else an opinion on http://forum.openclonk.org/topic_show.pl?tid=859
<ck> make cross-compiling work again, please
<Guenther> what's your general opinion on how the engine should get the unpacked data from the repository?
<ck> It should read the files in planet/
<Isilkor> when cross-compiling, we shouldn't create any symlinks at all
<Isilkor> because the generated binary won't run on the host anyway, so why have a symlink there
<Guenther> Isilkor: wine
<Isilkor> what I'm doing at home is I have a symlink in planet to each of the binaries
<Guenther> on windows?
<Isilkor> yes
<ck> I don't think the Makefile should create any symlinks
<Isilkor> I don't think so either
<Guenther> I kinda like a symlink to planet from the build dir to the source dir for the case where the two aren't the same
<Guenther> since you're presumably using an otherwise empty build dir for an out-of-tree build
<Isilkor> I am
<Isilkor> I'm both using an out-of-tree build at home and for the autobuilds
<Guenther> Should the binary know it got created in a Release/ or Debug/ etc. directory and search in ../planet or ./planet, or should we use symlinks to make both cases look the same for the binary?
<Isilkor> that's tricky
<Isilkor> I would be in favor of not special-casing MSVC, but that'll make things weird for newbies who just want to compile their stuff themselves by following our tutorial
<Guenther> There's no special casing specifically MSVC, it's all CMake that configures those paths
<Isilkor> I mean having MSVC builds search for their files in multiple locations, while g++ builds don't
<Isilkor> or a different location, for that matter
<Isilkor> basically I shouldn't have to care about whether I use a MSVC binary or a g++ binary in the end, I just drop it at a specified location and it works
<Isilkor> if that's planet or planet/.. isn't really the issue I believe
<Isilkor> from an end-user perspective, I mean
<Guenther> Well, if you move the binary, you should just use the search path used by installed binaries
<Isilkor> I clone the repos, and drop a prebuilt openclonk.exe into whatever directory, and it should work
<Guenther> Well, on windows that happens to work
<ck> what about building release and debug engines into the same directory but calling them differently?
<Isilkor> ck: that's what BUILD_TO_PLANET did, I believe
<Guenther> we could reuse that logic, but put the binaries into the build dir instead of the repos/planet dir
<Guenther> and not make it an option
<ck> I don't mind whether the binary ends up in planet or elsewhere
<ck> Just to avoid special casing MSVC
<Isilkor> I do mind when it ends up in planet, because planet is outside of my build root
<Isilkor> building to the build root would work just fine for me though
<Guenther> And I also don't want to bring the option back
<Guenther> That would be way worse than making multi-config cmake targets work differently
<Isilkor> we should just build it to ${CMAKE_CURRENT_BINARY_DIR}/openclonk{-debug,-minsizerel,-relwithdebinfo}
<Isilkor> unconditionally
<Isilkor> but have either the release or the release+debuginfo binary just called openclonk
We then argued some more, but didn't arrive at any better solution. (Though continuing to call the release binary clonk(.exe) would be less work than switching to openclonk(.exe)) I think Isilkor volunteered to do the CMake work. :-)
> <Isilkor> because the generated binary won't run on the host anyway, so why have a symlink there
On the other hand - it doesn't hurt either, does it? I feel like making sure that you can actually do something with the binary no matter what paths you configured is good practice.
If people feel symlinks are the wrong way to do it - especially on Windows - we could think about just leaving a file there, like I proposed earlier. Maybe support loading configuration (as in C4Config) partially from a file, then have CMake generate one, overwriting the system data path? I don't know, there's a lot of ways we can do this. I think we are giving up on the out-of-tree-build feature a bit too hastily :)
> <Isilkor> we should just build it to ${CMAKE_CURRENT_BINARY_DIR}/openclonk{-debug,-minsizerel,-relwithdebinfo}
Fair enough, I guess that's the safe solution. Note though that this is positively incompatible with how Mac builds work.
> Note though that this is positively incompatible with how Mac builds work.
The Mac build is sufficiently different from everything else that I didn't want to touch it. Especially since I don't have one, nor do I have any experience with compiling/packaging OS X software.
> I think we are giving up on the out-of-tree-build feature a bit too hastily :)
I've been building Clonk exclusively out-of-tree for years, and will continue to do so. I wouldn't call that "giving up".
Yeah, but as long as it only works on certain platforms and requires setting up undocumented magic behind the scenes, it's more of a hack than a proper feature.
Up to 5.2.x, it wasn't any worse than in-tree builds, so it's not the end of the world ;-) I'm probably just a bit cranky because I spent some time to improve the in-tree situation to not require symlinks or non-obvious cmake options, most of that testing it with MSVC.
Hey, and I spent quite some time trying to get the symlinks solution to work under Windows. Just to be instantly reverted. Can I be cranky too? :P
Not because of the revert - the build failures are sufficient for that. In a push-then-review process, a revert doesn't mean permanent rejection, but only that somebody else noticed some breakage and didn't have time, ability or motivation for a "proper" fix. But feel free to get cranky because I disagree with you. :-)
> I think Isilkor volunteered to do the CMake work. :-)
"This changeset also revives looking for game data in the same directory as the binary" is wrong. On windows, SystemDataPath is the exepath. So all the patch did in that regard is to move the UserDataPath behind the SystemDataPath on windows (and the Linux releases using the same layout), while platforms with a different SystemDataPath will have the UserDataPath first. I think that's unfortunate.
Also, is MSVC really the only multi-configuration CMake target? I think the test should be for that, not for one specific CMake target.
> Also, is MSVC really the only multi-configuration CMake target? I think the test should be for that, not for one specific CMake target.
CMake docs propose MSVC and XCode. They also don't say how to test for it at all (CMAKE_CONFIGURATION_TYPES maybe?). Since XCode uses different targets anyway, I've limited the change to MSVC only.
> The good thing about symlinks is that we can get pretty close to what the binary sees in an actual installation
We can get there fully by creating the packed groups in the configuration specific directory instead of the toplevel build directory like we do now. But we don't actually want to get close to the actual installation, we want to work with the unpacked data directly from the repository, without having to call make every time we edit the game data. Running different code paths is unavoidable, the handful of lines of code for planet are only an insignificant part of them.
> Also, isn't it about time that we stop calling it "planet"? Maybe "game" or "data"?
The time for renaming it was when switching to HG, or when the term first became inaccurate, or with the *.oc? transition. Now it'd just churn, making everybody adjust without much benefit. Though we may want to do it when we're switching to git, hiding it in the bigger adjustments.
I feel like we already did changes with more churn and less benefit. It would safe people at least one initial "WTF".
I also object to the inappropriate use of "when" for an event that's far from certain to happen :P
I also object to the inappropriate use of "when" for an event that's far from certain to happen :P
> I feel like we already did changes with more churn and less benefit.
Which ones? The *.oc? change would certainly have nicely hidden the planet->something_else transition, but other than that all changes I can recall had real benefits.
>Also, isn't it about time that we stop calling it "planet"? Maybe "game" or "data"?
When we moved from c4 to oc, the size of the repos jumped up about 100MBs (or more?). I don't want that to happen again. And 'about time', you're 8 years late...
Well, now we are about to hard-code it more then ever before. Seems like the right moment to me.
>Also, symlink and windows?
Builtin from Vista, needs junction.exe otherwise. Works on all NTFS-Volumes.
>if you followed the tutorial in the wiki, you will have an out of tree build - nearly everyone used the tutorial to set up his environment.
Neat, I didn't think it was of use to change the target path to planet. Well, anyway, the 'work' so far is to start the cmake gui and strip the planet/ from the target path.
The occasion for me to finally change this was the recent divergence of the mac build. But the main motivation was to make the case of CMakeLists.txt and CMakeCache.txt in the same directory work, because that is what anybody who hasn't read the tutorial would do. I think I simply expected the planet/CMakeCache.txt case to still work as before, because the tutorial doesn't use BUILD_TO_PLANET, and I assumed that was because the engine landed in planet/ and would find the data like an installed engine would. Now I'm somewhat baffled that the the tutorial has worked at all.The simple solution is that everyone puts their CMakeCache.txt in the same folder as the CMakeLists.txt. That'll reduce the diversity of build environments and prevent future breakages. I'll start by making the tutorial say that.
Another solution would be to hardcode the path to planet in the executable, instead of looking for planet in the directory where the exe or the folder with the exe is. But I don't really want to leak the build environment into the release binaries like that.
> The occasion for me to finally change this was the recent divergence of the mac build
Care to fill me in?
> make the case of CMakeLists.txt and CMakeCache.txt in the same directory work
In what way did it not work before?
The Mac build now always expects the game data to be bundled with the executable. The reason behind that is that just searching in the package's directory is no good idea - for Mac that's possibly
So the way I solved it is to either copy/pack or symlink the game data into the package. See the script.
/Applications, and there's all kind of stuff in there (some of which actually made Clonk crash due to uncaught DirectoryIterator exceptions).So the way I solved it is to either copy/pack or symlink the game data into the package. See the script.
Powered by mwForum 2.29.7 © 1999-2015 Markus Wichitill
