>Would such a project even make sense as a fork or something on the main repository? Should I create a topic "OC the next mega cool version" ^^?

Would it make sense as a serious attempt to fix issues with the OC engine?
No, not at all.
Would it make sense for you for your own growth?
Yes! Definitely! You will learn so, so much by starting such a project. And that will help you for about anything you will do later in your life.

If you really wanted to change certain aspects about the game engine, then the more sensible (but also more boring) approach would be taking a specific part of the game engine (e.g. movement code), understanding how it works, cleaning up and refactoring all the parts that belong to it, and then finally rewriting it.
This is about what I did for the particle system a few years ago.
The disadvantage is that it's obviously less cool than writing your own engine ;)
The advantage is that it actually has a decent prospect of being finished. And if enough people do that in different parts of the engine then this is more than enough to break up some crusts.
For example, changing the mapping of skins to players is probably a rather small project compared to re-doing the engine from scratch