In some ways refactoring code like planting a tree. Not just the growing, branching, and pruning aspects (although that's something to talk about at another time), but also, like the old proverb about the best time to plant a tree being 20 years ago, the best time to refactor your code was back during the design phase. If you had made the change then you wouldn't be in the mess you're in now, and you'd be able to add all those new features with minimal friction.
The second best time is now. Sometimes it's actually the best time. You've come this far without needing the changes, so it wasn't too big of a mistake. You know a lot more about the situation, so you can make a better decision. You can set yourself up for the future.
Back when I was still at Microsoft we we built a data validation/QA tool to help people build up truth sets, easily view maps and fix/mark issues, and compare the 2D, 3D, and StreetSide (SS) views of the same area. And one of the things we knew from the beginning was that we needed to have a way to keep the viewpoints in sync. But there were issues with that. We could think of a bunch of different ways to keep 2D and 3D in sync, with different leaders/followers and considering if you were looking from a point or looking at a point and all of the different combinations of them. Some of those combinations were familiar to our staff, but some weren't, and. And none of them had any clue about how they would add a SS image cube to the mix, let alone be able to draw in one view and see it in the others, in real time.
So our solution was to handle the 2D/3D case and not worry about the SS cubes. We made the 2D/3D part clean, and applied pretzel logic to make the SS view work. That let us get the tool out to our production team so we could add value and find out exactly what they thought, where the pains were, and what more we needed to do.
Then, with that information in hand, we did the refactor and made SS really be just another view. And since we did it when we knew enough to make it simple to use but easy to extend, when someone came up with the idea to change our SS cubes into a voxel based world, that view just slipped right in and worked without a hitch.
If you want more information on exactly how to refactor, check out refactoring.com.