If you dread or fear refactoring, it’s because your code is a big ugly mess.
If your code is a big ugly mess, it’s because you didn’t refactor soon enough or often enough.
To make your code not be a big ugly mess, you need to refactor it.
So, refactor every time you see the opportunity. Keep the code lean and pretty. Then refactoring stays simple and easy and doesn’t take long or hurt.
I say do this even if you feel mounting deadline pressure. In the long run, the gains in maintainability will far outweigh perceived slowdown of productivity. Working on maintainable code is far more productive than working on unmaintainable code.
Don’t ship messy code. Messy code hides many flaws. Flaws that you’ll need to fix sooner or later anyway. When you fix a mess, it’s a lot more work than fixing something that’s clean.