Probably the most optimal time to get a refinance on your mortgage is when you can improve your mortgage terms and lower your cost of borrowing. If you are not improving the terms of your mortgage and you are not actually lowering the cost of borrowing, then it’s not a good idea to refinance.