Check around the house for air leaks. Hold either a lit candle or incense stick and move close to the windows and doors. You will see it flicker if there is a draft. (Make sure you don't hold the candle close to the curtains!).
Seal any leaks with caulk.