Should I Be Wearing My Retainer Longer?

My orthodontist said that I should wear my retainer for 15 hours a day (so basically from about 9 PM to 12 PM the next day) 7 days per week. However, I’ve been wearing it about 12 hours (12 to 12) a day for maybe 6 or sometimes 5 days per week. Is this a big deal or is it important that I wear it strictly as instructed?
It is an invisalign lifetime-wear retainer.