At this time it is not possible to remove someone from your favourites list, but we have this feature on the development list and hopefully it will be available in the near future.
The clothes you choose to wear on a date should be appropriate for the type of date. If you need help choosing what to wear, take a look at this article.