You should tell him exactly how you're feeling...it's better to deal with these kinds of problems straight-up.
the bipolar is probably affecting it.