It seems that he just really does not care about you. Nor does he really want to be a part of your life or your daughter's. The way he keeps going through women may be interpreted as his inability to act as a mature adult who is willing to commit to a healthy relationship. Much less be a father figure. He does not seem reliable and does not seem like he will be a good influence on your daughter. Personally, I think that all he is trying to do right now is simply use you. You said that he moved to this other woman's house, got her pregnant and then broke up with her. It looks like he was basically trying to live off the poor woman. Once she got pregnant, he didn't want the responsibility of being a father, and so he bailed.
You yourself said you don't love him nor want him back. And, again in my opinion, if you really respect yourself for who you are, I don't see why you should. If you take him back, he'll probably bring you nothing but trouble and pain. He will also hurt your daughter if he does not change his ways, which reminds me of the saying "you can't teach an old dog new tricks". It's true that it is better to have both parents, but it's only the case when both parents are qualified and are a good influence. Sometimes, having one parent is much better than having two.
Since he is continuously pestering you, and sees no progress, he may use the "father" card in the end to get whatever it is that he wants from you, but you might want to ignore that and even get a restraining order if needed. Since you said that he never asked for your daughter and that you think that he doesn't really care for her, it looks like he hasn't really done much of anything for the child. All he might do would be just either ask for money or simply laze around your place if you give him another chance.
If you really don't care about him and want what is best for your child, you might want to cut all ties with him completely. It is harsh, but he does not sound like such a great person to me. But you know him much better than I do so it all comes down to you. It's just that if I had a child, I would want what I think would be best for him or her, I wouldn't want my child to develop what some may call "daddy issues" due to his/her father's behavior, because it looks like if you give him another chance, he'll just come and go as he pleases when it comes to your lives. That just won't be good for the child's mental health.
Well, sorry for short novel and whatever your decision is, wish you the best.
- Response by peaceoffspring
, A Creative, Female, 26-28, Who Cares?