Recalled back to prison - from Anonymous

Hi. Please don't put my real name down. 
My partner was recently released from prison and I thought everything would work out fine. As far as doing well in prison; he did well, or so I thought. When he was arrested, the trauma me and our daughter went through was immense. It was painful and took a lot of soul searching. 
Last week he was recalled to prison because he failed to keep his appointments with his offender manager. I thought he was attending and had no reason to believe he was skipping his appointments. 
I also found out that he was back on the drugs and after standing by him through his prison sentence, he goes and does it to us again. 
I can't do this anymore and I have sent him a letter ending our relationship. I feel so guilty about doing this but could find no other way around it. He has let me and his daughter down so badly and he's let himself down. I know I have done the right thing but I keep on questioning myself. I have told him should his daughter want to visit him, I will bring her and want this as amicable as possible. It is a nightmare and my feelings are torn. 
I know I have to put me and my daughter first now at all costs but I feel so guilty about walking away. I just need to let off some steam and to know that if there are any other people who are going through the same thing just to say I know how difficult it is. 
Thanks for listening. A partner of a prisoner