I've Been Made An Outcast - By A Prisoners Partner

Hi. I would like to ask anyone if they have been made an outcast in their neighbourhood because of a loved ones imprisonment? There is always a group a women stood outside their doors chatting about everyone, and the topic of conversation is me because my partner has just been sent to prison. I hate walking past them when I go to the shop because as soon as I am out of ear shot, they start tickle tackling away. I used to love where I lived, but now it's a nightmare and my kids are being bullied in the street too because their dad is in prison. Is this a common thing that happens when a loved one is sent to prison? It makes you feel so alone and embarrassed too!