Lou you know there is a good idea to come out of this... Be honest? Then no one can out you/open their big mouth... Personally I couldn't give a toss if you are with ste, or not but don't try to fool people, it makes them feel like you think they are stupid. Toxic or not, if he's your mate, you ain't ready to dead things with him, whatever reason you're still talking, own it. You are a grown ass woman. Not one of those saying don't be with him explains who they are with... But they also don't play games pretending not to be with someone for the reason of conning people out of money/gifts. If people are going to help/enable you they will regardless.
You could of come out and said you have no intention of stopping talking to him, he's coming around for some food etc but you tried to hide it, while making out you are lonely etc which again the same as every other time pisses people off.