How do I make this better?

My very close friend and I have been sleeping together for a while (over a year off and on) and I just spent yesterday with one of his best friends, and was told that the man I'm sleeping with likes me, but feels like I'm shameful becuase his friends don't like me at all. But he continues to see me on occasion. How can I get past this?