I seriously doubt that your credibility is shot. It's fairly common for coworkers drink together--builds "team morale." Or something. I don't think anyone thought the worse of you, given what you've told us.

Just a thought: maybe you're a bit conflicted over wanting to fit in with a bunch of people you don't particularly like, and now you have regrets about letting your guard down, especially considering the icky dynamics.