Is it rude to say "sorry?" when you didn't hear what someone said?

When someone says something and I don't understand or more likely didn't hear them well enough, I find myself saying "sorry?" instead of pardon me or come again. Is that a rude thing to say when it's someone you don't know very well/ at all?