Hey, you're not new here by any chance are you!?

Welcome to PSG!
As for "rough" edges. It's hard to say exactly what to do without seeing them, but I'll take a couple guesses.
1. You're using polagonal lasso tool, so you are getting sharp corners. Don't do that. Use the pen tool to make smoother shapes.
2. You've got "anti-aliasing" turn off and you're getting jaggedy pixelated edges to your diagonals and curves. Turn "anti-aliasing" on.
3. You've made a selection with your tools... made a new layer... filled it with a color... changed your mind and filled again... and again and... Don't do that. Once you fill, the anti-aliasing won't be filled again and you'll end up with edge artifacts. Either undo to recolor or turn off your selection and lock transparency.
Hopefully one of those will clear up the "rough edges" problem. If not, post a picture for us to look at to see what's actually going on.