I always keep some heavy-duty hand cleaner around to really scrub my hands if I get anything less than desirable on them. The brand doesn't matter - just about any hardware store will carry this stuff in one form or another. It's often orange and smells like it. (TheWife points out oranges have natural cleaning properties.)
One of the things that's pretty universal with these types of cleaners and makes them so great is they're gritty. It's like washing your hands with soap that's full of sand. This grit gets into all the joints of your fingers too and acts a little like sand paper or a file.
Basically, wash your hands like normal but give them a good scrub. The grit will help clean off your skin. I tend to repeat a couple times and finish off with some regular soap.