If the angry build-up is too great, leave the room. A small spank could escalate to something violent. Regardless of the seriousness of the misbehaviour, the use of physical force is a NO. If you let yourself reach the limit where you'd inflict physical pain on your child, then you have rethink and set a limit.
Voice out to your kids when they're starting to annoy you, instead of trying to tolerate it. 'Cause you know what'll happen? You end up lashing out. According to experts, if it did happen, the next step is to apologise to the child. You may have to consider therapy for your anger issue.