I think everybody is worried about that. And it is a valid worry because on the one hand, it is human to make mistakes. With all the recent studies on human error, everybody is very aware of how common mistakes are. On the other hand, it is financially lucrative for patients and lawyers to sue, and mistakes are punished in the courtroom.
I think I deal with it by avoiding high-risk activities and patients, and by trying to bond with my patients, too. I also try not to think about it too much. Hopefully with time and experience you will start feeling less worried about it.