I'd say take him with you if you are going to be gone more than a few days and since it is a 16 hour drive you more than likely are going to be gone for a week or more.
I would be a little reluctant to leave him at home with someone who treats him with contempt and that you think would try to get him out to handle him, that would probably result in a nasty bite and maybe have Oliver stressed out over it too.
I think you can handle the little ones who will be excited to see him by explaining that they need to keep a distance so they don't get hurt. I think Oliver will enjoy traveling with you and seeing new people.
I would let him ride in his carrier with some food, water and a couple of toys to keep him busy. When I travel and take Josey with me she rides in her travel cage, I don't get her out to sit on me and you shouldn't either since you will have a 3 year old with you, what if she rolled the window down and Oliver got out, I shudder to think.
So in my opinion take him with you, you will enjoy having his company and he will get to meet new people and I think he will be less stressed that way.