Sorry Jenny but I've gotta go the other way here.
Your little poo and puke factory won't miss you for 5 days. Its 5 days out of the next whole 18 months or so you have of baby time!!
See it as a little 'you' time. I honestly would go this year, leave him with the 'rents, and enjoy yourself. I can't see it being any easier to go and leave a real young toddler in a year or two's time, when they can't wander around, pull things, fall over and draw on walls!!!
Don't do it. Five days is nothing in the grand scheme of things.