In took my daughter at 6 weeks old in 2019, only for the Sunday though.
i had her in a sling and carried her, baby facilities in the kids field were amazing and I sat with lots of other breastfeeding mums in a nice quiet dark tent at the back of the field.
However, it was very hot during some parts of the day and I freaked out when walking in the crowds with her.
i think with careful planning ( ie staying away from the crowds as much as possible), having a nice comfy sling or carrier for baby and the option to leave site and stay in a nice hotel for Mum and baby, you will be absolutely fine. You can also get cute little ear defenders for newborns with have a soft material bands so they don’t squash babies head and are nice and comfy.
As others have said it’s probably the easiest age to take a child.
Also, you could book Ashcombe festival parking, which is about 20 min walk for gate d… at least you’ll know where you’ll be parked everyday and won’t end up in the furthest car park away.