Whenever we've done it there has been no issues from anyone we've seen at all.
use your common sense (I.e, park/pennard hill was being used for clay pigeon shooting/certain hidden bar was fenced off so gave them both a moss). If asked to leave say no worries and move along.
Gave a few people from Worthy Farm a wave whilst walking and they all simply waved back, didn't seem too angry to us.