no one has a problem with you enjoying your inheritance. live it up, take a plane somewhere like your son, or visit your brothers yacht for a holiday - no one cares.
calling people with workplace pensions (pretty much everyone) tax avoiders is a massive reach. just like when you called someone a free loader for being given a coffee.