I think people do associate a certain level of competence with your rates, whether or not that’s what you want.
I have recently raised my rate per person, but it was time for an increase. I have also learned that being the “cheap” anything is nothing but a hassle. The more people buy in to the program, the higher adherence to the program you typically have.
Does that mean you should never do something to accommodate those for whom your prices are out of reach? No. You just have to decide how to best do that (such as group sessions, donated sessions/volunteering, etc).
If someone doesn’t like your rates, and they are comparable to others in your area with your experience, then keep them where they are. Not everyone is going to like it, but those were probably the clients you didn’t want in the first place!