In this day and age of specialized diets, it seems that all sorts of foods are being shunned. But when you’re unwilling (or unable) to break bread, it can set you up for feeling left out.

That’s according to several Cornell University studies, which found that when adults and children with diet restrictions—due to allergies, health concerns or religious norms—can’t share in the joy of what family or friends are eating, they are more likely to feel stricken with a sense of loneliness.

In one experiment, published in the Journal of Personality and Social Psychology, individuals with no dietary restrictions were assigned to a food-restricted eating plan. While on this plan, they reported feeling more lonely than when they could eat what everyone else at the table was eating. In another instance, a survey of observers of the Jewish holiday of Passover revealed that participants’ sense of loneliness was elevated when they were reminded of the leavened foods that were off-limits during the holiday.

This all makes sense when you consider that mealtime can be an opportunity for bonding and is inherently a social experience. So, watching others enjoy eating what you can’t have could leave people feeling that they are not wholly included in the group; they may even start to believe that they are being judged.