Another consideration are what are called "subclinical symptoms." For those who are highly sensitive to any food, full reactions are obvious and potentially quite dangerous. But ingesting extremely small amounts of that food, even if no obvious symptoms are observable, can adversely affect the person - symptoms might be there but are small enough that they won't be noticed.
One common example is that the ingestion might lead to greater inflammation in the sinuses, interfering with drainage and increasing the likelihood of a bacterial sinusitis. That may not happen until the next time the person gets a cold. Making the link is certainly not obvious.
Or the ingestion might just push the person closer to their "threshold of reactivity." In these cases, they're just fine, but a small amount of some other allergen, e.g., dust, mold, pollen, or even another food allergen, becomes more likely than normal to trigger symptoms. Since that typically happens later, it's also hard to link the ingestion of a lightly contaminated food to the reaction.
In general, we try to moderate our son's consumption of these "borderline" foods. If he seems overall more symptomatic (itchy, sneezy, etc.), we ask him to dial back on the borderline items, especially during high pollen season. It usually works, but it's often a fight. He'll opt for the tasty treat and being more itchy, cranky, etc. After a point, we get more insistent and he sometimes sees our point.
Not easy, but certainly necessary. He's a person. His diet is limited. He deserves some treats. We're his parents; we need to protect him; and we need to live with him. Nobody said parenting was easy. Parenting a food-allergic kid is far from that.
Mark