From an evolutionary perspective I think dairy is quite a bit less
troublesome than grains are.  It may well be that we were herding animals
before we were farming grains.  Clearly, at least 25% of the world
population has developed lactose-tolerance, which proves there has been
some evolutionary adaptation to dairy products--and since I fall into the
25% that has no trouble with dairy, for me personally I'm a lot less
worried about it than I am for someone who is lactose intolerant.

I do, however, think that dairy should be looked at with some suspicion.  I
won't eat it more than occasionally, and usually in only small quantities.
Modern milk is dangerous for all the chemicals they pump into it and the
other weird things they do to it to make it pastueurized, homogenized,
laced with artificial vitamin D, etc.  Not to mention the further chemicals
they dump into skim-milk.  I am a lot less worried about a drop of cream
here and there, a pat of butter, or a wedge of cheese than I am about
drinking a glass of milk or making dairy a primary food source.