< But I've always thought that it requires many thousands
 of years before we adapt to "new" food. Or is milk one of those
 "easy" adaptations (through neoteny) that require much less time?>

it's simply a case of not losing the lactose-digesting capability.  anyone
who keeps this into adulthood will have a distinct advantage. 8 generations
later, behold, most of the community can take it.

OTOH, if you have a particular sensitivity to say the beta-lactalbumin
protein, this need not have any genetic basis. It's just the luck of the
draw regarding your biochemistry.  Just as you can eat pawpaw and your
sister can't.  In that way milk becomes like any vegetable, fruit, nut or
seafood.

If you can't take cow's milk, you mght be able to take yoghurt. Or sheep
milk. or cheese. If not cheddar, then camambert. if not raw, then cooked in
a fondue.  you just have to experiment, as you do with berries or nuts.  I
can take apples, but not oranges. mangoes, but not pawpaw. its the same
story. experiment.

Andrew