While this is written from a gluten point-of-view, you can see that lactose
is one of the allowed ingredients.

Excepts from Ener-G Food's "Detailed Ingredients Listing":

Annatto Color
  Because ethyl alcohol and/or caramel color can be present in annatto
  color and is not declared, people with Celiac-Sprue should consider
  avoiding this ingredient.

Caramel Color
  The problem with caramel color is it may or may not contain gluten
  depending on how it is manufactured. In the U.S.A. caramel color must
  conform with the FDA standard of identity from 21CFR CH.1. This statute
  says- The color additive caramel is the dark-brown liquid or solid
  material resulting from the carefully controlled heat treatment of the
  following food-grade carbohydrates:
        Dextrose (corn sugar)
        Invert sugar
        Lactose (milk sugar)
        Malt syrup (usually from barley malt)
        Molasses (from cane)
        Starch Hydrolysates and fractions thereof (can include wheat)
        Sucrose (cane or beet)
  (Also acids, alkalis and salts are listed which may be employed to assist
  caramelization)