While this is written from a gluten point-of-view, you can see that lactose
is one of the allowed ingredients.
Excepts from Ener-G Food's "Detailed Ingredients Listing":
Annatto Color
Because ethyl alcohol and/or caramel color can be present in annatto
color and is not declared, people with Celiac-Sprue should consider
avoiding this ingredient.
Caramel Color
The problem with caramel color is it may or may not contain gluten
depending on how it is manufactured. In the U.S.A. caramel color must
conform with the FDA standard of identity from 21CFR CH.1. This statute
says- The color additive caramel is the dark-brown liquid or solid
material resulting from the carefully controlled heat treatment of the
following food-grade carbohydrates:
Dextrose (corn sugar)
Invert sugar
Lactose (milk sugar)
Malt syrup (usually from barley malt)
Molasses (from cane)
Starch Hydrolysates and fractions thereof (can include wheat)
Sucrose (cane or beet)
(Also acids, alkalis and salts are listed which may be employed to assist
caramelization)