Do you have any trouble digesting fats? Or do you ever leave “skid marks” in the toilet bowl? That would suggest lack of bile production and possible liver/gallbladder problems. Have you seen your doctor about it?
The bile won’t turn brown unless it’s metabolised by the bacteria in your intestines though, so that’s likely more your problem since you’re on this site. Restoring a healthy gut flora should restore a healthy brown stool.
I’d guess the dairy changes the colour since the delayed transit time allows more time for the bacteria to metabolise the bile. Perhaps your usual transit time is too fast? It should be around 24-hours. It’s possible to check by taking a charcoal capsule.