It’s pretty common and typical actually. I personally was pretty allergic to dairy starting out (to milk and cream) and now I am not nearly as allergic but still am a little bit. I wasn’t allergic to greek yogurt for some reason
You might want to consider trying out goat’s milk kefir next; if you cannot find it, you can make it at home.
Also if it was not organic, this could be apart of the problem as well.