I tend to wonder just how much those genetic difference really matter

Well, they exist. How much they matter is totally open to interpretation. Therein lies our problem. Humans will always find a reason to say "that group that is demonstrably not like me is inferior because [made up reason]." If we didn't have races, we have genders, or religions, or nationalities, or height, or weight .... If we didn't have those, we'd have "Let That Be Your Last Battlefield." It's a human trait to need to divide other humans into "acceptable" and "nonacceptable." It's fear and a weak ego and a desire to have all the toys. It's about being top dog. There's no science and no logic to it (well, beyond the logic that it's always better to be a winner than a loser).