Kaffir is a racist slur in South Africa referring to black people. It means non-believer or infidel and is/was also used by muslims to refer derogatorily to non-muslims.
So, the word can be very offensive, although there is no clear connection between the slur and the fruit. In South Africa, the fruit is called 'Thai lime'.
Makrut is the more common name in Thailand and so has become a more neutral descriptor.