hold something against somebody

to continue to dislike somebody who did something unpleasant to you.

Examples

What do you hold against my brother?