Rav Yitzchak Blazer zt'l explains this matter in the name of Rav Yisrael Salanter zt'l, in his Sefer Nesivos Or (printed together with Or Yisrael by Rav Yisrael Salanter), pages 115-6.

1) If someone sins against his fellow man, he must appease him, as we learn in Yoma 85b. As long as he has not asked for pardoning, the person who has been offended is allowed to bear "Tar'omes" and be particular (Makpid) with him. However, if he does apologize, then the person who was offended is not allowed to act cruelly in any way towards the penitent offender, and he should forgive him. Similarly, if someone damaged his fellow indirectly, even though "Gerama" is exempt from payment in Beis Din, the damaged person is allowed to have "Tar'omes" since it is forbidden to damage in such a way.

2) However, it is a great sin to have "Tar'omes" or be Makpid against one's fellow man if there is no justifiable reason for this according to the Halachah. Therefore, in order to hold "Tar'omes" against someone, one needs a special Heter, because otherwise it is a crime to bear "Tar'omes."

3) This is what the Mishnah (end of 75b) means when it states that the employer and the workers who tricked each other have only "Tar'omes" against the other party. Even though they cannot claim money from the other side in Beis Din because it is a case of "Gerama," nevertheless they are allowed to hold this complaint against the other, because what one did to the other was against the Halachah.

If not for the special Heter of the Mishnah, it would be forbidden to bear "Tar'omes" without any good reason.

Kol Tuv,

Dovid Bloom