Shalom u'Verachah,

The Ramban (in Bereishis 37:18) explains that, initially the brothers tried to get rid of Yosef without actually laying a hand on him, and they decided to throw him into the pit only when that backfired. Presumably, they did so because they considered him a Rodef -- which does not require a Beis-Din. The Or ha'Chayim (Pasuk 20) assumes that they did sentence him to death, but since they had the status of Bnei Noach, they did not require a Sanhedrin. See the Or ha'Chayim there as to why they nevertheless combined to judge him.

According to those who maintain that the Shevatim had the Din of a Yisrael, it is difficult to see how they could have judged him according to Din Torah, seeing as (a) there was no warning and (b) there were no witnesses and no Beis Din -- as brothers are disqualified from testifying as well as from judging.

B'Virchas Kol Tuv,

Eliezer Chrysler