Shalom,

You are asking a difficult question. On the one hand, the rules are that once the witnesses gave their testimony, they cannot retract (unless other witnesses come and contradict them or say they weren't in the place they claim were in when they saw what they claim they saw). On the other hand, it was close to obvious that these people who were unhappy with Shimon Ben Shetach plotted against his son.

It is worthwhile seeing the Yerushalmi (Sanhedrin Perek 6, Halachah 3) which sheds a bit more light on the story. Here is a translation of the Yerushalmi:

Shimon ben Sheṭacḥ's hands were hot (he was quick in prosecuting persons not conforming to rabbinic standards). There came a group of scoffers who said, "Let us take scheme, testify against his son, and kill him." They testified against him. He was sentenced to be executed. When he was taken to be killed, they told him, "Our master, we are liars." His father wanted to return him; his son told him, "My father, if you want that salvation come through you (by making a Kidush Hash-m, that even your own son you do not return once the verdict has been passed), treat me as a door threshold (that everyone tramples)."

Shimon ben Shetach probably did not participate in the panel that tried his son, but he had an influence on what was happening. From the story it seems that he almost failed to deviate from the rules of the law not because of public considerations but in order to save his son from the terrible injustice that had been done to him. But here his son stood up and told him that in this case the public interest actually demanded that the rules be adhered to. In that generation, rabbinic Judaism was struggling with various sins that were plaguing the people, on the one hand, and with the Tzedukim's opposition to the Perushim's tradition of the law, on the other hand. Those who seek our harm could not have found a better weapon than compelling the greatest of the generation to twist the rules in order to save his own son. Therefore, his son said to him, with terrible devotion that if Shimon ben Shetach wanted salvation to come through him and his son, he should not try to save him.

There is no doubt that the story is shocking and a bit difficult to understand, but one must try to connect both to the Halachic aspect and the general rules of Halachic law, as well as to the conditions of that time and the struggles of Shimon Ben Shetach and his generation.

I hope this helps!

Aharon Steiner