A number of commentators discuss your questions (see the comprehensive explanation of this Sugya in the Ya'aros Devash of Rav Yehonasan Eibeshitz). Rav Elazar Rokach, the author of Ma'aseh Hash-m, has an extensive discussion of this topic, which is quoted by the Chida (Rav Chayim Yosef David Azulai) in his Petach Einayim commentary on Berachos. Rav Rokach also asks why Rebbi Shimon was not satisfied with Rebbi Yehoshua's answer and proceeded to ask the question again to Raban Gamliel, seemingly slighting Rebbi Yehoshua in the process.

It is well known that Rebbi Shimon bar Yochai used to study Torah together with his colleagues unremittingly. The Gemara in Shabbos therefore concludes that he and his colleagues would be absolved from the obligation to pray, since he was involved in Torah study. This exemption may be derived from the story of Yakov Avinu, who established the Ma'ariv prayer while on his way to Lavan's house. It could be asked, why was the Ma'ariv prayer mentioned at this juncture? Chazal say that he spent 14 years previously studying Torah and did not rest in bed, so he was absolved from Ma'ariv until this point, and this is the source for Rebbi Shimon's conduct. However, if Ma'ariv is only an optional prayer whereas the other prayers are obligatory, Rebbi Shimon might have had to stop for those obligatory prayers, and this is the reason why he posed the question in the Beis Midrash. Rebbi Yehoshua, knowing Rebbi Shimon's dedication to Torah study, told him that Ma'ariv is not obligatory, and, therefore, for obligatory prayers he should stop learning Torah and pray (which he seemingly did; see Shabbos 33a, where Rebbi Shimon's daily prayers are described).

Rav Rokach notes that Rebbi Shimon's question was then asked by a "questioner" and not by him personally, as the custom was to let one person pose the questions in the Beis Midrash. Raban Gamliel then answered (without knowing who was asking) that Ma'ariv is an obligation incumbent on everyone, as they do not all study with such intensity.

However, the Gemara concludes by telling us that in reality the questioner was Rebbi Shimon. (We could also suggest that he was too humble to present the question himself to Raban Gamliel or did not want to offend Rebbi Yehoshua, so he sent it via a "questioner.")

Rav Rokach concludes that there is proof for this in the Zohar (Parshas Chayei Sarah) which tells of David, king of Israel, being absolved from Ma'ariv, as he did not sleep in the normal manner at night (see Berachos 3b) and spent the night immersed in Torah study.

Yoel Domb