Sen claims that his 2009 theory of justice is based in part upon Smith’s idea of the “impartial spectator”. His claim has received criticism: some authors have responded that his interpretation of Smith’s concept is unfaithful to the original (e.g., Ege, Igersheim and Le Chapelain 2012); others, focusing on internal features of Sen’s analysis, critique his use of the Smithian impartial spectator, arguing that it is a weak point in his comparative theory of justice (e.g., Shapiro 2011). In this paper we address both sets of criticisms. While agreeing with commentators that Sen’s reading of Smith is somewhat unfaithful, we reiterate that his aim in The Idea of Justice is not to provide an exegesis of Smith but rather to build his own comparative theory of justice by “extending Adam Smith’s idea of the impartial spectator” (IJ: 134) to his own project. After clarifying their distinct approaches to the concept of the impartial spectator, we draw upon our account of these differences to evaluate Sen’s own use of the concept. Despite significant divergences, we show that Sen’s version of the impartial spectator is not inconsistent with Smith’s analysis. Though it does not correspond to Smith’s concept, i.e. to what the Scottish philosopher sometimes calls the “man within”, it is reminiscent of another figure from Smith’s moral philosophy: the “man without”. Beyond this analogy, there are further connections between Smith’s imaginary figure of the “man within” and Sen’s account of “common beliefs”—both notions are ways of representing our beliefs regarding what is moral or just. But whereas Smith’s moral philosophy offers an analysis of the process by which the “man without” influences the “man within”, nothing of that kind is to be found in Sen’s conception of public reasoning. And it is here that Smith’s famous concept of “sympathy” can supplement Sen’s theory, in a way which furnishes an answer to Shapiro’s (2011) criticism regarding the possibility of the spontaneous change of beliefs toward greater impartiality.