Recent years have seen a remarkable development of deep neural network techniques for data analysis, along with their increasing application in scientific research across different disciplines. The field of mathematics has not been exempted from this general trend. The present paper proposes a philosophical assessment of the epistemological claims and conditions of such attempts. After a quick survey of recent applications of neural models to mathematical knowledge, we address the philosophical significance of those results, focusing on the specific problem of mathematical textuality and the somewhat surprising circumstance that semantic aspects of mathematical knowledge can be inferred from pure syntax. We then analyze the renewed role of distributionalism in neural models and propose an alternative understanding of its relation to meaning. Finally, we present an illustration, based on empirical evidence, of how aspects of arithmetical content such as recursive structure and total order could be inferred through explicit and interpretable means from the distributional properties of a natural language corpus.