We employ finely geo-coded firm-level panel data to assess the long-standing question whether agglomeration economies derive from specialization (within-industry), diversity (between-industry) or overall density. Rather than treating the city as a single unit, we focus our analysis on how the inner industry structures of cities influence firm-level productivity. Our results illustrate the co-existence of several externalities that differ in their spatial distribution and attenuation within cities. First, we find robust positive effects of neighborhood-level specialization on TFP as well as a small effect of diversity at the same fine spatial level. These effects are highly localized and dissipate beyond the immediate within-city neighborhood level. Second, we also find that firms benefit from the overall density of the wider city. The results emphasize the relevance of “opening up” cities to study the workings of their inner organization, and support the idea that location in a within-city industry cluster in a diversified and dense city boosts productivity.