Black phosphorus (BP) is receiving significant attention because of its direct 0.4–1.5 eV layer-dependent bandgap and high mobility. Because BP devices rely on exfoliation from bulk crystals, there is a need to understand the native impurities and defects in the source material. In particular, samples are typically p-doped, but the source of the doping is not well understood. Here, we use scanning tunneling microscopy and spectroscopy to compare the atomic defects of BP samples from two commercial sources. Even though the sources produced crystals with an order of magnitude difference in impurity atoms, we observed a similar defect density and level of p-doping. We attribute these defects to phosphorus vacancies and provide evidence that they are the source of p-doping. We also compare these native defects to those induced by air exposure and show that they are distinct and likely more important for the control of electronic structure. These results indicate that impurities in BP play a minor role compared to vacancies, which are prevalent in commercially available materials, and call for better control of vacancy defects.