Plasmonic cavities represent a promising platform for controlling light–matter interaction due to their exceptionally small mode volume and high density of photonic states. Using plasmonic cavities for enhancing light’s coupling to individual two-level systems, such as single semiconductor quantum dots (QD), is particularly desirable for exploring cavity quantum electrodynamic (QED) effects and using them in quantum information applications. The lack of experimental progress in this area is in part due to the difficulty of precisely placing a QD within nanometers of the plasmonic cavity. Here, we study the simplest plasmonic cavity in the form of a spherical metallic nanoparticle (MNP). By controllably positioning a semiconductor QD in the close proximity of the MNP cavity via atomic force microscope (AFM) manipulation, the scattering spectrum of the MNP is dramatically modified due to Fano interference between the classical plasmonic resonance of the MNP and the quantized exciton resonance in the QD. Moreover, our experiment demonstrates that a single two-level system can render a spherical MNP strongly anisotropic. These findings represent an important step toward realizing quantum plasmonic devices.