L-cysteine derivatives induce and modulate the optical activity of achiral cadmium selenide (CdSe) and cadmium sulfide (CdS) quantum dots (QDs). Remarkably, N-acetyl-L-cysteine-CdSe and L-homocysteine-CdSe as well as N-acetyl-L-cysteine-CdS and L-cysteine-CdS showed "mirror-image" circular dichroism (CD) spectra regardless of the diameter of the QDs. This is an example of the inversion of the CD signal of QDs by alteration of the ligand's structure, rather than inversion of the ligand's absolute configuration. Non-empirical quantum chemical simulations of the CD spectra were able to reproduce the experimentally observed sign patterns and demonstrate that the inversion of chirality originated from different binding arrangements of N-acetyl-L-cysteine and L-homocysteine-CdSe to the QD surface. These efforts may allow the prediction of the ligand-induced chiroptical activity of QDs by calculating the specific binding modes of the chiral capping ligands. Combined with the large pool of available chiral ligands, our work opens a robust approach to the rational design of chiral semiconducting nanomaterials.