Colloids with lobed architectures have been shown to self-assemble into promising porous structures with potential biomedical applications. The synthesis of these colloids via experiments can be tuned to vary the number and the position of the lobes. However, the polydispersity involving the numbers, sizes, and the dispositions of lobes, that is often observed in particle designs, can significantly affect their self-assembled structures. In this work, we go beyond the uniform lobe size conditions commonly considered in molecular simulations, and probe the effect of polydispersity due to non-uniform lobe sizes by studying self-assembly in three experimentally observable designs of lobed particles (dumbbell, two lobes; trigonal planar, three lobes; and tetrahedral, four lobes), using coarse-grained Langevin dynamics simulations in the NVT ensemble. With increasing polydispersity, we observed the formation of a crystalline structure from a disordered state for the dumbbell system, and a loss of order in the crystalline structures for the trigonal planar system. The tetrahedral system retained a crystalline structure with only a minor loss in compactness. We observed that the effect of polydispersity on the self-assembled morphology of a given system can be minimized by increasing the number of lobes. The polydispersity in the lobe size may also be useful in tuning self-assemblies toward desired structures.