Performing microstructure sensitive metal-forming simulations is widely recognized as a computational challenge because of the need to store large sets of state variables related to microstructure data. We present a rigorous methodology for the compaction of microstructural data associated with a material point and show that the statistical distributions of microstructure of any size can be compacted to several hundred grains. The methodology is based on the spectral representation of microstructure distribution functions through the use of generalizes spherical harmonics. Subsequently, we present a computational framework aimed at dramatically reducing time needed for microstructure sensitive simulations of metal forming processes. The framework is based on a combination of the recently developed numerical implementations of crystal plasticity models in the spectral representation for obtaining the response of single crystals and specialized computer hardware that integrates a graphics-processing unit. We apply these two methodologies on a plane strain compression case study and obtain speedup factors exceeding three orders of magnitude.