||In this work we consider the effects of gravitons and their fluctuations on the dynamics of two masses using the Feynman-Vernon influence functional formalism, applied earlier to nonequilibrium quantum field theory [Calzetta and Hu, Nonequilibrium Quantum Field Theory (Cambridge University Press, Cambridge, England, 2008)] and semiclassical stochastic gravity [Hu and Verdaguer, Semiclassical and Stochastic Gravity: Quantum Field Effects on Curved Spacetime (Cambridge University Press, Cambridge, England, 2020)], and most recently, to this problem by Parikh et al., [Phys. Rev. Lett. 127, 081602 (2021); Phys. Rev. D 104, 046021 (2021)]. The Hadamard function of the gravitons yields the noise kernel acting as a stochastic tensorial force in a Langevin equation governing the motion of the separation of the two masses. The fluctuations of the separation due to the graviton noise are then solved for various quantum states including the Minkowski vacuum, thermal, coherent and squeezed states. The previous considerations of Parikh et al. are only for some selected modes of the graviton, while in this work we have included all graviton modes and polarizations. We comment on the possibility of detecting these fluctuations in primordial gravitons using interferometors with long baselines in deep space experiments.