Viscoelastic dampers, as supplementary energy dissipation devices, have been used in building structures under seismic excitation or wind loads. Different analytical models have been proposed to describe their dynamic force deformation characteristics. Among these analytical models, the fractional derivative models have attracted more attention as they can capture the frequency dependence of the material stiffness and damping properties observed from tests very well. In this paper, a Fourier-transform-based technique is presented to obtain the fractional unit impulse function and the response of structures with added viscoelastic dampers whose force-deformation relationship is described by a fractional derivative model. Then, a Duhamel integral-type expression is suggested for the response analysis of a fractional damped dynamic system subjected to deterministic or random excitation. Through numerical verification, it is shown that viscoelastic dampers are effective in reducing structural responses over a wide frequency range, and the proposed schemes can be used to accurately predict the stochastic seismic response of structures with added viscoelastic dampers described by a Kelvin model with fractional derivative.