Venus' rotation is the slowest of all planetary objects in the solar system. It is commonly admitted that it results of the balance between the torques created by the internal gravitational tides and by the atmospheric thermal tides that apply on the planet (Dobrovolskis et Ingersoll, 1980; Leconte et al., 2015; Revol et al., in press). The gravitational tides drive the planet into synchronization (deceleration) while the atmospheric thermal tides tend to accelerate the planet out of this synchronization (Correia et Laskar, 2001). The purpose of this work is to quantify the impact of the internal structure and its past evolution on the gravitational tides and thus on the rotation history of Venus. We determine the tidal gravitational torque from the tidal Love number, k2, and the dissipation function Q, which are computed using a viscoelastic Andrade rheology and interior structure model following the approach of Dumoulin et al. (2017). Using 3D numerical simulations performed as in Gillmann et Tackley (2014) as reference for the past evolution of the thermal profile in Venus' mantle, we compute the gravitational torque since the formation of Venus. Using the present-day atmospheric torque estimated by Leconte et al. (2015) from LMD GCM simulations, we show that the current viscosity structure of Venus must be similar to Earth's, with a viscosity of the lower mantle around 1022-1023 Pa.s to maintain the present rotation state at equilibrium. However, this study also highlights the necessity to have a more intense tidal dissipation in the past to initiate the tidal despinning. This would imply a much hotter mantle during early evolution, with a much lower viscosity. Even considering an upper mantle including 20-30% of partial melt (Kervazo et al., 2021) during the first billion years, our calculations show that slowing down Venus' rotation by internal friction alone appears unrealistic from an initial period shorter than 2-3 days. If Venus had an initial rotation comparable to the Earth or Mars, additional processes are required to start the despinning (e.g. oceanic dissipation, giant impacts).