I'm playing on a potato and I had the EXACT same problem, 30fps basically everywhere except taverns which dropped below 10. I think this has to do with the high concentration of shadows/NPCs in a small area. Here are a few things you can do that helped me a lot, and you can mix and match to see what works best.
1. Turn off interior shadows.
This one gives the most immediate framerate in taverns specifically.
2. Reduce all graphics settings.
This is another obvious one but setting graphics to fastest rather than fantastic can certainly help.
3. Reduce screen resolution.
Turning down the resolution from 1080p to say 1200x720 can give a huge boost, and even got my exterior framerate up to 60fps. It may seem counter intuitive to use a resolution lower than your monitor's native resolution, but considering that without graphics mods the textures and models are already low-res, there's really not that much difference visually, considering the massive increase in performance.
4. Retro-rendering mode
This has the most modest FPS improvement but can help with things none the less. It also fits the low-res aesthetic of the game better assuming you're not using graphics mods.
I hope this helps!
I know this post is old but I thought I'd reply anyway for future visitors.