Right so you are disregarding all of the new security fixes etc for an older version just based on that assumption in your post.
You not taken into consideration anything else that could affect the poor loading speeds that you seem to have got now, like how you have set up the site so that your dom size is too large and slowing things down, inefficient usage of images affecting speed, your own hosting provider used, too many other plugins installed that you are not using and therefore redundant, not making use of a caching plugin.....
I would definitely not go backwards in versions regardless of your so-called benefits when a lot of your so-called issues will rest with your own setup as well.