There actually is a performance penalty so it likely is loading slower for people.
Edit: It seems that a header needs to be set with HTTPS to allow caching. I'm not sure if this was done when the change happened, but if it's not set to allow caching, that would actually explain significantly slower load times. (Since you would need to download all the content you are opening every time.) Fortunately if this is a problem, it seems to be an easily fixable one.
Edit 2: In windows you can open up PC settings, type in internet options and then from there go into the advanced tab. (In previous versions of windows it should be in the control panel.) Scroll down to security and check to see if the "Do not save encrypted pages to disk" is checked. If it is, uncheck it. MS
should cache
regardless of HTTPS if it's unchecked.