How come TTFB reduces so much, and how can I improve TTFB back to 50ms?

First of all, I am using this on AWS (thank you, I love it!):
WordPress with LiteSpeed Cache (Powered OpenLiteSpeed)
(1) | 1.4.42-1 Previous versions | By LiteSpeed Technologies Inc.
$0.00/hr for software + AWS usage fees
Linux/Unix, Ubuntu 18.04.1 | 64-bit (x86) Amazon Machine Image (AMI) | Updated: 1/15/19

So heres where I am confused:
On a fresh installation, the TTFB is great <50ms.

As soon as I load my backup of an existing site which is not that heavy (sorry dont want to share my URL in public):
41 requests
400kb transferred
1.6mb resources
Finish loading in 3.5s to 4.5s
TTFB 500ms to 1.5s

Ive tried:
disabling all plugins - small difference
disabling theme - small difference
optimised the database using litespeed plugin
Redis is on (memcached connection shows as failed upon fresh installation of OLS)

What else could be the cause of this massive (1s) TTFB increase?