With no government enforcement from the Chinese or the British aside from a few raids by the Hong Kong Police, the walled city became a haven for crime and drugs. It was only during a 1959 trial for a murder that occurred within the walled city that the Hong Kong government was ruled to have jurisdiction there.
The KMT repeatedly sent requests to reclaim the entire region but Imperial Britain pretty much refused (they proposed a ton of alternative solutions) and didn't govern it either. So yes, it's Imperial Britain's fault. Since the day Britain agreed to transfer the territory to the CCP there was a declared intent to demolish the place.