The cam being retarded is killing it IMO. You are not allowing the engine to demand enough airflow at low engine speeds to fulfill the airflow requirement to generate boost with that turbocharger. Meaning a delay in spool time. This is why you're not seeing any boost until higher engine speeds where the engine is generating enough VE to finally get the turbo going.

You either need to advance the cam back to it's straight up position or have a cam ground that will aid in spool up.

The LS9 cam already has a very late ICL and thus because of that a late IVC event. Especially for a 5.3 liter engine as it was designed for a 376" engine with a positive displacement blower that makes massive off idle torque and very little torque at high RPM. This is one of the main reasons GM designed the cam with such a late IVC event. To aid the smaller positive displacement blower in generating power at higher RPM's. Even when a LS9 camshaft is installed straight up and not retarded as the OP has done, it really isn't the best cam for a turbocharged engine. This late IVC event will really hurt spool time as the engine doesn't move enough airflow to really get a turbo of that size going at low engine speeds.

A custom ground cam with a little more overlap(not a lot, but a little more than the LS9 cam) and an earlier IVC event will help the engine generate more VE from 3000-3500rpm. Which in turn will move more airflow through the engine at these RPM's. This should help generate more compressor speed from the added airflow demand at low engine speeds. Thus aiding spool time.

I wouldn't worry about losing any of the top end you have now either with a different cam. You have an extremely large turbine and compressor wheel for a relatively stock 325" engine. It will have no trouble feeding your engine the airflow it requests at higher engine speeds. Especially with an aftermarket camshaft adding more duration and curtain area at the valve to fill the cylinder more efficiently at higher engine speeds.

 

Thanks everyone.

Cam is straight up now and it is way better, but we also found the turbo to be bad. That's why it did not pick up when first psi was available.

So now I have to choose witch turbo combo to use.
I do have the T4 PT4788 (0.96) from the last year combo but new headers are T6.

This PT4788 should accept Garrett T6 GT47 housings also, so I could buy one and have the benefit of smaller turbine wheel but reduce the BP to managable level with larger A/R.

 

Everyone is spot on with advice.

Set the cam back at 0

Since the converter is going to 3000 rpm at WOT, it will spool. 3600 and 1 psi showed something else was wrong like a stuck open gate..but you found the turbo to be bad so that explained it.

I'm not sure how much power your wanting to make but the 4788 T6 would be a good choice.

 

Basically the question is what A/R turbine housing is needed to get all cfm out of the compressor side of this PT4788. (1000+ rwhp)

Basically I think I have to choose between 1.23 undivided or 1.41 divided.
First is around 550$ new, second one is 250$ new.