I'm in Sequence 4, I've recruited the Thieves, Courtesans, and Mercenaries, and I've infiltrated the Castle. I've met with Leonardo (got the dual wrist blades) and he told me about the gun and blueprints. I can see a memory start point down in the Southwest part of the map, but it's past the dotted line area where I can't go, past the ghost-like barriers.
I destroyed all the Borgia towers that I can access, the only ones I haven't are still beyond the barriers. What am I missing? What do I need to do to get to that memory start point. I can't complete sequence 4 until I can.