Bingo - In my case the modified door had some nuts acting as spacers on the other side which meant the winder assembly sat too short for the winder mechanism to grab onto. You can also see someone has cut into the door frame to allow for the 240z assembly to be fitted into a 260z door. Coupled with incorrect fasteners which means the whole assembly "floats" when you turn the crank (not fixed in place) and has play, you get a less than ideal window winding experience.
It worked ok with the custom door trims the previous owner had in the car as they sort of sunk inwards as there was more give in the card backings. But once I put proper OEM trims in I couldn't get the winder back on!
We have removed the nuts acting as spacers as a temp fix - which brought the winder forward (sticking out from the door trim), but the window is difficult to wind up and down, because of the reasons mentioned above.
I always wondered why that was and what could be done to improve it, but now that we've determined then doors are wrong it's obvious why the planetary gears are not aligned ideally.
I'll be putting a 240z door back in at some point, since I couldn't live with a hack job like that on 1 of my cars. Ok on a race car, not so good on a good street car.
What I'll do with a hacked 260z door later, I don't know? Might put it on the wall as wall art.