Bill,
By definition controlling rendered will perform better then display:none because XPages will ignore non rendered code at various phases of the lifecycle. display:non means xpages will still be doing all the work of those panels - they just don't show up.
My understanding on loaded vs. rendered is that loaded is only evaluated during page load. Not refresh. So if you set loaded to false - it's not coming back until you do a full refresh of the page. I could be wrong though.
It depends on what you need functionality. If you must have access to those panels then you probable can't use rendered. Because basically if it's not being rendered then it doesn't exist. So you can't refresh or otherwise interact with a non rendered component.
If you need access to "hidden panels" then you might want to think about what you're doing. Is it hidden for convenience ? or hidden as a means of security. And we all know that "Security through obscurity is not security". :-)
Possibly you're business logic shouldn't be in the panel but be in something like a managed bean. Then you could have access to all the data fields while not rendering specific panels.
Just a thought.