To work in XPages without either Upgrade Pack 1 for 8.5.3 or the Ext. library severely handicaps your development capabilities. I would strongly recommend you go to the powers that be and say you at least need the IBM supported Upgrade Pack 1 for your domino 8.5.3 servers. IBM provides it and "supports" it. That really should be good enough for the typical pain in the butt Admin.
however - Even though the Ext. library is "not supported" - in fact it has better support because it gets bug fixes where the upgrade pack did not. Those fixes are done by the core xpages dev team so to me, the "unsupported" Ext. Library has much better support overall then the "supported" Upgrade pack.
Domino 9 and 9.01 has the ext. library built it so that would be the preferred solution for you though even there you should use the Ext. library for the improved mobile features.
I just don't get companies that tell you to develop but then are unwilling to provide the tools to allow you do do the developing.
Anyway... enough about that...
I can use Dojo to make a dialogbox but dojo will move the box outside of the form tags i believe. So it then can't be submitted back to the server. The "ZetaOne" code was done by Jeremy Hodge and that's designed to move it back inside the form tags. I guess that post you linked to is heading in the right direction... I don't remember all the custom controls... I remember a function in a script library that just needed to be called.... but that was a while ago and I might not be remembering it right.
good luck!!!
Dave