I have ran an in-place upgrade from 8.5.3/32, to 9.0/64 with great success. The 8.5.3 was originally running on Win 2003/32. This was then moved to a Win 2008R2/64 platform (Run the Domino 8.5.3 install, replace the data catalog with the one from the old server, copy NOTES.INI and make the necessary changes to reflect the new server's directory structure). Make sure that the new server is running well with 8.5.3. Then, I ran the 9.0 install. The install program found the previous installation and automatically upgraded the old installation to 9.0/64. Quite amazing! After a server restart (just to be safe), Domino 9 went up and running like a breeze.

Having said that, it's always a good thing to have a new and fresh Domino installation without any trace of previous versions. My first server above was a test sever, the next server to be upgraded was the production server. This server was set up with Domino 9.0/64 from the beginning. Then, I moved all the databases from the old data directory to the new server. But before this was done, I ran a "cleanup" procedure against the old data directory, removing all the old ntf's, all the old system-nsf's which had been replaced with newer ones on R9. And of course, again you have to make the necessary changes in NOTES.INI.

Regarding Traveler, I would uninstall this before doing anything else. Then, make a new, fresh installation of the 64 bit version after you have the R9 server up and running. Traveler itself does not carry any data itself (except for the sync history in the Derby database), and the migration will affect users only marginally. (A full resync will be performed the first time the mobile devices connect to Traveller)