I have been able to recreate your problem on an 8.5.3 server/client. I also managed to solve it using one of the URL styles I posted previously (the first one, the other two I posted had spurious / before the ?OpenDocument).
I did the following:
1. Created a new db based on 8.5.3 Discussion template.
2. Created a main document in the db in the Notes client.
3. Opened the db in Firefox.
4. Copied the url to the doc.
5. Opened a new tab and pasted the url in.
6. Changed the protocol from http:// to notes://
7. Voila. DB opens at default view...
I then changed the url to be of the form:
notes://server/db.nsf/UNID?OpenDocumentand the document opened as expected.
If yours is still playing up after trying this way, it would be prudent to verify that the document you are trying to get to does exists and is accessible by the user you are logged in as.