The Traveler server connects to the Domino Mail Server of the user with NRPC and with its ID so the Traveler Server does have all the rights to the Directory Assistance Database and Directories it points to.....
No way to manage that you do have to add readers/authors fields on the document pointed by directory assistance to hide some people to eh traveler user. But is it worth the job?