I am working on a project which uses dojox enhanced grids extensively. Typically we populate the grids by using agents (Lotusscript) which return json into a store. The store is bound to the grid. I am not using XPages though, working in R7. The data stores take a url parameter for the data, eg:
var exampleStore = new dojo.data.ItemFileWriteStore({url:'agentName?openagent&parm1=x&parm2=y'});
the store can be assigned to the grid and a 'fetch' will load the data into the store; and the onComplete param renders the grid and displays the data:
exampleGrid.store = exampleStore;
exampleGrid.store.fetch({onComplete:exampleGrid.render()});
These actions are addOnLoad or onClick typically.
As you have noticed the RestService from the viewItemFileService is not discriminating enough for your purposes. A LotusScript agent should discriminate data to your hearts content. A readers field on documents should restrict the documents to the contents of the readers field. If the server can see the documents but the logged on user cannot see the data then I imagine you will get the type of erroneous result you are seeing. I don't know enough about what you are doing or the type of components you are using but make sure any service/agent is running as a Web User rather than as the server.
Quite possibly off-track, and it doesn't answer your JQuery question, however what I mean is that Dojo can do the job.