Check out the NotesIn9, I think it's episode 25 which talks about selecting documents from a repeat control. http://notesin9.com/index.php/2011/04/01/notesin9-025-selecting-documents-from-a-repeat-control/

 

It's effectively the same technique you need for a data table. I also wrote a series on validation of checkboxes, which has the whole aspect of setting/unsetting properties in a scoped variable based on whether the checkbox is ticked or not. It was some time ago, but covers various version of XPages and may be of interest http://hermes.intec.co.uk/Intec/Blog.nsf/archive?openview&title=Validation%20Tutorial&type=cat&cat=Validation%20Tutorial.

 

If you need to set the checkbox to checked or not, you can just set the defaultChecked property to check against the scoped variable you set in the onClick.