I just tried a simple "window.close();" in client-side JavaScript for a button, but that does not seem to do anything. Surprising.
Another more difficult way would be to use our XPages2Eclipse product (http://xpages2eclipse.mindoo.com).
The following code snippet (in SSJS) grabs a handle on the currently active tab (called "Perspective" in Eclipse) and closes it:
var conn=X2E.createConnection();
var pUI=com.x2e.PlatformUIAPI.getUI(conn);
var wb=pUI.getWorkbench();
var activeWindow=wb.getActiveWorkbenchWindow();
var activePage=activeWindow.getActivePage();
var perspective=activePage.getPerspective();
var saveEditors=true;
var closePage=false;
activePage.closePerspective(perspective, saveEditors, closePage);
API documentation, tutorials and samples can be found here: http://xpages2eclipse.mindoo.com/web/x2ewiki.nsf