This depends on how you are doing your validation.
If you are using the standard validators that are properties of the xPages controls, then you can simply set the message property for the validator, and then use Display Error or Display Errors controls to display the message when validation fails (i.e. your code returns false).
If however you are trying to create your own SSJS in a script library to validate then add the following function to your code:
function postValidationError(control, msg) {
if ((typeof msg) != "string")
return;
var msgObj = new javax.faces.application.FacesMessage(
javax.faces.application.FacesMessage.SEVERITY_ERROR, msg, msg);
facesContext.addMessage(control.getClientId(facesContext), msgObj);
control.setValid(false);
}
This will send the message you pass in to the Display Error control for the field you are validating.