The built in REST API is a little limiting for sure.
I do think there's a few more options then just "Java Servlets" - and I'm not exactly sure what your colleague is actually talking about there. I guess you can do actual Java servlets somehow in the classic domino web programming.
But with XPages you could pull off you're own API a couple different ways. There's some build in controls in the Ext. Library that let's you get a handle on things. Another means would be in creating "XAgents". That's pretty well documented I think. Also with Java inside the .nsf you can create an actual Servlet - which is really an XAgent without the UI Component for the most part. That's been blogged about lately- especially by Eric McCormick I believe.
So there should be several options available to you.