Maybe there is a problem with rsp (Roku Server Protocol). Both FirePlay and the Fireflyclient (java) uses rsp. Itunes uses the DAAP-protocol.
If you browse to http://www.yourserver.com:3689/index.html what does it say under “RSP”?
Mine looks like this:
