I’ve basically given up at this point on using mt-daapd to stream these video files. The only thing I noticed while comparing packet dumps is that when connecting to the iTunes server, the client actually makes TWO simultaneous connections. The first downloads the file in its entirety. The second begins after about 5 seconds, and downloads the file using bytes ranges (essentially streaming). I’m not sure if the first connection is terminated at some point.
While connecting to mt-daapd, the second connection is never initiated. Not sure why, but I didn’t spend all that much time trying to figure it out. I made a few simple/superficial changes to mt-daapd to try to make mt-daapd appear more like iTunes server in its response headers, but it had no positive effect (e.g. mt-daapd returns audio/mp4 as content-type instead of video/mp4, etc).
Since streaming seems to work just fine (using qt-faststart) over httpd, I just wrote a little website to serve my videos over http (which also makes access via the web easier). Though it would be nice to also share over iTunes if that gets worked out at some point in the future.
In regards to your uncertainty about iTunes streaming and H.264 streaming, I’m certain that iTunes is doing streaming in its sharing of video files. About H.264 vs Xvid, my knowledge in this area is very limited, but I believe that streaming occurs just fine with either codec within iTunes.