FireFly Media Server (formerly mt-daapd) › Firefly Media Server Forums › Firefly Media Server › General Discussion › Scanning outside of the specified directories?
- This topic has 18 replies, 4 voices, and was last updated 11 years, 7 months ago by
EVILRipper.
-
AuthorPosts
-
28th May 2009 at 11:42 pm #2907
Anonymous
InactiveI am not entirely sure, but I have the feeling that svn-1696 scans outside of the directories I specified.
OS: Kubuntu 8.04
Architecture: i686The directories are always /media/something for me.
Still, enabling level 9 logging and glancing there, I see it scans the entire “/”.
Is it just my impression, or “Found /path/to/unspecified/directory” is really the proof it scans /path/to/unspecified/directory?
It lacks logic.
If
mp3_dir = /media/storage,/media/sda9
then under NO circumstance should it scan in /home/user!
Log excerpt:
2009-05-29 02:28:21 (b74c46c0): Found snpstd.. recursing
2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/usnpstd.exe
2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/h71rstd.reg
etc.
29th May 2009 at 7:16 am #18555EVILRipper
ParticipantIt scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.
29th May 2009 at 8:40 am #18556Anonymous
Inactive@EVILRipper wrote:
It scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.
I figured it scans recursively, but it should scan only the specified directories.
I have mounted the partition /dev/sda9 into the /media/sda9 location.
If by “when there is a symbolic link somewhere in those directories, pointing somewhere else” you mean that if there is a symlink somewhere in /media/sda9 pointing to the, say, “/home/user/something” location, then it would be OK if it would only scan “/home/user/something” (if such a location exists).
What I find counter-intuitive (and devious haha) is the fact that it scans all /home/user.
Surely, the majority of people would perceive the expression “recursing scan” as a scan done in “the/specified/directory/” and “the/specified/directory/and/all/its/subdirectories/”, i.e. a “downward” recurse, not an “upward” one.
It’s confusing to perceive a “recursing scan” in “the/specified/directory/” as a scan done also in “/the/parent/of/the/specified/directory”.
Wait, does what I’m saying here make sense? : )
29th May 2009 at 9:59 am #18557EVILRipper
ParticipantUmm, I’m not sure. 🙂
A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.
29th May 2009 at 10:02 am #18558Anonymous
Inactive@EVILRipper wrote:
Umm, I’m not sure. 🙂
A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.Oh, okay, then it means I must have been too obscure. Coherent wording is not my gift.
However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.
OK, I see. In clear terms, what I wanted to say was that if there’s a symbolic link that points to /home/user/temp, then only /home/user/temp should be scanned. However, from my logs, I got that /home/user was scanned, entirely.
29th May 2009 at 10:41 am #18559EVILRipper
Participantfind /media/sda9 -type l
This should find all symbolic links within /media/sda9.
When results were found, go to the directory containing that file/folder.ls -l
Will show where the symbolic links points at.
29th May 2009 at 7:10 pm #18560Anonymous
Inactive@EVILRipper wrote:
find /media/sda9 -type l
This should find all symbolic links within /media/sda9.
When results were found, go to the directory containing that file/folder.ls -l
Will show where the symbolic links points at.
I know that. What I am trying to point out, EVILRipper, is the fact that no symbolic link in the directories specified as “mp3_dir” points to my home directory (“/home/alex”).
Look…
[email protected]:/tmp# grep mp3_dir /etc/mt-daapd.conf
mp3_dir = /media/storage,/media/sda9So now let’s take a look at those two partitions.
[email protected]:/tmp# for mp3dir in storage sda9 ; do find "/media/${mp3dir}" -type l -exec ls -l {} ; >> symlinks_${mp3dir}.txt ; echo "$mp3dir: " ; grep "/home/alex" symlinks_${mp3dir}.txt ; done
storage:
sda9:
lrwxrwxrwx 1 alex alex 39 2007-10-14 18:44 /media/sda9/other/___non-metal_-_non-ambient___/Vangelis -> /media/sda6/home/alex/Desktop/Vangelis/
I think it’s pretty straightforward. What I did was to create two files called symlinks_storage.txt and symlinks_sda9.txt which contain the output of that `find’ command. That `find’ finds all symbolic links in those directories and does a `ls -l’ on them. The next thing I did was to grep through the text files to search for any occurrence of “/home/alex” which is my home partition and which should NOT get scanned.
The symlink you see there, in sda9, pointing to /media/sda6/home/alex/Desktop/Vangelis, is an old and broken symbolic link. There is no /media/sda6 location on my system.
So then, why does mt-daapd scan in “/home/alex” after all?
29th May 2009 at 8:05 pm #18561EVILRipper
ParticipantI’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..
29th May 2009 at 8:55 pm #18562Anonymous
Inactive@EVILRipper wrote:
I’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..
Well, I grep-ed on “/home/alex” because I noticed a disturbing number of lines in the log file relating to scans in /home/alex.
Other than that, I couldn’t have missed a symlink to anything containing “/home/alex” using grep and yes that broken symlink was indeed the only thing (it pointed to a long-discarded partition which had a then-home-directory mounted).
For the sake of it, I removed the broken symlink. Now there’s no such thing.
So it shouldn’t scan /home/alex.
Well… did the same thing: emptied log, deleted database, started mt-daapd.
It still scans /home/alex. :-/
30th May 2009 at 4:28 pm #18564fizze
ParticipantCare to post your mt-daapd.conf?
What version of firefly do you use?30th May 2009 at 5:02 pm #18565Anonymous
Inactive@fizze wrote:
Care to post your mt-daapd.conf?
Sure.
[email protected]:/home/alex# grep -v ^# /etc/mt-daapd.conf | grep -v ^$
[general]
web_root = /usr/share/mt-daapd/admin-root
port = 3689
admin_pw = mt-daapd
db_type = sqlite3
db_parms = /usr/var/cache/mt-daapd
mp3_dir = /media/storage,/media/sda9
servername = Firefly %v on %h
runas = mt-daapd
extensions = .mp3,.m4a,.m4p,.ogg,.flac,.mpc
scan_type = 2
logfile = /var/log/mt-daapd.log
debuglevel = 9
[plugins]
plugin_dir = /usr/lib/mt-daapd/plugins
plugins = rsp.so,ssc-ffmpeg.so
[scanning]
process_playlists = 1
process_itunes = 1
process_m3u = 1@fizze wrote:
What version of firefly do you use?
Firefly Media Server: Version svn-1696
1st June 2009 at 8:09 am #18563fizze
ParticipantOk, that looks reasonable.
Care to post your mount status? 😉
What fs types do you use?I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
Could it be there are hardlinks that cause firefly to trip?1st June 2009 at 6:37 pm #18566Anonymous
Inactive@fizze wrote:
Ok, that looks reasonable.
Care to post your mount status? 😉
What fs types do you use?Sure… I have two SATA HDDs. The Linux partitions are ext3.
mount output:
/dev/sda6 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-24-generic/volatile type tmpfs (rw)
/dev/sda7 on /home type ext3 (rw,relatime)
/dev/sda8 on /media/sda8 type ext3 (rw,relatime)
/dev/sda9 on /media/sda9 type ext3 (rw,relatime)
/dev/sdb2 on /media/storage type ext3 (rw,relatime)
/dev/sdb1 on /media/winnew type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
/dev/sda1 on /media/winold type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)The relevant parts (sda9, storage and home) are written in bold.
@fizze wrote:
I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
Could it be there are hardlinks that cause firefly to trip?I am not entirely sure I understand what you mean by hardlinks. Are you referring to the names of the devices – or to the devices themselves?
As for any symlinks pointing to the home partition, there’s none. (I even removed the only broken symbolic link pointing there.)
Next, I followed your suggestion. Maybe there is something wrong with mt-daapd and mp3_dir referring to locations on two different hard drivers.
So I replaced
mp3_dir = /media/storage,/media/sda9
with
mp3_dir = /media/storage
Re-ran the test… it all went fine. However, when I added the second location (/media/sda9), it began scanning /home/alex (and also crashed, same as before).
Then I started it only with
mp3_dir = /media/sda9
Same erratic behavior (scanning in home, crash).
The interesting bit is that, as you can see, both /media/sda9/ and /home are parts of the same physical hard drive.
It shouldn’t make any difference, but just to make sure:
fdisk -l
Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x8f8004b0
Device Boot Start End Blocks Id System
/dev/sda1 * 1 7833 62918541 7 HPFS/NTFS
/dev/sda2 7834 48641 327790260 5 Extended
/dev/sda5 7834 8029 1574338+ 82 Linux swap / Solaris
/dev/sda6 8030 10640 20972826 83 Linux
/dev/sda7 10641 13251 20972826 83 Linux
/dev/sda8 13252 15862 20972826 83 Linux
/dev/sda9 15863 48641 263297286 83 Linux
Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000c54f2
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 15680 125949568+ 7 HPFS/NTFS
/dev/sdb2 15681 60801 362434432+ 83 Linux
I also have enough free space on all devices:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 20G 3.4G 16G 18% /
varrun 1014M 300K 1014M 1% /var/run
varlock 1014M 0 1014M 0% /var/lock
udev 1014M 72K 1014M 1% /dev
devshm 1014M 0 1014M 0% /dev/shm
lrm 1014M 40M 975M 4% /lib/modules/2.6.24-24-generic/volatile
/dev/sda7 20G 13G 6.3G 67% /home
/dev/sda8 20G 19G 312M 99% /media/sda8
/dev/sda9 248G 218G 18G 93% /media/sda9
/dev/sdb2 341G 281G 43G 87% /media/storage
/dev/sdb1 121G 46G 75G 39% /media/winnew
/dev/sda1 61G 15G 46G 25% /media/winold
There are just 56 symbolic links on the whole sda9 partition… most are broken, pointing to where-some-mount-points-used-to-be. If you think it’s relevant, I’ll post that. Although… there’s nothing even mentioning “home” there. :-/
Well, at least we’re narrowing the field. There’s obviously something wrong in some cases when the media directories are on different physical hard drives.
1st June 2009 at 9:15 pm #18567fizze
ParticipantOk, that all looks good.
Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.
Maybe broken hardlinks can point across partitions? hmmm. Thats a long shot, anyway, and it seems unlikely.But clearly something on sda9 seems to throw firefly off.
You did erase your songs.db at a certain point, right?
Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.
1st June 2009 at 10:06 pm #18568Anonymous
Inactive@fizze wrote:
Ok, that all looks good.
Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.Oh, I see, thanks for clarifying this.
@fizze wrote:
You did erase your songs.db at a certain point, right?
Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.
Yes, I was erasing both songs3.db and songs3(something)journal each time I was starting the test again.
However, I decided to delete all the broken symbolic links… after this, it worked.
@fizze wrote:
But clearly something on sda9 seems to throw firefly off.
You got that right. Here’s what happened: I took a better look at all the symbolic links in /media/sda9 and found a really broken one (I don’t know how it actually got there in the first place)…
lrwxrwxrwx 1 root root 1 2007-11-11 11:45 /media/sda9/backup/b4gg/root/.wine/dosdevices/z: -> /
Some weird backup of the entire /root home directory before some old, forgotten install.
So I uhm am sorry for all the trouble, it works alright… Thanks for the feedback and for taking the time to debug me.
This is embarrassing. 😛
-
AuthorPosts
- The forum ‘General Discussion’ is closed to new topics and replies.