Scanning outside of the specified directories?

[Anonymous]

I am not entirely sure, but I have the feeling that svn-1696 scans outside of the directories I specified.

OS: Kubuntu 8.04
Architecture: i686

The directories are always /media/something for me.

Still, enabling level 9 logging and glancing there, I see it scans the entire “/”.

Is it just my impression, or “Found /path/to/unspecified/directory” is really the proof it scans /path/to/unspecified/directory?

It lacks logic.

If

mp3_dir = /media/storage,/media/sda9

then under NO circumstance should it scan in /home/user!

Log excerpt:

2009-05-29 02:28:21 (b74c46c0): Found snpstd.. recursing

2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/usnpstd.exe

2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/h71rstd.reg

etc.

[EVILRipper]

It scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.

[Anonymous]

@EVILRipper wrote:

It scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.

I figured it scans recursively, but it should scan only the specified directories.

I have mounted the partition /dev/sda9 into the /media/sda9 location.

If by “when there is a symbolic link somewhere in those directories, pointing somewhere else” you mean that if there is a symlink somewhere in /media/sda9 pointing to the, say, “/home/user/something” location, then it would be OK if it would only scan “/home/user/something” (if such a location exists).

What I find counter-intuitive (and devious haha) is the fact that it scans all /home/user.

Surely, the majority of people would perceive the expression “recursing scan” as a scan done in “the/specified/directory/” and “the/specified/directory/and/all/its/subdirectories/”, i.e. a “downward” recurse, not an “upward” one.

It’s confusing to perceive a “recursing scan” in “the/specified/directory/” as a scan done also in “/the/parent/of/the/specified/directory”.

Wait, does what I’m saying here make sense? : )

[EVILRipper]

Umm, I’m not sure. 🙂
A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.

However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.

[Anonymous]

@EVILRipper wrote:

Umm, I’m not sure. 🙂
A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.

Oh, okay, then it means I must have been too obscure. Coherent wording is not my gift.

However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.

OK, I see. In clear terms, what I wanted to say was that if there’s a symbolic link that points to /home/user/temp, then only /home/user/temp should be scanned. However, from my logs, I got that /home/user was scanned, entirely.

[EVILRipper]

find /media/sda9 -type l

This should find all symbolic links within /media/sda9.
When results were found, go to the directory containing that file/folder.

ls -l

Will show where the symbolic links points at.

[Anonymous]

@EVILRipper wrote:

find /media/sda9 -type l

This should find all symbolic links within /media/sda9.
When results were found, go to the directory containing that file/folder.

ls -l

Will show where the symbolic links points at.

I know that. What I am trying to point out, EVILRipper, is the fact that no symbolic link in the directories specified as “mp3_dir” points to my home directory (“/home/alex”).

Look…

[email protected]:/tmp# grep mp3_dir /etc/mt-daapd.conf

mp3_dir = /media/storage,/media/sda9

So now let’s take a look at those two partitions.

[email protected]:/tmp# for mp3dir in storage sda9 ; do find "/media/${mp3dir}" -type l -exec ls -l {} ; >> symlinks_${mp3dir}.txt ; echo "$mp3dir: " ; grep "/home/alex" symlinks_${mp3dir}.txt ; done

storage:

sda9:

lrwxrwxrwx 1 alex alex 39 2007-10-14 18:44 /media/sda9/other/___non-metal_-_non-ambient___/Vangelis -> /media/sda6/home/alex/Desktop/Vangelis/

I think it’s pretty straightforward. What I did was to create two files called symlinks_storage.txt and symlinks_sda9.txt which contain the output of that `find’ command. That `find’ finds all symbolic links in those directories and does a `ls -l’ on them. The next thing I did was to grep through the text files to search for any occurrence of “/home/alex” which is my home partition and which should NOT get scanned.

The symlink you see there, in sda9, pointing to /media/sda6/home/alex/Desktop/Vangelis, is an old and broken symbolic link. There is no /media/sda6 location on my system.

So then, why does mt-daapd scan in “/home/alex” after all?

[EVILRipper]

I’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..

[Anonymous]

@EVILRipper wrote:

I’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..

Well, I grep-ed on “/home/alex” because I noticed a disturbing number of lines in the log file relating to scans in /home/alex.

Other than that, I couldn’t have missed a symlink to anything containing “/home/alex” using grep and yes that broken symlink was indeed the only thing (it pointed to a long-discarded partition which had a then-home-directory mounted).

For the sake of it, I removed the broken symlink. Now there’s no such thing.

So it shouldn’t scan /home/alex.

Well… did the same thing: emptied log, deleted database, started mt-daapd.

It still scans /home/alex. :-/

[fizze]

Care to post your mt-daapd.conf?
What version of firefly do you use?

[Anonymous]

@fizze wrote:

Care to post your mt-daapd.conf?

Sure.

[email protected]:/home/alex# grep -v ^# /etc/mt-daapd.conf | grep -v ^$

[general]

web_root = /usr/share/mt-daapd/admin-root

port = 3689

admin_pw = mt-daapd

db_type = sqlite3

db_parms = /usr/var/cache/mt-daapd

mp3_dir = /media/storage,/media/sda9

servername = Firefly %v on %h

runas = mt-daapd

extensions = .mp3,.m4a,.m4p,.ogg,.flac,.mpc

scan_type = 2

logfile = /var/log/mt-daapd.log

debuglevel = 9

[plugins]

plugin_dir = /usr/lib/mt-daapd/plugins

plugins = rsp.so,ssc-ffmpeg.so

[scanning]

process_playlists = 1

process_itunes = 1

process_m3u = 1

@fizze wrote:

What version of firefly do you use?

Firefly Media Server: Version svn-1696

[fizze]

Ok, that looks reasonable.
Care to post your mount status? 😉
What fs types do you use?

I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
Could it be there are hardlinks that cause firefly to trip?

[Anonymous]

@fizze wrote:

Ok, that looks reasonable.
Care to post your mount status? 😉
What fs types do you use?

Sure… I have two SATA HDDs. The Linux partitions are ext3.

mount output:

/dev/sda6 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-24-generic/volatile type tmpfs (rw)
/dev/sda7 on /home type ext3 (rw,relatime)
/dev/sda8 on /media/sda8 type ext3 (rw,relatime)
/dev/sda9 on /media/sda9 type ext3 (rw,relatime)
/dev/sdb2 on /media/storage type ext3 (rw,relatime)
/dev/sdb1 on /media/winnew type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
/dev/sda1 on /media/winold type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

The relevant parts (sda9, storage and home) are written in bold.

@fizze wrote:

I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
Could it be there are hardlinks that cause firefly to trip?

I am not entirely sure I understand what you mean by hardlinks. Are you referring to the names of the devices – or to the devices themselves?

As for any symlinks pointing to the home partition, there’s none. (I even removed the only broken symbolic link pointing there.)

Next, I followed your suggestion. Maybe there is something wrong with mt-daapd and mp3_dir referring to locations on two different hard drivers.

So I replaced

mp3_dir = /media/storage,/media/sda9

with

mp3_dir = /media/storage

Re-ran the test… it all went fine. However, when I added the second location (/media/sda9), it began scanning /home/alex (and also crashed, same as before).

Then I started it only with

mp3_dir = /media/sda9

Same erratic behavior (scanning in home, crash).

The interesting bit is that, as you can see, both /media/sda9/ and /home are parts of the same physical hard drive.

It shouldn’t make any difference, but just to make sure:

fdisk -l



Disk /dev/sda: 400.0 GB, 400088457216 bytes

255 heads, 63 sectors/track, 48641 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x8f8004b0



Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *           1        7833    62918541    7  HPFS/NTFS

/dev/sda2            7834       48641   327790260    5  Extended

/dev/sda5            7834        8029     1574338+  82  Linux swap / Solaris

/dev/sda6            8030       10640    20972826   83  Linux

/dev/sda7           10641       13251    20972826   83  Linux

/dev/sda8           13252       15862    20972826   83  Linux

/dev/sda9           15863       48641   263297286   83  Linux



Disk /dev/sdb: 500.1 GB, 500107862016 bytes

255 heads, 63 sectors/track, 60801 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x000c54f2



Device Boot      Start         End      Blocks   Id  System

/dev/sdb1   *           1       15680   125949568+   7  HPFS/NTFS

/dev/sdb2           15681       60801   362434432+  83  Linux

I also have enough free space on all devices:

 df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda6              20G  3.4G   16G  18% /

varrun               1014M  300K 1014M   1% /var/run

varlock              1014M     0 1014M   0% /var/lock

udev                 1014M   72K 1014M   1% /dev

devshm               1014M     0 1014M   0% /dev/shm

lrm                  1014M   40M  975M   4% /lib/modules/2.6.24-24-generic/volatile

/dev/sda7              20G   13G  6.3G  67% /home

/dev/sda8              20G   19G  312M  99% /media/sda8

/dev/sda9             248G  218G   18G  93% /media/sda9

/dev/sdb2             341G  281G   43G  87% /media/storage

/dev/sdb1             121G   46G   75G  39% /media/winnew

/dev/sda1              61G   15G   46G  25% /media/winold

There are just 56 symbolic links on the whole sda9 partition… most are broken, pointing to where-some-mount-points-used-to-be. If you think it’s relevant, I’ll post that. Although… there’s nothing even mentioning “home” there. :-/

Well, at least we’re narrowing the field. There’s obviously something wrong in some cases when the media directories are on different physical hard drives.

[fizze]

Ok, that all looks good.
Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.
Maybe broken hardlinks can point across partitions? hmmm. Thats a long shot, anyway, and it seems unlikely.

But clearly something on sda9 seems to throw firefly off.

You did erase your songs.db at a certain point, right?
Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?

Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.

[Anonymous]

@fizze wrote:

Ok, that all looks good.
Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.

Oh, I see, thanks for clarifying this.

@fizze wrote:

You did erase your songs.db at a certain point, right?
Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?

Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.

Yes, I was erasing both songs3.db and songs3(something)journal each time I was starting the test again.

However, I decided to delete all the broken symbolic links… after this, it worked.

@fizze wrote:

But clearly something on sda9 seems to throw firefly off.

You got that right. Here’s what happened: I took a better look at all the symbolic links in /media/sda9 and found a really broken one (I don’t know how it actually got there in the first place)…

lrwxrwxrwx 1 root root 1 2007-11-11 11:45 /media/sda9/backup/b4gg/root/.wine/dosdevices/z: -> /

Some weird backup of the entire /root home directory before some old, forgotten install.

So I uhm am sorry for all the trouble, it works alright… Thanks for the feedback and for taking the time to debug me.

This is embarrassing. 😛

[Author]

Posts