Scanning outside of the specified directories?

This topic contains 18 replies, has 4 voices, and was last updated by  EVILRipper 7 years, 11 months ago.

Viewing 15 posts - 1 through 15 (of 19 total)
  • Author
    Posts
  • #2907

    Anonymous

    I am not entirely sure, but I have the feeling that svn-1696 scans outside of the directories I specified.

    OS: Kubuntu 8.04
    Architecture: i686

    The directories are always /media/something for me.

    Still, enabling level 9 logging and glancing there, I see it scans the entire “/”.

    Is it just my impression, or “Found /path/to/unspecified/directory” is really the proof it scans /path/to/unspecified/directory?

    It lacks logic.

    If

    mp3_dir = /media/storage,/media/sda9

    then under NO circumstance should it scan in /home/user!

    Log excerpt:

    2009-05-29 02:28:21 (b74c46c0): Found snpstd.. recursing
    2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/usnpstd.exe
    2009-05-29 02:28:21 (b74c46c0): Found /home/alex/.wine/drive_c/Program Files/Common Files/snpstd/h71rstd.reg
    etc.
    #18555

    EVILRipper
    Participant

    It scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.

    #18556

    Anonymous

    @evilripper wrote:

    It scans recursively. The only reason it should/could scan other directories is when there is a symbolic link somewhere in those directories, pointing somewhere else. For example, if you have mounted /media/sda9 to your own harddisk, it will scan your whole filesystem.

    I figured it scans recursively, but it should scan only the specified directories.

    I have mounted the partition /dev/sda9 into the /media/sda9 location.

    If by “when there is a symbolic link somewhere in those directories, pointing somewhere else” you mean that if there is a symlink somewhere in /media/sda9 pointing to the, say, “/home/user/something” location, then it would be OK if it would only scan “/home/user/something” (if such a location exists).

    What I find counter-intuitive (and devious haha) is the fact that it scans all /home/user.

    Surely, the majority of people would perceive the expression “recursing scan” as a scan done in “the/specified/directory/” and “the/specified/directory/and/all/its/subdirectories/”, i.e. a “downward” recurse, not an “upward” one.

    It’s confusing to perceive a “recursing scan” in “the/specified/directory/” as a scan done also in “/the/parent/of/the/specified/directory”.

    Wait, does what I’m saying here make sense? : )

    #18557

    EVILRipper
    Participant

    Umm, I’m not sure. 🙂
    A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.

    However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.

    #18558

    Anonymous

    @evilripper wrote:

    Umm, I’m not sure. 🙂
    A downward recurse.. I believe there’s no such thing, as that would result in a full filesystem scan.

    Oh, okay, then it means I must have been too obscure. Coherent wording is not my gift.

    However, if there’s a symbolic link that points to /home, all subdirs of /home would be scanned. That’s what I meant.

    OK, I see. In clear terms, what I wanted to say was that if there’s a symbolic link that points to /home/user/temp, then only /home/user/temp should be scanned. However, from my logs, I got that /home/user was scanned, entirely.

    #18559

    EVILRipper
    Participant
    find /media/sda9 -type l

    This should find all symbolic links within /media/sda9.
    When results were found, go to the directory containing that file/folder.

    ls -l

    Will show where the symbolic links points at.

    #18560

    Anonymous

    @evilripper wrote:

    find /media/sda9 -type l

    This should find all symbolic links within /media/sda9.
    When results were found, go to the directory containing that file/folder.

    ls -l

    Will show where the symbolic links points at.

    I know that. What I am trying to point out, EVILRipper, is the fact that no symbolic link in the directories specified as “mp3_dir” points to my home directory (“/home/alex”).

    Look…

    [email protected]:/tmp# grep mp3_dir /etc/mt-daapd.conf
    mp3_dir = /media/storage,/media/sda9

    So now let’s take a look at those two partitions.

    [email protected]:/tmp# for mp3dir in storage sda9 ; do find "/media/${mp3dir}" -type l -exec ls -l {} ; >> symlinks_${mp3dir}.txt ; echo "$mp3dir: " ; grep "/home/alex" symlinks_${mp3dir}.txt ; done
    storage:
    sda9:
    lrwxrwxrwx 1 alex alex 39 2007-10-14 18:44 /media/sda9/other/___non-metal_-_non-ambient___/Vangelis -> /media/sda6/home/alex/Desktop/Vangelis/

    I think it’s pretty straightforward. What I did was to create two files called symlinks_storage.txt and symlinks_sda9.txt which contain the output of that `find’ command. That `find’ finds all symbolic links in those directories and does a `ls -l’ on them. The next thing I did was to grep through the text files to search for any occurrence of “/home/alex” which is my home partition and which should NOT get scanned.

    The symlink you see there, in sda9, pointing to /media/sda6/home/alex/Desktop/Vangelis, is an old and broken symbolic link. There is no /media/sda6 location on my system.

    So then, why does mt-daapd scan in “/home/alex” after all?

    #18561

    EVILRipper
    Participant

    I’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..

    #18562

    Anonymous

    @evilripper wrote:

    I’m not sure. Maybe firefly has strange behavior when scanning a broken symlink. Maybe you missed a symlink because you grepped on /home/alex? (unless that was just for the single relevant result). I just haven’t seen anybody saying firefly scans in the wrong dir..

    Well, I grep-ed on “/home/alex” because I noticed a disturbing number of lines in the log file relating to scans in /home/alex.

    Other than that, I couldn’t have missed a symlink to anything containing “/home/alex” using grep and yes that broken symlink was indeed the only thing (it pointed to a long-discarded partition which had a then-home-directory mounted).

    For the sake of it, I removed the broken symlink. Now there’s no such thing.

    So it shouldn’t scan /home/alex.

    Well… did the same thing: emptied log, deleted database, started mt-daapd.

    It still scans /home/alex. :-/

    #18564

    fizze
    Participant

    Care to post your mt-daapd.conf?
    What version of firefly do you use?

    #18565

    Anonymous

    @fizze wrote:

    Care to post your mt-daapd.conf?

    Sure.

    [email protected]:/home/alex# grep -v ^# /etc/mt-daapd.conf | grep -v ^$
    [general]
    web_root = /usr/share/mt-daapd/admin-root
    port = 3689
    admin_pw = mt-daapd
    db_type = sqlite3
    db_parms = /usr/var/cache/mt-daapd
    mp3_dir = /media/storage,/media/sda9
    servername = Firefly %v on %h
    runas = mt-daapd
    extensions = .mp3,.m4a,.m4p,.ogg,.flac,.mpc
    scan_type = 2
    logfile = /var/log/mt-daapd.log
    debuglevel = 9
    [plugins]
    plugin_dir = /usr/lib/mt-daapd/plugins
    plugins = rsp.so,ssc-ffmpeg.so
    [scanning]
    process_playlists = 1
    process_itunes = 1
    process_m3u = 1

    @fizze wrote:

    What version of firefly do you use?

    Firefly Media Server: Version svn-1696

    #18563

    fizze
    Participant

    Ok, that looks reasonable.
    Care to post your mount status? 😉
    What fs types do you use?

    I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
    Could it be there are hardlinks that cause firefly to trip?

    #18566

    Anonymous

    @fizze wrote:

    Ok, that looks reasonable.
    Care to post your mount status? 😉
    What fs types do you use?

    Sure… I have two SATA HDDs. The Linux partitions are ext3.

    mount output:

    /dev/sda6 on / type ext3 (rw,relatime,errors=remount-ro)
    proc on /proc type proc (rw,noexec,nosuid,nodev)
    /sys on /sys type sysfs (rw,noexec,nosuid,nodev)
    varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
    varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
    udev on /dev type tmpfs (rw,mode=0755)
    devshm on /dev/shm type tmpfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    lrm on /lib/modules/2.6.24-24-generic/volatile type tmpfs (rw)
    /dev/sda7 on /home type ext3 (rw,relatime)
    /dev/sda8 on /media/sda8 type ext3 (rw,relatime)
    /dev/sda9 on /media/sda9 type ext3 (rw,relatime)
    /dev/sdb2 on /media/storage type ext3 (rw,relatime)

    /dev/sdb1 on /media/winnew type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
    /dev/sda1 on /media/winold type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
    securityfs on /sys/kernel/security type securityfs (rw)
    rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    nfsd on /proc/fs/nfsd type nfsd (rw)

    The relevant parts (sda9, storage and home) are written in bold.

    @fizze wrote:

    I’ve never seen anything like this, and I’ve used firefly over multiple disks also.
    Could it be there are hardlinks that cause firefly to trip?

    I am not entirely sure I understand what you mean by hardlinks. Are you referring to the names of the devices – or to the devices themselves?

    As for any symlinks pointing to the home partition, there’s none. (I even removed the only broken symbolic link pointing there.)

    Next, I followed your suggestion. Maybe there is something wrong with mt-daapd and mp3_dir referring to locations on two different hard drivers.

    So I replaced

    mp3_dir = /media/storage,/media/sda9

    with

    mp3_dir = /media/storage

    Re-ran the test… it all went fine. However, when I added the second location (/media/sda9), it began scanning /home/alex (and also crashed, same as before).

    Then I started it only with

    mp3_dir = /media/sda9

    Same erratic behavior (scanning in home, crash).

    The interesting bit is that, as you can see, both /media/sda9/ and /home are parts of the same physical hard drive.

    It shouldn’t make any difference, but just to make sure:


    fdisk -l

    Disk /dev/sda: 400.0 GB, 400088457216 bytes
    255 heads, 63 sectors/track, 48641 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x8f8004b0

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 7833 62918541 7 HPFS/NTFS
    /dev/sda2 7834 48641 327790260 5 Extended
    /dev/sda5 7834 8029 1574338+ 82 Linux swap / Solaris
    /dev/sda6 8030 10640 20972826 83 Linux
    /dev/sda7 10641 13251 20972826 83 Linux
    /dev/sda8 13252 15862 20972826 83 Linux
    /dev/sda9 15863 48641 263297286 83 Linux

    Disk /dev/sdb: 500.1 GB, 500107862016 bytes
    255 heads, 63 sectors/track, 60801 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x000c54f2

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 15680 125949568+ 7 HPFS/NTFS
    /dev/sdb2 15681 60801 362434432+ 83 Linux

    I also have enough free space on all devices:

     df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda6 20G 3.4G 16G 18% /
    varrun 1014M 300K 1014M 1% /var/run
    varlock 1014M 0 1014M 0% /var/lock
    udev 1014M 72K 1014M 1% /dev
    devshm 1014M 0 1014M 0% /dev/shm
    lrm 1014M 40M 975M 4% /lib/modules/2.6.24-24-generic/volatile
    /dev/sda7 20G 13G 6.3G 67% /home
    /dev/sda8 20G 19G 312M 99% /media/sda8
    /dev/sda9 248G 218G 18G 93% /media/sda9
    /dev/sdb2 341G 281G 43G 87% /media/storage
    /dev/sdb1 121G 46G 75G 39% /media/winnew
    /dev/sda1 61G 15G 46G 25% /media/winold

    There are just 56 symbolic links on the whole sda9 partition… most are broken, pointing to where-some-mount-points-used-to-be. If you think it’s relevant, I’ll post that. Although… there’s nothing even mentioning “home” there. :-/

    Well, at least we’re narrowing the field. There’s obviously something wrong in some cases when the media directories are on different physical hard drives.

    #18567

    fizze
    Participant

    Ok, that all looks good.
    Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.
    Maybe broken hardlinks can point across partitions? hmmm. Thats a long shot, anyway, and it seems unlikely.

    But clearly something on sda9 seems to throw firefly off.

    You did erase your songs.db at a certain point, right?
    Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?

    Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.

    #18568

    Anonymous

    @fizze wrote:

    Ok, that all looks good.
    Hardlinks are created by using ln without -s. You can hardlink to files on the same partition iirc.

    Oh, I see, thanks for clarifying this.

    @fizze wrote:

    You did erase your songs.db at a certain point, right?
    Maybe there are zombie’d entries left in there that cause firefly to try to scan those files?

    Erase songs.db (or songs3.db) , set the mp3_dir to sda9 only and then start up firefly.

    Yes, I was erasing both songs3.db and songs3(something)journal each time I was starting the test again.

    However, I decided to delete all the broken symbolic links… after this, it worked.

    @fizze wrote:

    But clearly something on sda9 seems to throw firefly off.

    You got that right. Here’s what happened: I took a better look at all the symbolic links in /media/sda9 and found a really broken one (I don’t know how it actually got there in the first place)…

    lrwxrwxrwx 1 root root 1 2007-11-11 11:45 /media/sda9/backup/b4gg/root/.wine/dosdevices/z: -> /

    Some weird backup of the entire /root home directory before some old, forgotten install.

    So I uhm am sorry for all the trouble, it works alright… Thanks for the feedback and for taking the time to debug me.

    This is embarrassing. 😛

Viewing 15 posts - 1 through 15 (of 19 total)
  • You must be logged in to reply to this topic.
Register here