GIANT Library Loading and Browsing Problems

Viewing 15 posts - 1 through 15 (of 17 total)
  • Author
    Posts
  • #1740
    ratbag21
    Participant

    I have searched the forums high and low, but have not found a clear answer or example for my issues.

    I have an NSLU2 unslung using the 6.8 firmware to disk2 [2GB flash drive], with a Western Dig My Book 500GB connected to disk1. I am using the 0.2.4 mt-daapd as my music server. The NSLU2 boots great, Firefly starts great, and works great I might add!!!! [big thanks to all involved in the project].

    My issue is that I have a 45,000+ library and cannot browse [even by artist – 2600+ different artists] at all. Small playlists [.m3u] will load in an instant, large playlists [8,000 – 12,000 entries] will load SOMETIMES.

    My question: is there a solution to possibly correct this behavior, or am I looking for something that does not exist? I would like to be able to browse songs etc, and load large playlists.

    thanks,
    Rat

    #12615
    rpedde
    Participant

    @ratbag21 wrote:

    I have searched the forums high and low, but have not found a clear answer or example for my issues.

    I have an NSLU2 unslung using the 6.8 firmware to disk2 [2GB flash drive], with a Western Dig My Book 500GB connected to disk1. I am using the 0.2.4 mt-daapd as my music server. The NSLU2 boots great, Firefly starts great, and works great I might add!!!! [big thanks to all involved in the project].

    My issue is that I have a 45,000+ library and cannot browse [even by artist – 2600+ different artists] at all. Small playlists [.m3u] will load in an instant, large playlists [8,000 – 12,000 entries] will load SOMETIMES.

    My question: is there a solution to possibly correct this behavior, or am I looking for something that does not exist? I would like to be able to browse songs etc, and load large playlists.

    thanks,
    Rat

    Is this against a soundbridge, I assume?

    If so, then nightlies will probably work better. At least faster, and people have reported with with sqlite3 as a backend, it browses better, too.

    Right now, the slug nightlies don’t support sqlite3, but I’ll have that in nightlies soon. Possibly as soon as this weekend.

    — Ron

    #12616
    ratbag21
    Participant

    Yes this is with a soundbridge M1001.

    So, this is related to db access, speed of slug vs. time-out on the soundbridge?

    so, you recommend that I wait to load nightlies until after the sqlite3 is enabled? …or will nightlies without sqlite3 also possibly/probably give a performance improvement?

    thanks for the response,
    Rat

    #12617
    mas
    Participant

    I think there are 2 problems:

    1) Timout. After about 60 secs or so it wont go on. With a slug its not easy to deliver the entries fast enough. Latest nightly plus sqlite3 selfcompiled is likely best. 4.3k work in about 15-20 sec so at around 12-15k it should time out.

    2) Memory in the Soundbridge. They never stated a maximum precisely but the thing has only 32MB memory of which only so many is free. I think over 10k you get a problem with that anyway.

    So basically, much more than 10k songs in one select wont work. You need to either split databases or only use playlists/selections of <10k.

    #12618
    rpedde
    Participant

    @mas wrote:

    Latest nightly plus sqlite3 selfcompiled is likely best. 4.3k work in about 15-20 sec so at around 12-15k it should time out.

    I’ve double-checked my build chain for unslung, and it’s configured for both sqlite and sqlite3.

    You should be able to just change the dbtype from “sqlite” to “sqlite3” in your config file and see what that does.

    — Ron

    #12619
    ratbag21
    Participant

    That change in DBtype could only be done via a “nightly” build, correct? The existing stable-build does not support this option, correct?

    One additional question and I am off to make some changes…

    That config change causes the App to create a DB file in a SQLite3 format and then reads that file directly? …Or is there a SQLite App that needs to be running as well on the slug?

    thanks again!!!

    BTW, the whole Firefly setup is by compare the best option out there for having music on demand via soundbridge.

    #12620
    rpedde
    Participant

    @ratbag21 wrote:

    That change in DBtype could only be done via a “nightly” build, correct? The existing stable-build does not support this option, correct?

    Correct.

    One additional question and I am off to make some changes…

    That config change causes the App to create a DB file in a SQLite3 format and then reads that file directly? …Or is there a SQLite App that needs to be running as well on the slug?

    Nope, sqlite is a “serverless” sql engine. Everything it needs is built into the daap server.

    BTW, the whole Firefly setup is by compare the best option out there for having music on demand via soundbridge.

    Well, once it gets stable again. 🙂

    — Ron

    #12621
    ratbag21
    Participant

    Ron,

    Thanks again for your response and for the maintenance of the Firefly.

    Would you recommend the 1668 build or the 1586 for the SQLite and stability?

    thanks,
    Rat

    #12622
    rpedde
    Participant

    @ratbag21 wrote:

    Ron,

    Thanks again for your response and for the maintenance of the Firefly.

    Would you recommend the 1668 build or the 1586 for the SQLite and stability?

    thanks,
    Rat

    1668 seems stableish, outside of some corner cases. 1671 is cooking now. There is really only one issue still outstanding, and that’s segfaults when connecting via rsp on mipsel boxes. Afaik.

    But were I you, and just wanted to test it, it makes sense to test it on a stable apples-to-apples basis, so I’d probably go 1586.

    — Ron

    #12623
    ratbag21
    Participant

    Ron,

    Ok I did the install on 1586. It seems to have gone smoothly, however the I get an “error on reading the config file”. I did a full edit on the new config file so all the location etc. parameters matched my previous config file.

    Do I need to install any other pkgs/files beyond “mt-daapd_svn-1586-1_armeb.ipk”?

    Do I need to do a cold/clean reboot of the slug/server after install?

    Can 1586 do sqlite3 ok [vs. other options] or do recommend another choice?

    Do I need to remove the old version/type db file?

    Can the sqlite3 db [or any db selected] location be the same as the db parameter location?

    Do the options [settings] in the config file need to have “=” between the option name and the setting, or can there just be spaces as in previous config format?

    I did an update of my “/etc/ipkg/firefly.conf” file [confirmed that the file contents read as directed] , however when I run the ipkg update command, the system says it is ignoring an unknown parameter, and then the package update comes from the “nslu2-linux.org”. The update to the file was to “src firefly http://ipkg.fireflymediaserver.org/armeb&#8221;, is this the correct syntax in the firefly.conf file for that location?

    sorry about all the questions and I appreciate the responses…I think that covers all the odds and ends…

    rat

    #12624
    rpedde
    Participant

    @ratbag21 wrote:

    Ok I did the install on 1586. It seems to have gone smoothly, however the I get an “error on reading the config file”. I did a full edit on the new config file so all the location etc. parameters matched my previous config file.

    Yeah, should have said it makes sense to let it overwrite your existing config file. You can let it do that by reinstalling with –reinstall and answering to let it overwrite the config.

    Do I need to install any other pkgs/files beyond “mt-daapd_svn-1586-1_armeb.ipk”?

    Yeah, a bunch. But it should drag in the dependancies when it installs automatically.

    Do I need to do a cold/clean reboot of the slug/server after install?

    Not strictly speaking, no. Just make sure that the mt-daapd processes are all gone before starting the new one.

    Can 1586 do sqlite3 ok [vs. other options] or do recommend another choice?

    Yes, it’s compiled in the nightlies.

    Do I need to remove the old version/type db file?

    Not unless you are particularly fastidious, no. Doesn’t hurt anything, just unnecessary.

    Can the sqlite3 db [or any db selected] location be the same as the db parameter location?

    That’s where it goes… in the directory used for db_parms. At some point, I expect to have other database backends that will require usernames and passwords and whatnot, but the sqlite/sqlite3 database only requires a path, so that’s all that’s in the db_parms — the path to put the database in.

    Do the options [settings] in the config file need to have “=” between the option name and the setting, or can there just be spaces as in previous config format?

    Has to be an =

    Well, not strictly speaking. But if it doesn’t find an “=”, then it thinks it’s an old-style config, and you’ll run into problems.

    I did an update of my “/etc/ipkg/firefly.conf” file [confirmed that the file contents read as directed] , however when I run the ipkg update command, the system says it is ignoring an unknown parameter, and then the package update comes from the “nslu2-linux.org”. The update to the file was to “src firefly http://ipkg.fireflymediaserver.org/armeb&#8221;, is this the correct syntax in the firefly.conf file for that location?

    Yeah, that looks right. Maybe double-check for typos? If not, you can just browse to that location and download the file and install it that way.

    sorry about all the questions and I appreciate the responses…I think that covers all the odds and ends…

    🙂

    np.

    — Ron

    #12625
    ratbag21
    Participant

    Ron,

    thanks again for all your help. I corrected the update location in the firefly.conf file. I had to remove the “quotes” from the single line in the file. I ran the update, worked now. Then did a -force-reinstall, edited the new config file to match all the previous music directory and db locations, I am now on 1671. One mistake I made the first time was that I set the db_dir AND the db_parm. I guess I figured out that the db_dir setting is not used in the new config.

    I will let you know how the browsing has improved. It might be awhile as the slug is probably going to take awhile to scan all the files and add to the new db.

    rat

    #12626
    mas
    Participant

    Are you having problems with sqlite3 and svn-1671? I ahd some trouble there and use 1586 with sqlite3. Seems to be a memory related problem with the new versions and sqlite3 only though.

    #12627
    ratbag21
    Participant

    No problem with 1671 and sqlite3. browsing artists, albums, and songs works great. i would say 1671 is a great upgrade from 0.2.4.

    one small item that i have not figured yet is reading static playlists. previously i was using the 0.2.4 version of FF. playlists worked fine, but i could not browse songs because of the large numbers. now i can’t seem to get the playlists to “read” but the songs load to browse great. FF sees them but does not read the contents of the files.

    [Ron, any clues that you could give on this playlist question??? – i have already been through the removal of spaces and changing the / to and that worked with the 0.2.4 FF version?]

    BTW, i have 45,000 mp3’s, M1001, and a slug on 1671 w/sqlite3 db. when i go to browse songs they ALL load to the M1001. i am aware that others have said there is a ~12,000 record limit on the M1001, but previously i had twonky serving and i was able to load the full 45,000 to the queue.

    my large playlists are between 11,000 and 13,000, small ones ~ 100-200.

    thanks,
    Rat

    #12628
    rpedde
    Participant

    @ratbag21 wrote:

    [Ron, any clues that you could give on this playlist question??? – i have already been through the removal of spaces and changing the / to and that worked with the 0.2.4 FF version?]

    Can you show one?

Viewing 15 posts - 1 through 15 (of 17 total)
  • The forum ‘General Discussion’ is closed to new topics and replies.