Posted by Phoenix
on 4/24/2008, 6:08 am
Once again I must praise you on Medusa. When I found a site filled with countless wallpapers, I decided it was time to pull out my old friend and let her do the work.
In any case, I was just running v1.1. Haven't had the money to get around to buying 1.2 yet. Only recently started really working. Anyways, I noticed a curious item...The website I was running on, getawallpaper.com, has a heck of a lot of pages. In excess of 150,000 pages. This is primarly due to the various in site categories and such. I'm running 5 Search, 2 Thumb, and 10 Download Threads. Earlier I was only running 2 Search's for bandwidth concerns. Anyways, I've had to restart this search about 4 times now, each one running about 4 hours min each. I noticed that after running for a few hours, one of the threads seemingly froze up. It ends up staying on 'Downloading Page ???' and never frees back up. In addition, prior to this, if I paused the search, my proccessor usage would go back down to the 10% it averages at the rest of the time. After this thread froze, it would not stop down. As if the thread was caught in an infinate loop of some sort. Since I have no way to cancel a thread's execution, I have no way to free the proccess up. I don't recall if this is fixed in 1.2, but I figured I'd bring it up.
And I can't leave without any ideas:
1. A 'Settings Pane' In the Threads Sub Window. Basically, this pane lists all the settings for the current search. Primarly for if we want to confirm we did the settings right before it gets to far gone. If done right, this could also be an area to modify certain sections of the search. IE: Auto-downloads - Spell a word wrong without realize, check it here and rename it.
1a. Put a lock on any mods here. IE: All threads must be paused to succesfully modify the search.
2. In the task manager, there is an ability to right click on a proccess in the proccesses list and change it's priorty (Don't make the mistake of putting 1.1 on realtime x_X. Froze my entire comp up) in realition to the other programs. You can make it: Real Time, Highest, Above Normal, Normal, Below Normal, And Low. I found it useful in putting Medusa at Below Norm or Low to be able to use my comp easier while medusa is running. I have a feeling that basic API's are used to access this and thus suggest the ability to change these in Medusa directly (As a part of ram saving controls). For example, when we start a search, choose it to be placed at Low Priorty, or add another button next to the pause button to change the priority...Just give a warning when going to Real Time.
2a. I think the primary reason my computer froze when going to real time was one of my search threads had already hit it's loop that I was discussing above.
Beyond that, I guess it's the standard stuff. Ram Saving Mode, And Search Saving Capabilities, again best done as a DB I believe. That or the Plugin Option. IE:
Name: TFMDPlugin.dll
Class Name: TFMDPlugin.Medusa (Medusa Checks for this class when the plugin is loaded. If an error occurs, the plugin isn't loaded).
Functions:
ProccessURL(strURL) - Returns the orginal URL sent, and URL, or nothing.
Properties:
Queue_Length - Total Items In Queue To Search (This can be used by ProccessURL so if a person wants Medusa to have free reign up to 100,000 Queued, then it can save urls sent through and send back nothing till the count is 50,000).
The Script Control Idea would be the same thing. A script is registered as VBS or JS, all URLs are sent to the function 'ProccessURL(strURL, dblQueue) or similar.
3. Thread Cancel/Refresh Abilities. Basically, this would help if the inifinate loop thing kept occuring. If our entire search is paused, we can choose to 'Refresh Threads' or right clicking on a thread to cancel it. (I know there's the Cancel download and all, but this for all threads). Basically, if we do this, the system polls all the search que (Which you may already have it doing...I'm unsure on this point), cancels/shuts down all the threads (or the selected one) and restarts anew. This would allow us to do a sort of ultimate pause during the day. Not to mention help prevent thread lock down the other guy mentioned a while back.
Anyways, don't watch to much TV or you'll fry your brains as bad as we already have, lol.
Peace.
Message Thread:
« Back to thread