Crawl Issues _ HELP!

[scottsaxton]

What is a resonable size to increase the memory?

My web site is not huge.  The sitemap that makes it to link dept 7 is only 2.73MB.

I increased the memory to 64MB and after several re-starting of the crawl from where it left off, It made it to link depth 22....and said complete, but it never really completed becuase teh change log was not updated and the date stamp did not change either.

I increaded the memory to 128MB and started a crawl and it is sitting at the crawling screen at link depth 7 and has not updated in over 15 minutes.

My other setting in the php.ini include:
max_execution_time = 1800
max_input_tion = 60

Do these seem reasonable?

[XML-Sitemaps Support]

Hello,

please try to execute sitemap generator in command line to see it's output.

[scottsaxton]

This might be a stupid question, but how do i do that?

SSH into my webserver?

[scottsaxton]

Okay another update?

I have not found a way to get access to a command line yet on my web server.

My php.ini config is the following:
;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

max_execution_time = 1800     ;Maximum execution time of each script, in seconds
max_input_time = 60     ; Maximum amount of time each script may spend parsing request data
memory_limit = 128M     ; Maximum amount of memory a script may consume (8MB)

When I sent the max depth level = 6, it runs fine and once crawl is complete goes to the View Sitemap Page.  Here are the results:

Request date:
19 March 2008, 15:13
Processing time:
23.27s
Pages indexed:
1706
Sitemap files:
1
Pages size:
2.78Mb

When I set the max depth level =7, it starts....then I see:

Links depth: 8
Current page: images/photo_gallery/Northeastern-PA-Flights-2004/1/lg/DSCF0059.htm
Pages added to sitemap: 3418
Pages scanned: 3418 (8,462.7 Kb)
Pages left: 0 (+ 0 queued for the next depth level)
Time passed: 1:17
Time left: 0:00
Memory usage: 3,552.5 Kb
Completed
Total pages indexed: 3418

Then nothing else happens.  When I click on the View Sitemap tab, I see this:

Warning: file(/hermes/bosweb/web145/b1450/ipw.weflyhot/public_html/generator/data/sitemap.xml) [function.file]: failed to open stream: No such file or directory in /hermes/bosweb/web145/b1450/ipw.weflyhot/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 40

Warning: implode() [function.implode]: Bad arguments. in /hermes/bosweb/web145/b1450/ipw.weflyhot/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 40

[XML-Sitemaps Support]

Hello,

yes - you should open an SSH session to your host to access command line.

Looks like sitemap crawling has been completed, but there is an error when creating sitemaps, you should see that error message in command line.

[scottsaxton]

I spoke with my web host today.  They told me that there isn't a way for me to use a command line through ssh or telnet.  Do you think that this is accurate?

So if that is the case, I cannot do a command line.  I did however run the command from the php section of the website control panel.

Using the command: [ External links are visible to forum administrators only ].  But I am not sure if this helps anyone.....huh it ran at 7 this time?  see below:

With the max_depth_level = 6 I get the following in command output window:
Resuming the last session (last updated: 1969-12-31 19:00:00)1 | 20 | 8.5 | 0:01 | 0:36 | 1 | 454.1 Kb | 1 | 0 | 454 20 | 1 | 174.9 | 0:02 | 0:00 | 1 | 556.4 Kb | 20 | 25 | 102 40 | 33 | 247.3 | 0:03 | 0:03 | 4 | 537.5 Kb | 40 | 40 | -19 60 | 13 | 372.3 | 0:04 | 0:01 | 4 | 659.5 Kb | 60 | 411 | 122 80 | 497 | 483.8 | 0:05 | 0:36 | 5 | 754.7 Kb | 80 | 26 | 95 100 | 477 | 588.7 | 0:06 | 0:30 | 5 | 807.5 Kb | 100 | 76 | 53 120 | 457 | 677.1 | 0:06 | 0:26 | 5 | 812.5 Kb | 120 | 110 | 5 140 | 437 | 776.4 | 0:07 | 0:23 | 5 | 866.6 Kb | 140 | 220 | 54 160 | 417 | 888.6 | 0:08 | 0:20 | 5 | 933.8 Kb | 160 | 255 | 67 180 | 397 | 983.8 | 0:08 | 0:18 | 5 | 938.7 Kb | 180 | 293 | 5 200 | 377 | 1,076.6 | 0:09 | 0:17 | 5 | 976.5 Kb | 200 | 404 | 38 220 | 357 | 1,181.5 | 0:09 | 0:15 | 5 | 1,030.3 Kb | 220 | 456 | 54 240 | 337 | 1,299.3 | 0:10 | 0:14 | 5 | 1,077.1 Kb | 240 | 512 | 47 260 | 317 | 1,417.0 | 0:10 | 0:13 | 5 | 1,109.9 Kb | 260 | 568 | 32 280 | 297 | 1,528.4 | 0:11 | 0:12 | 5 | 1,134.8 Kb | 280 | 616 | 25 300 | 277 | 1,646.2 | 0:12 | 0:11 | 5 | 1,168.6 Kb | 300 | 672 | 34 320 | 257 | 1,736.6 | 0:12 | 0:10 | 5 | 1,159.6 Kb | 320 | 828 | -9 340 | 237 | 1,801.9 | 0:13 | 0:09 | 5 | 1,181.3 Kb | 340 | 868 | 22 360 | 217 | 1,867.2 | 0:13 | 0:08 | 5 | 1,206.2 Kb | 360 | 908 | 25 380 | 197 | 1,932.6 | 0:14 | 0:07 | 5 | 1,231.5 Kb | 380 | 948 | 25 400 | 177 | 1,997.9 | 0:14 | 0:06 | 5 | 1,259.6 Kb | 400 | 987 | 28 420 | 157 | 2,063.1 | 0:15 | 0:05 | 5 | 1,289.1 Kb | 420 | 1028 | 30 440 | 137 | 2,130.6 | 0:16 | 0:05 | 5 | 1,315.0 Kb | 440 | 1063 | 26 460 | 117 | 2,213.3 | 0:16 | 0:04 | 5 | 1,370.0 Kb | 460 | 1083 | 55 480 | 97 | 2,331.8 | 0:17 | 0:03 | 5 | 1,430.9 Kb | 480 | 1083 | 60 500 | 77 | 2,450.0 | 0:18 | 0:02 | 5 | 1,445.6 Kb | 500 | 1083 | 15 520 | 57 | 2,558.1 | 0:18 | 0:02 | 5 | 1,445.7 Kb | 520 | 1083 | 0 540 | 37 | 2,660.1 | 0:19 | 0:01 | 5 | 1,451.2 Kb | 540 | 1177 | 6 560 | 17 | 2,768.5 | 0:21 | 0:00 | 5 | 1,538.7 Kb | 560 | 1458 | 87 580 | 1126 | 2,851.4 | 0:22 | 0:43 | 6 | 1,477.6 Kb | 580 | 0 | -61 600 | 1106 | 2,851.4 | 0:22 | 0:41 | 6 | 1,490.5 Kb | 600 | 0 | 13 620 | 1086 | 2,851.4 | 0:22 | 0:39 | 6 | 1,503.1 Kb | 620 | 0 | 13 640 | 1066 | 2,851.4 | 0:22 | 0:37 | 6 | 1,515.5 Kb | 640 | 0 | 12 660 | 1046 | 2,851.4 | 0:22 | 0:35 | 6 | 1,528.1 Kb | 660 | 0 | 13 680 | 1026 | 2,851.4 | 0:22 | 0:34 | 6 | 1,541.4 Kb | 680 | 0 | 13 700 | 1006 | 2,851.4 | 0:22 | 0:32 | 6 | 1,554.7 Kb | 700 | 0 | 13 720 | 986 | 2,851.4 | 0:22 | 0:31 | 6 | 1,567.8 Kb | 720 | 0 | 13 740 | 966 | 2,851.4 | 0:22 | 0:29 | 6 | 1,580.5 Kb | 740 | 0 | 13 760 | 946 | 2,851.4 | 0:22 | 0:28 | 6 | 1,593.1 Kb | 760 | 0 | 13 780 | 926 | 2,851.4 | 0:22 | 0:27 | 6 | 1,605.6 Kb | 780 | 0 | 12 800 | 906 | 2,851.4 | 0:22 | 0:25 | 6 | 1,618.4 Kb | 800 | 0 | 13 820 | 886 | 2,851.4 | 0:22 | 0:24 | 6 | 1,631.0 Kb | 820 | 0 | 13 840 | 866 | 2,851.4 | 0:22 | 0:23 | 6 | 1,643.8 Kb | 840 | 0 | 12 860 | 846 | 2,851.4 | 0:22 | 0:22 | 6 | 1,656.5 Kb | 860 | 0 | 13 880 | 826 | 2,851.4 | 0:22 | 0:21 | 6 | 1,668.4 Kb | 880 | 0 | 12 900 | 806 | 2,851.4 | 0:22 | 0:20 | 6 | 1,680.3 Kb | 900 | 0 | 12 920 | 786 | 2,851.4 | 0:22 | 0:19 | 6 | 1,692.2 Kb | 920 | 0 | 12 940 | 766 | 2,851.4 | 0:22 | 0:18 | 6 | 1,704.1 Kb | 940 | 0 | 12 960 | 746 | 2,851.4 | 0:22 | 0:17 | 6 | 1,716.0 Kb | 960 | 0 | 12 980 | 726 | 2,851.4 | 0:23 | 0:17 | 6 | 1,727.9 Kb | 980 | 0 | 11 1000 | 706 | 2,851.4 | 0:23 | 0:16 | 6 | 1,739.8 Kb | 1000 | 0 | 12 1020 | 686 | 2,851.4 | 0:23 | 0:15 | 6 | 1,751.7 Kb | 1020 | 0 | 12 1040 | 666 | 2,851.4 | 0:23 | 0:14 | 6 | 1,767.6 Kb | 1040 | 0 | 16 1060 | 646 | 2,851.4 | 0:23 | 0:14 | 6 | 1,779.5 Kb | 1060 | 0 | 12 1080 | 626 | 2,851.4 | 0:23 | 0:13 | 6 | 1,791.4 Kb | 1080 | 0 | 12 1100 | 606 | 2,851.4 | 0:23 | 0:12 | 6 | 1,803.1 Kb | 1100 | 0 | 12 1120 | 586 | 2,851.4 | 0:23 | 0:12 | 6 | 1,815.0 Kb | 1120 | 0 | 12 1140 | 566 | 2,851.4 | 0:23 | 0:11 | 6 | 1,826.9 Kb | 1140 | 0 | 11 1160 | 546 | 2,851.4 | 0:23 | 0:10 | 6 | 1,838.8 Kb | 1160 | 0 | 12 1180 | 526 | 2,851.4 | 0:23 | 0:10 | 6 | 1,850.6 Kb | 1180 | 0 | 12 1200 | 506 | 2,851.4 | 0:23 | 0:09 | 6 | 1,862.7 Kb | 1200 | 0 | 12 1220 | 486 | 2,851.4 | 0:23 | 0:09 | 6 | 1,875.4 Kb | 1220 | 0 | 13 1240 | 466 | 2,851.4 | 0:23 | 0:08 | 6 | 1,888.1 Kb | 1240 | 0 | 13 1260 | 446 | 2,851.4 | 0:23 | 0:08 | 6 | 1,900.8 Kb | 1260 | 0 | 12 1280 | 426 | 2,851.4 | 0:23 | 0:07 | 6 | 1,913.5 Kb | 1280 | 0 | 13 1300 | 406 | 2,851.4 | 0:23 | 0:07 | 6 | 1,926.1 Kb | 1300 | 0 | 13 1320 | 386 | 2,851.4 | 0:23 | 0:06 | 6 | 1,938.8 Kb | 1320 | 0 | 12 1340 | 366 | 2,851.4 | 0:23 | 0:06 | 6 | 1,951.5 Kb | 1340 | 0 | 13 1360 | 346 | 2,851.4 | 0:23 | 0:05 | 6 | 1,964.1 Kb | 1360 | 0 | 13 1380 | 326 | 2,851.4 | 0:23 | 0:05 | 6 | 1,976.7 Kb | 1380 | 0 | 12 1400 | 306 | 2,851.4 | 0:23 | 0:05 | 6 | 1,989.3 Kb | 1400 | 0 | 13 1420 | 286 | 2,851.4 | 0:23 | 0:04 | 6 | 2,002.0 Kb | 1420 | 0 | 13 1440 | 266 | 2,851.4 | 0:23 | 0:04 | 6 | 2,014.6 Kb | 1440 | 0 | 12 1460 | 246 | 2,851.4 | 0:23 | 0:03 | 6 | 2,027.2 Kb | 1460 | 0 | 13 1480 | 226 | 2,851.4 | 0:23 | 0:03 | 6 | 2,039.8 Kb | 1480 | 0 | 12 1500 | 206 | 2,851.4 | 0:23 | 0:03 | 6 | 2,052.9 Kb | 1500 | 0 | 13 1520 | 186 | 2,851.4 | 0:23 | 0:02 | 6 | 2,065.6 Kb | 1520 | 0 | 13 1540 | 166 | 2,851.4 | 0:23 | 0:02 | 6 | 2,078.2 Kb | 1540 | 0 | 13 1560 | 146 | 2,851.4 | 0:23 | 0:02 | 6 | 2,090.9 Kb | 1560 | 0 | 12 1580 | 126 | 2,851.4 | 0:23 | 0:01 | 6 | 2,103.6 Kb | 1580 | 0 | 13 1600 | 106 | 2,851.4 | 0:23 | 0:01 | 6 | 2,116.1 Kb | 1600 | 0 | 13 1620 | 86 | 2,851.4 | 0:23 | 0:01 | 6 | 2,128.7 Kb | 1620 | 0 | 12 1640 | 66 | 2,851.4 | 0:23 | 0:00 | 6 | 2,141.3 Kb | 1640 | 0 | 13 1660 | 46 | 2,851.4 | 0:23 | 0:00 | 6 | 2,153.9 Kb | 1660 | 0 | 12 1680 | 26 | 2,851.4 | 0:23 | 0:00 | 6 | 2,166.5 Kb | 1680 | 0 | 13 1700 | 6 | 2,851.4 | 0:23 | 0:00 | 6 | 2,179.1 Kb | 1700 | 0 | 13 1706 | 0 | 2,851.4 | 0:23 | 0:00 | 7 | 1,949.9 Kb | 1706 | 0 | -230 top.location = 'index.php?op=view'

With the max_depth_level=7 I get:
Resuming the last session (last updated: 1969-12-31 19:00:00)1 | 20 | 8.5 | 0:01 | 0:39 | 1 | 454.2 Kb | 1 | 0 | 454 20 | 1 | 174.9 | 0:02 | 0:00 | 1 | 556.5 Kb | 20 | 25 | 102 40 | 33 | 247.3 | 0:03 | 0:02 | 4 | 534.2 Kb | 40 | 40 | -22 60 | 13 | 372.3 | 0:03 | 0:00 | 4 | 659.7 Kb | 60 | 411 | 125 80 | 497 | 483.8 | 0:03 | 0:23 | 5 | 753.8 Kb | 80 | 26 | 94 100 | 477 | 588.7 | 0:04 | 0:19 | 5 | 806.2 Kb | 100 | 76 | 53 120 | 457 | 677.1 | 0:04 | 0:16 | 5 | 809.5 Kb | 120 | 110 | 3 140 | 437 | 776.4 | 0:04 | 0:14 | 5 | 864.3 Kb | 140 | 220 | 55 160 | 417 | 888.6 | 0:05 | 0:14 | 5 | 931.7 Kb | 160 | 255 | 67 180 | 397 | 983.8 | 0:06 | 0:13 | 5 | 935.5 Kb | 180 | 293 | 4 200 | 377 | 1,076.6 | 0:06 | 0:11 | 5 | 975.7 Kb | 200 | 404 | 40 220 | 357 | 1,181.5 | 0:06 | 0:10 | 5 | 1,030.2 Kb | 220 | 456 | 55 240 | 337 | 1,299.3 | 0:06 | 0:09 | 5 | 1,076.9 Kb | 240 | 512 | 46 260 | 317 | 1,417.0 | 0:07 | 0:08 | 5 | 1,109.7 Kb | 260 | 568 | 33 280 | 297 | 1,528.4 | 0:07 | 0:08 | 5 | 1,134.1 Kb | 280 | 616 | 25 300 | 277 | 1,646.2 | 0:08 | 0:07 | 5 | 1,168.0 Kb | 300 | 672 | 34 320 | 257 | 1,736.6 | 0:09 | 0:07 | 5 | 1,159.1 Kb | 320 | 828 | -9 340 | 237 | 1,801.9 | 0:09 | 0:06 | 5 | 1,180.8 Kb | 340 | 868 | 21 360 | 217 | 1,867.2 | 0:10 | 0:06 | 5 | 1,205.8 Kb | 360 | 908 | 25 380 | 197 | 1,932.6 | 0:10 | 0:05 | 5 | 1,231.1 Kb | 380 | 948 | 26 400 | 177 | 1,997.9 | 0:10 | 0:04 | 5 | 1,259.1 Kb | 400 | 987 | 28 420 | 157 | 2,063.1 | 0:11 | 0:04 | 5 | 1,288.3 Kb | 420 | 1028 | 29 440 | 137 | 2,130.6 | 0:11 | 0:03 | 5 | 1,314.1 Kb | 440 | 1063 | 26 460 | 117 | 2,213.3 | 0:12 | 0:03 | 5 | 1,369.2 Kb | 460 | 1083 | 55 480 | 97 | 2,331.8 | 0:12 | 0:02 | 5 | 1,430.2 Kb | 480 | 1083 | 61 500 | 77 | 2,450.0 | 0:13 | 0:02 | 5 | 1,445.0 Kb | 500 | 1083 | 15 520 | 57 | 2,558.1 | 0:13 | 0:01 | 5 | 1,444.7 Kb | 520 | 1083 | -1 540 | 37 | 2,660.1 | 0:13 | 0:00 | 5 | 1,450.4 Kb | 540 | 1177 | 6 560 | 17 | 2,768.5 | 0:14 | 0:00 | 5 | 1,538.3 Kb | 560 | 1458 | 88 580 | 1126 | 2,861.2 | 0:15 | 0:30 | 6 | 1,478.0 Kb | 580 | 9 | -60 600 | 1106 | 2,938.7 | 0:16 | 0:30 | 6 | 1,545.0 Kb | 600 | 59 | 67 620 | 1086 | 3,028.3 | 0:17 | 0:30 | 6 | 1,576.0 Kb | 620 | 92 | 31 640 | 1066 | 3,093.2 | 0:18 | 0:31 | 6 | 1,570.4 Kb | 640 | 134 | -6 660 | 1046 | 3,201.4 | 0:19 | 0:31 | 6 | 1,672.3 Kb | 660 | 183 | 102 680 | 1026 | 3,307.4 | 0:20 | 0:30 | 6 | 1,691.8 Kb | 680 | 216 | 19 700 | 1006 | 3,398.9 | 0:20 | 0:29 | 6 | 1,700.7 Kb | 700 | 261 | 9 720 | 986 | 3,497.2 | 0:22 | 0:30 | 6 | 1,738.2 Kb | 720 | 382 | 38 740 | 966 | 3,562.9 | 0:23 | 0:30 | 6 | 1,743.3 Kb | 740 | 423 | 5 760 | 946 | 3,628.2 | 0:23 | 0:29 | 6 | 1,771.3 Kb | 760 | 465 | 28 780 | 926 | 3,712.9 | 0:24 | 0:29 | 6 | 1,844.9 Kb | 780 | 515 | 73 800 | 906 | 3,828.5 | 0:25 | 0:29 | 6 | 1,916.7 Kb | 800 | 571 | 72 820 | 886 | 3,924.1 | 0:26 | 0:28 | 6 | 1,898.8 Kb | 820 | 602 | -18 840 | 866 | 4,041.8 | 0:27 | 0:28 | 6 | 1,970.1 Kb | 840 | 652 | 72 860 | 846 | 4,144.3 | 0:28 | 0:28 | 6 | 1,965.3 Kb | 860 | 718 | -5 880 | 826 | 4,209.9 | 0:30 | 0:28 | 6 | 1,954.8 Kb | 880 | 760 | -11 900 | 806 | 4,322.9 | 0:30 | 0:27 | 6 | 2,058.1 Kb | 900 | 801 | 104 920 | 786 | 4,392.1 | 0:31 | 0:26 | 6 | 2,029.9 Kb | 920 | 863 | -29 940 | 766 | 4,503.9 | 0:32 | 0:26 | 6 | 2,096.8 Kb | 940 | 897 | 67 960 | 746 | 4,578.9 | 0:32 | 0:25 | 6 | 2,088.9 Kb | 960 | 943 | -8 980 | 726 | 4,655.7 | 0:33 | 0:25 | 6 | 2,105.8 Kb | 980 | 980 | 17 1000 | 706 | 4,768.7 | 0:34 | 0:24 | 6 | 2,192.0 Kb | 1000 | 1021 | 87 1020 | 686 | 4,839.9 | 0:35 | 0:23 | 6 | 2,176.0 Kb | 1020 | 1097 | -16 1040 | 666 | 4,956.9 | 0:36 | 0:23 | 6 | 2,265.5 Kb | 1040 | 1142 | 89 1060 | 646 | 5,075.0 | 0:36 | 0:22 | 6 | 2,293.3 Kb | 1060 | 1198 | 28 1080 | 626 | 5,193.1 | 0:37 | 0:21 | 6 | 2,320.2 Kb | 1080 | 1254 | 27 1100 | 606 | 5,299.7 | 0:38 | 0:21 | 6 | 2,327.6 Kb | 1100 | 1299 | 7 1120 | 586 | 5,416.4 | 0:39 | 0:20 | 6 | 2,371.6 Kb | 1120 | 1337 | 44 1140 | 566 | 5,527.9 | 0:40 | 0:19 | 6 | 2,397.0 Kb | 1140 | 1481 | 26 1160 | 546 | 5,639.7 | 0:40 | 0:19 | 6 | 2,453.3 Kb | 1160 | 1554 | 56 1180 | 526 | 5,737.8 | 0:41 | 0:18 | 6 | 2,458.4 Kb | 1180 | 1591 | 5 1200 | 506 | 5,833.3 | 0:42 | 0:17 | 6 | 2,493.7 Kb | 1200 | 1645 | 35 1220 | 486 | 5,937.2 | 0:43 | 0:17 | 6 | 2,535.4 Kb | 1220 | 1695 | 42 1240 | 466 | 6,025.2 | 0:44 | 0:16 | 6 | 2,537.3 Kb | 1240 | 1729 | 2 1260 | 446 | 6,120.2 | 0:45 | 0:16 | 6 | 2,580.1 Kb | 1260 | 1835 | 43 1280 | 426 | 6,210.4 | 0:46 | 0:15 | 6 | 2,618.2 Kb | 1280 | 1875 | 38 1300 | 406 | 6,318.1 | 0:47 | 0:14 | 6 | 2,693.6 Kb | 1300 | 1913 | 75 1320 | 386 | 6,436.6 | 0:48 | 0:14 | 6 | 2,737.1 Kb | 1320 | 1969 | 44 1340 | 366 | 6,533.0 | 0:50 | 0:13 | 6 | 2,747.4 Kb | 1340 | 2078 | 10 1360 | 346 | 6,652.1 | 0:50 | 0:12 | 6 | 2,829.6 Kb | 1360 | 2134 | 82 1380 | 326 | 6,761.0 | 0:51 | 0:12 | 6 | 2,837.9 Kb | 1380 | 2172 | 8 1400 | 306 | 6,864.5 | 0:52 | 0:11 | 6 | 2,866.0 Kb | 1400 | 2222 | 29 1420 | 286 | 6,982.2 | 0:54 | 0:10 | 6 | 2,918.2 Kb | 1420 | 2349 | 52 1440 | 266 | 7,101.8 | 0:55 | 0:10 | 6 | 2,979.3 Kb | 1440 | 2413 | 61 1460 | 246 | 7,192.1 | 0:56 | 0:09 | 6 | 2,965.4 Kb | 1460 | 2451 | -14 1480 | 226 | 7,297.0 | 0:57 | 0:08 | 6 | 3,020.8 Kb | 1480 | 2503 | 55 1500 | 206 | 7,423.2 | 0:58 | 0:08 | 6 | 3,069.3 Kb | 1500 | 2581 | 49 1520 | 186 | 7,539.6 | 0:59 | 0:07 | 6 | 3,115.2 Kb | 1520 | 2639 | 46 1540 | 166 | 7,642.4 | 1:00 | 0:06 | 6 | 3,120.0 Kb | 1540 | 2689 | 5 1560 | 146 | 7,739.8 | 1:01 | 0:05 | 6 | 3,151.0 Kb | 1560 | 2735 | 31 1580 | 126 | 7,847.3 | 1:02 | 0:04 | 6 | 3,188.2 Kb | 1580 | 2765 | 37 1600 | 106 | 7,949.4 | 1:03 | 0:04 | 6 | 3,215.6 Kb | 1600 | 2896 | 27 1620 | 86 | 8,052.3 | 1:04 | 0:03 | 6 | 3,270.9 Kb | 1620 | 2944 | 55 1640 | 66 | 8,146.9 | 1:05 | 0:02 | 6 | 3,282.7 Kb | 1640 | 2974 | 12 1660 | 46 | 8,250.9 | 1:07 | 0:01 | 6 | 3,330.9 Kb | 1660 | 3032 | 48 1680 | 26 | 8,345.0 | 1:08 | 0:01 | 6 | 3,344.3 Kb | 1680 | 3086 | 14 1700 | 6 | 8,432.1 | 1:09 | 0:00 | 6 | 3,367.3 Kb | 1700 | 3128 | 23 1720 | 1698 | 8,462.7 | 1:09 | 1:08 | 7 | 2,900.4 Kb | 1720 | 0 | -467 1740 | 1678 | 8,462.7 | 1:09 | 1:07 | 7 | 2,912.8 Kb | 1740 | 0 | 12 1760 | 1658 | 8,462.7 | 1:09 | 1:05 | 7 | 2,925.3 Kb | 1760 | 0 | 13 1780 | 1638 | 8,462.7 | 1:09 | 1:04 | 7 | 2,937.8 Kb | 1780 | 0 | 12 1800 | 1618 | 8,462.7 | 1:09 | 1:02 | 7 | 2,950.2 Kb | 1800 | 0 | 13 1820 | 1598 | 8,462.7 | 1:09 | 1:01 | 7 | 2,962.6 Kb | 1820 | 0 | 12 1840 | 1578 | 8,462.7 | 1:09 | 0:59 | 7 | 2,975.1 Kb | 1840 | 0 | 13 1860 | 1558 | 8,462.7 | 1:09 | 0:58 | 7 | 2,988.2 Kb | 1860 | 0 | 13 1880 | 1538 | 8,462.7 | 1:09 | 0:57 | 7 | 3,001.4 Kb | 1880 | 0 | 13 1900 | 1518 | 8,462.7 | 1:09 | 0:55 | 7 | 3,014.8 Kb | 1900 | 0 | 13 1920 | 1498 | 8,462.7 | 1:09 | 0:54 | 7 | 3,028.3 Kb | 1920 | 0 | 14 1940 | 1478 | 8,462.7 | 1:09 | 0:53 | 7 | 3,042.0 Kb | 1940 | 0 | 14 1960 | 1458 | 8,462.7 | 1:09 | 0:52 | 7 | 3,055.7 Kb | 1960 | 0 | 13 1980 | 1438 | 8,462.7 | 1:10 | 0:50 | 7 | 3,069.4 Kb | 1980 | 0 | 14 2000 | 1418 | 8,462.7 | 1:10 | 0:49 | 7 | 3,082.6 Kb | 2000 | 0 | 13 2020 | 1398 | 8,462.7 | 1:10 | 0:48 | 7 | 3,095.8 Kb | 2020 | 0 | 13 2040 | 1378 | 8,462.7 | 1:10 | 0:47 | 7 | 3,108.3 Kb | 2040 | 0 | 13 2060 | 1358 | 8,462.7 | 1:10 | 0:46 | 7 | 3,128.7 Kb | 2060 | 0 | 20 2080 | 1338 | 8,462.7 | 1:10 | 0:45 | 7 | 3,141.2 Kb | 2080 | 0 | 13 2100 | 1318 | 8,462.7 | 1:10 | 0:44 | 7 | 3,153.7 Kb | 2100 | 0 | 12 2120 | 1298 | 8,462.7 | 1:10 | 0:43 | 7 | 3,165.9 Kb | 2120 | 0 | 12 2140 | 1278 | 8,462.7 | 1:10 | 0:41 | 7 | 3,177.9 Kb | 2140 | 0 | 12 2160 | 1258 | 8,462.7 | 1:10 | 0:40 | 7 | 3,190.2 Kb | 2160 | 0 | 13 2180 | 1238 | 8,462.7 | 1:10 | 0:39 | 7 | 3,202.7 Kb | 2180 | 0 | 12 2200 | 1218 | 8,462.7 | 1:10 | 0:38 | 7 | 3,215.2 Kb | 2200 | 0 | 13 2220 | 1198 | 8,462.7 | 1:10 | 0:38 | 7 | 3,227.3 Kb | 2220 | 0 | 12 2240 | 1178 | 8,462.7 | 1:10 | 0:37 | 7 | 3,239.2 Kb | 2240 | 0 | 12 2260 | 1158 | 8,462.7 | 1:10 | 0:36 | 7 | 3,251.1 Kb | 2260 | 0 | 12 2280 | 1138 | 8,462.7 | 1:10 | 0:35 | 7 | 3,263.0 Kb | 2280 | 0 | 12 2300 | 1118 | 8,462.7 | 1:10 | 0:34 | 7 | 3,275.2 Kb | 2300 | 0 | 12 2320 | 1098 | 8,462.7 | 1:10 | 0:33 | 7 | 3,287.7 Kb | 2320 | 0 | 12 2340 | 1078 | 8,462.7 | 1:10 | 0:32 | 7 | 3,300.1 Kb | 2340 | 0 | 13 2360 | 1058 | 8,462.7 | 1:10 | 0:31 | 7 | 3,312.3 Kb | 2360 | 0 | 12 2380 | 1038 | 8,462.7 | 1:10 | 0:30 | 7 | 3,324.5 Kb | 2380 | 0 | 12 2400 | 1018 | 8,462.7 | 1:10 | 0:30 | 7 | 3,336.7 Kb | 2400 | 0 | 12 2420 | 998 | 8,462.7 | 1:10 | 0:29 | 7 | 3,348.9 Kb | 2420 | 0 | 12 2440 | 978 | 8,462.7 | 1:10 | 0:28 | 7 | 3,361.5 Kb | 2440 | 0 | 13 2460 | 958 | 8,462.7 | 1:10 | 0:27 | 7 | 3,374.4 Kb | 2460 | 0 | 13 2480 | 938 | 8,462.7 | 1:10 | 0:26 | 7 | 3,387.1 Kb | 2480 | 0 | 13 2500 | 918 | 8,462.7 | 1:11 | 0:26 | 7 | 3,399.8 Kb | 2500 | 0 | 12 2520 | 898 | 8,462.7 | 1:11 | 0:25 | 7 | 3,412.5 Kb | 2520 | 0 | 13 2540 | 878 | 8,462.7 | 1:11 | 0:24 | 7 | 3,425.2 Kb | 2540 | 0 | 13 2560 | 858 | 8,462.7 | 1:11 | 0:23 | 7 | 3,437.9 Kb | 2560 | 0 | 12 2580 | 838 | 8,462.7 | 1:11 | 0:23 | 7 | 3,450.5 Kb | 2580 | 0 | 13 2600 | 818 | 8,462.7 | 1:11 | 0:22 | 7 | 3,463.2 Kb | 2600 | 0 | 13 2620 | 798 | 8,462.7 | 1:11 | 0:21 | 7 | 3,475.8 Kb | 2620 | 0 | 12 2640 | 778 | 8,462.7 | 1:11 | 0:21 | 7 | 3,488.4 Kb | 2640 | 0 | 13 2660 | 758 | 8,462.7 | 1:11 | 0:20 | 7 | 3,501.1 Kb | 2660 | 0 | 13 2680 | 738 | 8,462.7 | 1:11 | 0:19 | 7 | 3,514.0 Kb | 2680 | 0 | 13 2700 | 718 | 8,462.7 | 1:11 | 0:19 | 7 | 3,526.7 Kb | 2700 | 0 | 12 2720 | 698 | 8,462.7 | 1:11 | 0:18 | 7 | 3,539.4 Kb | 2720 | 0 | 13 2740 | 678 | 8,462.7 | 1:11 | 0:17 | 7 | 3,552.0 Kb | 2740 | 0 | 13 2760 | 658 | 8,462.7 | 1:11 | 0:17 | 7 | 3,564.7 Kb | 2760 | 0 | 12 2780 | 638 | 8,462.7 | 1:11 | 0:16 | 7 | 3,577.3 Kb | 2780 | 0 | 13 2800 | 618 | 8,462.7 | 1:11 | 0:15 | 7 | 3,590.0 Kb | 2800 | 0 | 13 2820 | 598 | 8,462.7 | 1:11 | 0:15 | 7 | 3,602.6 Kb | 2820 | 0 | 12 2840 | 578 | 8,462.7 | 1:11 | 0:14 | 7 | 3,615.3 Kb | 2840 | 0 | 13 2860 | 558 | 8,462.7 | 1:11 | 0:14 | 7 | 3,628.0 Kb | 2860 | 0 | 13 2880 | 538 | 8,462.7 | 1:11 | 0:13 | 7 | 3,640.7 Kb | 2880 | 0 | 12 2900 | 518 | 8,462.7 | 1:11 | 0:12 | 7 | 3,653.4 Kb | 2900 | 0 | 13 2920 | 498 | 8,462.7 | 1:11 | 0:12 | 7 | 3,666.3 Kb | 2920 | 0 | 13 2940 | 478 | 8,462.7 | 1:11 | 0:11 | 7 | 3,679.0 Kb | 2940 | 0 | 13 2960 | 458 | 8,462.7 | 1:12 | 0:11 | 7 | 3,691.8 Kb | 2960 | 0 | 12 2980 | 438 | 8,462.7 | 1:12 | 0:10 | 7 | 3,704.4 Kb | 2980 | 0 | 13 3000 | 418 | 8,462.7 | 1:12 | 0:10 | 7 | 3,717.0 Kb | 3000 | 0 | 13 3020 | 398 | 8,462.7 | 1:12 | 0:09 | 7 | 3,730.3 Kb | 3020 | 0 | 13 3040 | 378 | 8,462.7 | 1:12 | 0:08 | 7 | 3,743.2 Kb | 3040 | 0 | 13 3060 | 358 | 8,462.7 | 1:12 | 0:08 | 7 | 3,756.0 Kb | 3060 | 0 | 13 3080 | 338 | 8,462.7 | 1:12 | 0:07 | 7 | 3,768.6 Kb | 3080 | 0 | 12 3100 | 318 | 8,462.7 | 1:12 | 0:07 | 7 | 3,781.2 Kb | 3100 | 0 | 13 3120 | 298 | 8,462.7 | 1:12 | 0:06 | 7 | 3,793.8 Kb | 3120 | 0 | 12 3140 | 278 | 8,462.7 | 1:12 | 0:06 | 7 | 3,806.4 Kb | 3140 | 0 | 13 3160 | 258 | 8,462.7 | 1:12 | 0:05 | 7 | 3,819.0 Kb | 3160 | 0 | 13 3180 | 238 | 8,462.7 | 1:12 | 0:05 | 7 | 3,831.6 Kb | 3180 | 0 | 12 3200 | 218 | 8,462.7 | 1:12 | 0:04 | 7 | 3,844.3 Kb | 3200 | 0 | 13 3220 | 198 | 8,462.7 | 1:12 | 0:04 | 7 | 3,857.0 Kb | 3220 | 0 | 13 3240 | 178 | 8,462.7 | 1:12 | 0:03 | 7 | 3,869.6 Kb | 3240 | 0 | 12 3260 | 158 | 8,462.7 | 1:12 | 0:03 | 7 | 3,882.3 Kb | 3260 | 0 | 13 3280 | 138 | 8,462.7 | 1:12 | 0:03 | 7 | 3,894.9 Kb | 3280 | 0 | 12 3300 | 118 | 8,462.7 | 1:12 | 0:02 | 7 | 3,907.5 Kb | 3300 | 0 | 13 3320 | 98 | 8,462.7 | 1:12 | 0:02 | 7 | 3,919.9 Kb | 3320 | 0 | 12 3340 | 78 | 8,462.7 | 1:12 | 0:01 | 7 | 3,932.5 Kb | 3340 | 0 | 13 3360 | 58 | 8,462.7 | 1:12 | 0:01 | 7 | 3,945.1 Kb | 3360 | 0 | 13 3380 | 38 | 8,462.7 | 1:12 | 0:00 | 7 | 3,957.7 Kb | 3380 | 0 | 12 3400 | 18 | 8,462.7 | 1:13 | 0:00 | 7 | 3,970.3 Kb | 3400 | 0 | 13 3418 | 0 | 8,462.7 | 1:13 | 0:00 | 8 | 3,552.5 Kb | 3418 | 0 | -418 top.location = 'index.php?op=view'

[scottsaxton]

Still messing around.  It looks like my changes to my php.ini file don't take affect.  I tried changing the max_execution_time = 0 and also tried max_execution_time = 1.....and it still seems to run longer then that.

I also tried putting a php.ini file in my public_html folder and the changes don't seem to take affect there?

Any ideas...?

[scottsaxton]

Yep.................. i am adding more again!!!!!!!!!!!!!!!!!!!!!!!!11

I finally got someone good at my webhost and they said that their max execution time is 60 seconds and their max_memory is 64MB.....I don't think the memory limit is the issue, but I bet the max_execution time is.   

Any suggestions!

[XML-Sitemaps Support]

Hello,

if you see this in the output:

Code Select Expand

3,552.5 Kb | 3418 | 0 | -418 top.location = 'index.php?op=view'then sitemap generator has been completed (and is redirected to sitemap view page).

[scottsaxton]

I only see .... Kb | 3418 | 0 | -418 top.location = 'index.php?op=view'
when i set the depth level to 6 other than that i still can not complete an entire crawl.

[scottsaxton]

Does the Maximum execution time, seconds: parameter in the sitemap.xml configuration screen work.  I have tried setting to different low values i.e. 15 or 20 seconds...and it seems to have no impact on the crawl.

Here is where I am today.  I know that my web host max execution time is 60 seconds.  So I am trying to find a way to get a complete crawl by "Save the script state, every X seconds:"

[XML-Sitemaps Support]

That option means that sitemap generator will interrupt itself after specified time, it will not create sitemap untill whole website is crawled though.

[scottsaxton]

Is there a way that you can give me a commad line function so I can initiate a crawl from my web server that will continue from an interruption?

[XML-Sitemaps Support]

An exact command line can be found on Crawling page of sitemap generator, it looks similar to:
/usr/local/bin/php /path/to/generator/runcrawl.php

[scottsaxton]

If I use the command line from the crawl page, will it continue on where it left off if I run the same command again?