|
@@ -291,7 +291,7 @@ Make sure the robots folder exists. All robots.txt files are stored in the robot
|
|
|
You can turn off checking for robots.txt files by commenting out the line calling the "checkrobots" function inside of cr.c.
|
|
You can turn off checking for robots.txt files by commenting out the line calling the "checkrobots" function inside of cr.c.
|
|
|
<br>
|
|
<br>
|
|
|
<br>
|
|
<br>
|
|
|
-If crawling through hyperlinks on a page, the following file types are accepted: html, htm, xhtml, txt, php, asp. Links containing parameters are ignored. These limitations do not apply to pages directly submitted by people.
|
|
|
|
|
|
|
+If crawling through hyperlinks on a page, the following file types are accepted: html, htm, xhtml, shtml, txt, php, asp. Links containing parameters are ignored. These limitations do not apply to pages directly submitted by people.
|
|
|
<br>
|
|
<br>
|
|
|
<br>
|
|
<br>
|
|
|
<h3>Start the core server</h3>
|
|
<h3>Start the core server</h3>
|
wibyweb