GitHub - dadmor/anotherstrangecrawler

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
inc		inc
workspace		workspace
EMAIL-TEST.sh		EMAIL-TEST.sh
META-VERSION-TEST.sh		META-VERSION-TEST.sh
PAGE-INSIGHT-CMS-TEST.sh		PAGE-INSIGHT-CMS-TEST.sh
WP-404-TEST.sh		WP-404-TEST.sh
app.js		app.js
mongo-export-csv.sh		mongo-export-csv.sh
parser.sh		parser.sh
readme.html		readme.html

Repository files navigation

<style>
.bash{
	color: #00d2ff;
	background: #4e4e4e;
	padding: 2px 4px;
	line-height: 2em;
	border-radius: 3px;
}
</style>

<body style="background:#333">
<a href="https://dadmor.github.io/anotherstrangecrawler/readme.html">
	PROJRCT PAGE
</a>
<pre style="margin:0 auto; display:table; color:#71e148">
<span style="color:#e1cf48">==========================================</span>
BingUltraSpeedCrw
require: phantomjs, mongoDB
<span style="color:#e1cf48">==========================================</span>
Install: Copy files to your directory,
install phantomjs, mongoDB and enjoy

------------------------------------------

Getting started:

Start MongoDb
<span class="bash">$ sudo mongod</span>

optionaly set database
<span class="bash">$ mongo</span>
<span class="bash">$ use local</span>

go to disk location
<span class="bash">$ cd ~/your directory</span>

<span style="color:#e1cf48">==========================================</span>

1. GET DOMAINS LIST FROM BING 

<span class="bash">$ sudo bash parser.sh --query --proxurl</span>

run it many times to diffrent queries

<span style="color:#e1cf48">==========================================</span>

OR TESTING DATABASE DIRECTLY:

<span style="color:#e1cf48">------------------------------------------</span>
1. PREPARE FOR TEST
<span style="color:#e1cf48">------------------------------------------</span>

Export all
<span class="bash">$ sudo bash mongo-export-csv.sh</span>

<span style="color:#e1cf48">------------------------------------------</span>
2. TESTS
<span style="color:#e1cf48">------------------------------------------</span>

404 Test
<span class="bash">$ sudo bash WP-404-TEST.sh</span>
Check wp-content and readme.html http code
<span style="color:#e1cf48">------------------------------------------</span>

Meta Generator Test
<span class="bash">$ sudo META-VERSION-TEST.sh</span>
Find meta_name="generator" and return it
<span style="color:#e1cf48">------------------------------------------</span>

Page insight CMS test
<span class="bash">$ sudo PAGE-INSIGHT-CMS-TEST.sh</span>
Find cms type and return it
<span style="color:#e1cf48">------------------------------------------</span>

Page insight score test
<span class="bash">$ sudo PAGE-INSIGHT-SCORE-TEST.sh</span>
Find speed score and return it
<span style="color:#e1cf48">------------------------------------------</span>

Grabage emails test
...
<span style="color:#e1cf48">------------------------------------------</span>

</pre>
</body>