From a712755d4a929e368e7f0bfc355ba5b31926aa8d Mon Sep 17 00:00:00 2001 From: Matthijs Iriks Date: Mon, 5 Jun 2023 13:47:45 +0200 Subject: [PATCH] Update bad-user-agents.list Added the following : aspiegel Datanyze spaziodati OPPO\sA33 AspiegelBot undefined Undefined --- _generator_lists/bad-user-agents.list | 227 +++++++++++++------------- 1 file changed, 117 insertions(+), 110 deletions(-) diff --git a/_generator_lists/bad-user-agents.list b/_generator_lists/bad-user-agents.list index 89fdb2dd425..7ee0b9dfc01 100755 --- a/_generator_lists/bad-user-agents.list +++ b/_generator_lists/bad-user-agents.list @@ -3,22 +3,21 @@ 404checker 404enemy 80legs -ADmantX -AIBOT -ALittle\ Client -ASPSeek Abonti Aboundex Aboundexbot Acunetix +ADmantX AdsTxtCrawlerTP AfD-Verbotsverfahren AhrefsBot +AIBOT AiHitBot Aipbot Alexibot -AllSubmitter +ALittle\ Client Alligator +AllSubmitter AlphaBot Anarchie Anarchy @@ -27,31 +26,33 @@ Ankit Anthill Apexoo Aspiegel +AspiegelBot +ASPSeek Asterias Atomseobot Attach AwarioRssBot AwarioSmartBot -BBBike -BDCbot -BDFetch -BLEXBot BackDoorBot -BackStreet -BackWeb Backlink-Ceck BacklinkCrawler +BackStreet +BackWeb Badass Bandit Barkrowler BatchFTP Battleztar\ Bazinga +BBBike +BDCbot +BDFetch BetaBot Bigfoot Bitacle -BlackWidow Black\ Hole Blackboard +BlackWidow +BLEXBot Blow BlowFish Boardreader @@ -66,17 +67,15 @@ BuiltWith Bullseye BunnySlippers BuzzSumo -CATExplorador -CCBot -CODE87 -CSHttp Calculon +CATExplorador CazoodleBot +CCBot Cegbfeieh CensysInspect -CheTeam CheeseBot CherryPicker +CheTeam ChinaClaw Chlooe Citoid @@ -84,6 +83,7 @@ Claritybot Cliqzbot Cloud\ mapping Cocolyzebot +CODE87 Cogentbot Collector Copier @@ -95,20 +95,20 @@ Crawling\ at\ Home\ Project CrazyWebCrawler Crescent CrunchBot +CSHttp Curious Custo CyotekWebCopy -DBLBot -DIIbot -DSearch -DTS\ Agent -DataCha0s DatabaseDriverMysqli +DataCha0s +Datanyze +DBLBot Demon Deusu Devil Digincore DigitalPebble +DIIbot Dirbuster Disco Discobot @@ -117,23 +117,25 @@ Dispatch DittoSpyder DnBCrawler-Analytics DnyzBot -DomCopBot DomainAppender DomainCrawler +Domains\ Project DomainSigmaCrawler DomainStatsBot -Domains\ Project +DomCopBot Dotbot Download\ Wonder Dragonfly Drip -ECCP/1.0 -EMail\ Siphon -EMail\ Wolf +DSearch +DTS\ Agent EasyDL Ebingbong +ECCP/1.0 Ecxi EirGrabber +EMail\ Siphon +EMail\ Wolf EroCrawler Evil Exabot @@ -145,8 +147,8 @@ Extreme\ Picture\ Finder EyeNetIE Ezooms FDM -FHscan FemtosearchBot +FHscan Fimap Firefox/7.0 FlashGet @@ -158,43 +160,42 @@ Fuzz FyberSpider Fyrebot G-i-g-a-b-o-t -GT::WWW GalaxyBot Genieo GermCrawler +Getintent GetRight GetWeb -Getintent Gigabot Go!Zilla Go-Ahead-Got-It -GoZilla Gotit -GrabNet +GoZilla Grabber +GrabNet Grafula GrapeFX GrapeshotCrawler GridBot -HEADMasterSEO -HMView -HTMLparser -HTTP::Lite -HTTrack +GT::WWW Haansoft HaosouSpider Harvest Havij +HEADMasterSEO Heritrix Hloader +HMView HonoluluBot +HTMLparser +HTTP::Lite +HTTrack Humanlinks HybridBot -IDBTE4M -IDBot -IRLbot Iblog Id-search +IDBot +IDBTE4M IlseBot Image\ Fetch Image\ Sucker @@ -204,18 +205,19 @@ InfoNaviRobot InfoTekies Intelliseek InterGET -InternetSeer Internet\ Ninja +InternetSeer Iria +IRLbot Iskanie IstellaBot -JOC\ Web\ Spider JamesBOT Jbrofuzz JennyBot JetCar Jetty JikeSpider +JOC\ Web\ Spider Joomla Jorgee JustView @@ -225,8 +227,6 @@ Keybot\ Translation-Search-Machine Keyword\ Density Kinza Kozmosbot -LNSpiderguy -LWP::Simple Lanshanbot Larbin Leap @@ -239,12 +239,12 @@ Libwhisker LieBaoFast Lightspeedsystems Likse -LinkScan -LinkWalker Linkbot LinkextractorPro LinkpadBot +LinkScan LinksManager +LinkWalker LinqiaMetadataDownloaderBot LinqiaRSSBot LinqiaScrapeBot @@ -252,15 +252,9 @@ Lipperhey Lipperhey\ Spider Litemage_walker Lmspider +LNSpiderguy Ltx71 -MFC_Tear_Sample -MIDown\ tool -MIIxpc -MJ12bot -MQQBrowser -MSFrontPage -MSIECrawler -MTRobot +LWP::Simple Mag-Net Magnet Mail.RU_Bot @@ -279,73 +273,81 @@ Meanpathbot Mediatoolkitbot MegaIndex.ru Metauri +MFC_Tear_Sample MicroMessenger Microsoft\ Data\ Access Microsoft\ URL\ Control +MIDown\ tool +MIIxpc Minefield Mister\ PiX +MJ12bot Moblie Safari Mojeek Mojolicious MolokaiBot Morfeus\ Fucking\ Scanner Mozlila +MQQBrowser Mr.4x3 +MSFrontPage +MSIECrawler Msrabot +MTRobot Musobot -NICErsPRO -NPbot Name\ Intelligence Nameprotect Navroad NearSite Needle Nessus +Net\ Vampire NetAnts +Netcraft NetLyzer NetMechanic NetSpider -NetZIP -Net\ Vampire -Netcraft Nettrack Netvibes +NetZIP NextGenSearchBot Nibbler +NICErsPRO Niki-bot Nikto NimbleCrawler Nimbostratus Ninja Nmap +NPbot Nuclei Nutch Octopus Offline\ Explorer Offline\ Navigator OnCrawl -OpenLinkProfiler -OpenVAS Openfind +OpenLinkProfiler +Openvas Openvas +OPPO\sA33 OrangeBot OrangeSpider OutclicksBot OutfoxBot -PECL::HTTP -PHPCrawl -POE-Component-Client-HTTP +Page\ Analyzer PageAnalyzer PageGrabber PageScorer PageThing.com -Page\ Analyzer Pandalytics Panscient Papa\ Foto Pavuk +PECL::HTTP PeoplePal Petalbot +PHPCrawl Pi-Monster Picscout Picsearch @@ -355,9 +357,10 @@ Pimonster Pixray PleaseCrawl Pockey +POE-Component-Client-HTTP +Probethenet ProPowerBot ProWebWalker -Probethenet Proximic Psbot Pu_iN @@ -366,7 +369,6 @@ PxBroker PyCurl QueryN\ Metasearch Quick-Crawler -RSSingBot RankActive RankActiveLinkBot RankFlex @@ -375,25 +377,19 @@ RankingBot2 Rankivabot RankurBot Re-re -ReGet RealDownload Reaper RebelMouse Recorder RedesScrapy +ReGet RepoMonkey Ripper RocketCrawler Rogerbot -SBIder -SEOkicks -SEOkicks-Robot -SEOlyticsCrawler -SEOprofiler -SEOstats -SISTRIX -SMTBot +RSSingBot SalesIntelligent +SBIder ScanAlert Scanbot ScoutJet @@ -410,22 +406,29 @@ Semrush SemrushBot SentiBot SenutoBot -SeoSiteCheckup SeobilityBot +SEOkicks +SEOkicks-Robot +SEOlyticsCrawler Seomoz +SEOprofiler +SeoSiteCheckup +SEOstats Shodan Siphon +SISTRIX +Site\ Sucker +Sitebeam SiteCheckerBotCrawler SiteExplorer +Siteimprove SiteLockSpider SiteSnagger SiteSucker -Site\ Sucker -Sitebeam -Siteimprove Sitevigil SlySearch SmartDownload +SMTBot Snake Snapbot Snoopy @@ -462,8 +465,8 @@ TeleportPro Telesoft Telesphoreo Telesphorep -TheNomad The\ Intraformant +TheNomad Thumbor TightTwatBot Titan @@ -479,15 +482,16 @@ TurnitinBot TwengaBot Twice Typhoeus -URLy.Warning -URLy\ Warning +Undefined UnisterBot Upflow +URLy.Warning +URLy\ Warning V-BOT -VB\ Project -VCI Vacuum Vagabondo +VB\ Project +VCI VelenPublicWebCrawler VeriCiteCrawler VidibleScraper @@ -495,51 +499,45 @@ Virusdie VoidEYE Voil Voltron -WASALive-Bot -WBSearchBot -WEBDAV -WISENutbot -WPScan -WWW-Collector-E -WWW-Mechanize -WWW::Mechanize -WWWOFFLE Wallpapers Wallpapers/3.0 WallpapersHD -WeSEE +WASALive-Bot +WBSearchBot +Web\ Auto +Web\ Collage +Web\ Enhancer +Web\ Fetch +Web\ Fuck +Web\ Pix +Web\ Sauger +Web\ Sucker +Webalta WebAuto WebBandit WebCollage WebCopier +WEBDAV WebEnhancer WebFetch WebFuck WebGo\ IS WebImageCollector WebLeacher +WebmasterWorldForumBot WebPix WebReaper WebSauger -WebStripper -WebSucker -WebWhacker -WebZIP -Web\ Auto -Web\ Collage -Web\ Enhancer -Web\ Fetch -Web\ Fuck -Web\ Pix -Web\ Sauger -Web\ Sucker -Webalta -WebmasterWorldForumBot Webshag +Website\ Quester WebsiteExtractor WebsiteQuester -Website\ Quester Webster +WebStripper +WebSucker +WebWhacker +WebZIP +WeSEE Whack Whacker Whatweb @@ -547,10 +545,16 @@ Who.is\ Bot Widow WinHTTrack WiseGuys\ Robot +WISENutbot Wonderbot Woobot Wotbox Wprecon +WPScan +WWW-Collector-E +WWW-Mechanize +WWW::Mechanize +WWWOFFLE Xaldon\ WebSpider Xaldon_WebSpider Xenu @@ -569,6 +573,7 @@ adscanner archive.org_bot arquivo-web-crawler arquivo.pt +aspiegel autoemailspider backlink-check cah.io.community @@ -623,12 +628,14 @@ sitechecker.pro siteripz sogouspider sp_auditbot +spaziodati spyfu sysscan tAkeOut trendiction.com trendiction.de ubermetrics-technologies.com +undefined voyagerx.com webgains-bot webmeup-crawler