Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to get results with DuckDuckGo engine #368

Closed
adbenitez opened this issue Apr 3, 2021 · 3 comments · Fixed by #373
Closed

Fail to get results with DuckDuckGo engine #368

adbenitez opened this issue Apr 3, 2021 · 3 comments · Fixed by #373

Comments

@adbenitez
Copy link
Contributor

The reply is received but it is not what it is expected, steps to reproduce:

  1. Set user agent to USER_AGENTS[1] == Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100 101 Firefox/22.0
  2. Execute howdoi -e duckduckgo format date bash
  3. Response HTML will be something like:
<!DOCTYPE html><html lang="en-US" class="no-js has-zcm  no-theme"><head><meta name="description" content="DuckDuckGo. Privacy, Simplified."><meta http-equiv="content-type" content="text/html; charset=utf-8"><title>site:stackoverflow.com format date bash at DuckDuckGo</title><link rel="stylesheet" href="/s1971.css" type="text/css"><link rel="stylesheet" href="/r1971.css" type="text/css"><meta name="robots" content="noindex,nofollow"><meta name="referrer" content="origin"><meta name="apple-mobile-web-app-title" content="site:stackoverflow.com format date bash"><link rel="preconnect" href="https://links.duckduckgo.com"><link rel="preload" href="/font/ProximaNova-Reg-webfont.woff2" as="font" type="font/woff2" crossorigin="anonymous" /><link rel="preload" href="/font/ProximaNova-Sbold-webfont.woff2" as="font" type="font/woff2" crossorigin="anonymous" /><link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" /><link id="icon60" rel="apple-touch-icon" href="/assets/icons/meta/DDG-iOS-icon_60x60.png?v=2"/><link id="icon76" rel="apple-touch-icon" sizes="76x76" href="/assets/icons/meta/DDG-iOS-icon_76x76.png?v=2"/><link id="icon120" rel="apple-touch-icon" sizes="120x120" href="/assets/icons/meta/DDG-iOS-icon_120x120.png?v=2"/><link id="icon152" rel="apple-touch-icon" sizes="152x152" href="/assets/icons/meta/DDG-iOS-icon_152x152.png?v=2"/><link rel="image_src" href="/assets/icons/meta/DDG-icon_256x256.png"/><script type="text/javascript">var ct,fd,fq,it,iqa,iqm,iqs,iqp,iqq,qw,dl,ra,rv,rad,r1hc,r1c,r2c,r3c,rfq,rq,rds,rs,rt,rl,y,y1,ti,tig,iqd,shfl,shrl,locale,settings_js_version='s2480.js',is_twitter='',rpl=1;fq=0;fd=1;it=0;iqa=0;iqbi=0;iqm=0;iqs=1;iqp=0;iqq=0;qw=4;dl='en';ct='CU';iqd=0;r1hc=0;r1c=0;r3c=0;rq='site%3Astackoverflow.com%20format%20date%20bash';rqd="site:stackoverflow.com format date bash";rfq=0;rt='';ra='hj';rv='';rad='';rds=30;rs=0;spice_version='2000';spice_paths='{}';locale='en_US';settings_url_params={};rl='us-en';shfl=0;shrl='us-en';rlo=0;df='';ds='';sfq='format%20date%20bash%20SITE%3Astackoverflow.com';iar='';vqd='3-282771024447202343161713231520080655472-199101826342951495864809732414974624031';safe_ddg=0;show_covid=0;</script><meta name="viewport" content="width=device-width, initial-scale=1" /><meta name="HandheldFriendly" content="true" /><meta name="apple-mobile-web-app-capable" content="no" /></head><body class="body--serp"><input id="state_hidden" name="state_hidden" type="text" size="1"><!-- Ignore this input please --><div id="spacing_hidden_wrapper"><div id="spacing_hidden"></div></div><script type="text/javascript" src="/lib/l119.js"></script><script type="text/javascript" src="/locale/en_US/duckduckgo14.js"></script><script type="text/javascript" src="/util/u538.js"></script><script type="text/javascript" src="/d2927.js"></script><div class="site-wrapper  js-site-wrapper"><div class="welcome-wrap js-welcome-wrap"></div><div id="header_wrapper" class="header-wrap js-header-wrap"><div id="header" class="header  cw"><div class="header__search-wrap"><a tabindex="-1" href="/?t=hj" class="header__logo-wrap js-header-logo"><span class="header__logo js-logo-ddg">DuckDuckGo</span></a><div class="header__content  header__search"><form id="search_form" class="search--adv  search--header js-search-form" name="x" action="/"><input type="text" name="q" tabindex="1" autocomplete="off" id="search_form_input" class="search__input search__input--adv js-search-input" value="site:stackoverflow.com format date bash"><input id="search_form_input_clear" class="search__clear  js-search-clear" type="button" tabindex="3" value="X"/><input id="search_button" class="search__button  js-search-button" type="submit" tabindex="2" value="S" /><a id="search_dropdown" class="search__dropdown" href="javascript:;" tabindex="4"></a><div id="search_elements_hidden" class="search__hidden  js-search-hidden"></div></form></div></div><div class="zcm-wrap-wrap"><div id="duckbar" class="zcm-wrap  zcm-wrap--header  is-noscript-hidden"></div><div class="zcm--right js-zcm-right"></div></div></div><div class="header--aside js-header-aside"></div></div><div id="zero_click_wrapper" class="zci-wrap"></div><div id="vertical_wrapper" class="verticals"></div><div id="web_content_wrapper" class="content-wrap "><div class="serp__top-right  js-serp-top-right"></div><div class="serp__bottom-right  js-serp-bottom-right"><div class="js-feedback-btn-wrap"></div></div><div class="cw"><div id="links_wrapper" class="serp__results js-serp-results"><div class="results--main"><div class="search-filters-wrap"><div class="js-search-filters search-filters"></div></div><noscript><meta http-equiv="refresh" content="0;URL=/html?q=site%3Astackoverflow.com%20format%20date%20bash"><link href="/css/noscript.css" rel="stylesheet" type="text/css"><div class="msg msg--noscript"><p class="msg-title--noscript">You are being redirected to the non-JavaScript site.</p>Click <a href="/html/?q=site%3Astackoverflow.com%20format%20date%20bash">here</a> if it doesn't happen automatically.</div></noscript><div id="message" class="results--message"></div><div class="ia-modules js-ia-modules"></div><div id="ads" class="results--ads results--ads--main is-invisible js-results-ads"></div><div id="links" class="results is-invisible js-results"></div></div><div class="results--sidebar js-results-sidebar"><div class="sidebar-modules js-sidebar-modules"></div><div class="is-invisible js-sidebar-ads"></div></div></div></div></div><div id="bottom_spacing2"> </div></div><script type="text/javascript"></script><script type="text/JavaScript">function nrji() {nrj('/t.js?q=site%3Astackoverflow.com%20format%20date%20bash&l=us-en&s=0&dl=en&ct=CU&ss_mkt=us&p_ent=&ex=-1');DDG.deep.initialize('/d.js?q=site%3Astackoverflow.com%20format%20date%20bash&l=us-en&s=0&a=hj&dl=en&ct=CU&ss_mkt=us&vqd=3-282771024447202343161713231520080655472-199101826342951495864809732414974624031&p_ent=&ex=-1&sp=1');;};DDG.ready(nrji, 1);</script><script src="/g2564.js"></script><script type="text/javascript">DDG.page = new DDG.Pages.SERP({ showSafeSearch: 0, instantAnswerAds: false, hostRegion: "use" });</script><div id="z2"> </div><div id="z"></div></body></html><script type="text/JavaScript">DDG.index = DDG.index || {}; DDG.index.signalSummary = "";</script>

Using duckduckgo.com/html?q=... instead of duckduckgo.com/?q=... does work, when not blocked with message mentioned in #367
@gleitz
Copy link
Owner

gleitz commented Apr 3, 2021

Should we perhaps remove that user agent? Or do you think we need to handle the DDG response (/html vs /) better?

@adbenitez
Copy link
Contributor Author

I think it is better to use /html directly instead of expecting to be redirected nicely

@gleitz
Copy link
Owner

gleitz commented Apr 7, 2021

I would accept a PR for that

@adbenitez adbenitez mentioned this issue Apr 7, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

use duckduckgo no-javascript version directly
2 participants
@gleitz @adbenitez