iGP UdZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZmZmZmZmZej8r dd lmZdd lm Z d d ddddZ!e"e#e#e$zfe%d<ddgZ&e'e#e%d<dZ(e$e%d<dZ)e$e%d<dZ*e$e%d< dZ+e#e%d< dddddZ,e"e#e#fe%d <da-e e%d!< eZ.e#e%d"<d#e fd$Z/d%e#e0zd&e#d'd(d#dfd)Z1d%e#d'd(d#e#fd*Z2 dZd+e d,e#d-e#d#e#dzfd.Z3d/d0d1d2d3d4d5d6d7d8d9d:d; Z4e"e#e#fe%d<<id=d>d?d>d7d>d@d>dAd>dBd>dCd>dDd>dEdFdGd>dHd>dId>dJd>dKd>dLd>dMd>dNd>d>d>d>d>d>d>d1d1d1dO Z5e"e#e#fe%dP<d%e#d#e#fdQZ6d%e#d'd(d#dfdRZ7dSefdTZ8dUdVd#efdWZ9dXe fdYZ:y)[a DuckDuckGo WEB ~~~~~~~~~~~~~~ DDG's WEB search: - DuckDuckGo WEB : ``https://links.duckduckgo.com/d.js?q=..`` (HTTP GET) - DuckDuckGo WEB no-AI: ``https://noai.duckduckgo.com/`` (HTTP GET) - DuckDuckGo WEB html : ``https://html.duckduckgo.com/html`` (HTTP POST no-JS / form data) - DuckDuckGo WEB lite : ``https://lite.duckduckgo.com/lite`` (HTTP POST no-JS / form data) DDG's content search / see engine ``duckduckgo_extra.py`` - DuckDuckGo Images : ``https://duckduckgo.com/i.js??q=...&vqd=...`` - DuckDuckGo Videos : ``https://duckduckgo.com/v.js??q=...&vqd=...`` - DuckDuckGo News : ``https://duckduckgo.com/news.js??q=...&vqd=...`` .. hint:: For WEB searches and to determine the ``vqd`` value, DDG-html (no-JS) is used. Special features of the no-JS services (DDG-lite & DDG-html): - The no-JS clients receive a form that contains all the controlling parameters. - When the form data is submitted, a real WEB browser sets the HTTP *Sec-Fetch* headers. HTML ``
``, HTTP-Headers & DDG's bot Blocker: The HTTP User-Agent_ (see below) is generated by the WEB-client and are checked by DDG's bot blocker. To simulate the behavior of a real browser session, it might be necessary to evaluate additional headers. For example, in the response from DDG, the Referrer-Policy_ is always set to ``origin``. A real browser would then include the following header in the next request:: Referer: https://html.duckduckgo.com/ The fields of the html-form are reverse-engineered from DDG-html and may be subject to additional bot detection mechanisms and breaking changes in the future. Query field: Intro page: https://html.duckduckgo.com/html/ - ``q`` (str): Search query string - ``b`` (str): Beginning parameter - empty string for first page requests. If a second page is requested, this field is not set! Search options: - ``kl`` (str): Keyboard language/region code (e.g. 'en-us' default: 'wt-wt') - ``df`` (str): Time filter, maps to values like 'd' (day), 'w' (week), 'm' (month), 'y' (year) The key/value pairs ``df`` and ``kl`` are additional saved in the cookies, example:: Cookie: kl=en-us; df=m *next page* form fields: - ``nextParams`` (str): Continuation parameters from previous page response, typically empty string. Opposite of ``b``; this field is not set when requesting the first result page. - ``api`` (str): API endpoint identifier, typically 'd.js' - ``o`` (str): Output format, typically ``json`` - ``v`` (str): Typically ``l`` for subsequent pages - ``dc`` (int): Display count - value equal to offset (s) + 1 - ``s`` (int): Search offset for pagination - ``vqd`` (str): Validation query digest General assumptions regarding DDG's bot blocker: - Except ``Cookie: kl=..; df=..`` DDG does not use cookies in any of its services. - DDG does not accept queries with more than 499 chars - The ``vqd`` value ("Validation query digest") is needed to pass DDG's bot protection and is used by all request to DDG. - The ``vqd`` value is generally not needed for the first query (intro); it is only required when additional pages are accessed (or when new content needs to be loaded for the query while scrolling). - The second page (additional content) for a query cannot be requested without ``vqd``, as this would lead to an immediate blocking, since such a use-case does not exist in the process flows provided by DDG (and is a clear indication of a bot). The following HTTP headers are being evaluated (and may possibly be responsible for issues): User-Agent_: The HTTP User-Agent is also involved in the formation of the vqd value, read `DuckDuckGo Bot Detection Research & Solution`_. However, it is not checked whether the UA is a known header. However, it is possible that certain UA headers (such as curl) are filtered. Sec-Fetch-Mode_: In the past, Sec-Fetch-Mode had to be set to 'navigate', otherwise there were problems with the bot blocker.. I don't know if DDG still evaluates this header today Accept-Language_: DDG-Lite and DDG-HTML TRY to guess user's preferred language from the HTTP ``Accept-Language``. Optional the user can select a region filter (but not a language). In DDG's bot blocker, the IP will be blocked (DDG does not have a client session!) - As far as is known, it is possible to remove a un-blocked an IP by executing a DDG query in a real web browser over the blocked IP (at least that's my assumption). How exactly the blocking mechanism currently works is not fully known, and there were also changes to the bot blocker in the period of Q3/Q4 2025: in the past, the IP blocking was implemented as a 'sliding window' (unblock after about 1 hour without requests from this IP) Terms / phrases that you keep coming across: - ``d.js``, ``i.js``, ``v.js``, ``news.js`` are the endpoints of the DDG's web API through which additional content for a query can be requested (vqd required) The ``*.js`` endpoints return a JSON response and can therefore only be executed on a JS-capable client. The service at https://lite.duckduckgo.com/lite offers general WEB searches (no news, videos etc). DDG-lite and DDG-html can be used by clients that do not support JS, aka *no-JS*. DDG-lite works a bit differently: here, ``d.js`` is not an endpoint but a field (``api=d.js``) in a form that is sent to DDG-lite. - The request argument ``origin=funnel_home_website`` is often seen in the DDG services when the category is changed (e.g., from web search to news, images, or to the video category) .. _DuckDuckGo Bot Detection Research & Solution: https://github.com/ggfevans/searxng/blob/mod-sidecar-harvester/docs/ddg-bot-detection-research.md .. _Sec-Fetch-Mode: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Sec-Fetch-Mode> .. _Referrer-Policy: https://developer.mozilla.org/docs/Web/HTTP/Reference/Headers/Referrer-Policy#directives .. _Referer: https://developer.mozilla.org/de/docs/Web/HTTP/Reference/Headers/Referer .. _User-Agent: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent .. _Accept-Language: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language N)locales) EngineCache) EngineTraits)SearxEngineCaptchaException)EXTERNAL_BANGSget_node) EngineResults) ElementType eval_xpatheval_xpath_getindexextr extract_text gen_useragent) SXNG_Response) OnlineParamsz!https://lite.duckduckgo.com/lite/Q12805FHTML)website wikidata_iduse_official_apirequire_api_keyresultsaboutgeneralweb categoriesTpagingtime_range_support safesearchz!https://html.duckduckgo.com/html/ddg_urldwmy)dayweekmonthyeartime_range_dict_CACHE_HTTP_User_Agentreturnc0t tdatS)N duckduckgo)r*r)/root/searxng/searx/engines/duckduckgo.py get_cacher2s ~\* Mr0queryvalueparamsrcxt}|j|d|dd}|j||dy)N//headers User-Agenti)keyr4expire)r2 secret_hashset)r3r4r5cacher:s r1set_vqdr?sA KE   ugRy(9,(G'HI JC II#U4I0r0ct}|j|d|dd}|j|xsd}|rtj d||S)zReturns the ``vqd`` value that fits to the *query* (and HTTP User-Agent_ header). :param query: the query term :param params: request parameters r7r8r9)r:z get_vqd: re-use cached value: %s)r2r<getloggerdebug)r3r5r>r:r4s r1get_vqdrEs] KE   ugRy(9,(G'HI JCs#)rE  7? Lr0 eng_traits sxng_localedefaultcl|j||}|jdj||xsdS)avGet DuckDuckGo's language identifier from SearXNG's locale. .. hint:: `DDG-lite `__ and the *no Javascript* page https://html.duckduckgo.com/html do not offer a language selection to the user. DDG defines its languages by a region code (:py:obj:`fetch_traits`). To get region and language of a DDG service use: .. code: python eng_region = traits.get_region(params["searxng_locale"], traits.all_locale) eng_lang = get_ddg_lang(traits, params["searxng_locale"]) It might confuse, but the ``l`` value of the cookie is what SearXNG calls the *region*: .. code:: python # !ddi paris :es-AR --> {'ad': 'es_AR', 'ah': 'ar-es', 'l': 'ar-es'} params['cookies']['ad'] = eng_lang params['cookies']['ah'] = eng_region params['cookies']['l'] = eng_region lang_regionN) get_languagecustomrB)rFrGrHlangs r1 get_ddg_langrNs:@"..{GDD   ] + / / T B JdJr0zh_TWzh_HKskipca_ESid_IDnb_NOja_JPko_KRar_SAsl_SIth_THvi_VN) ztw-tzhzhk-tzhzct-cazes-cazid-enzno-nozjp-jpzkr-krzxa-arzsl-slzth-enzvn-en ddg_reg_mapar_DZrJar_JObn_INde_CHen_AUen_CAen_GBeo_XXeoes_ARes_CLes_COes_CRes_ECes_MXes_PEes_UY) es_VEfr_CAfr_CHfr_BEnl_BEpt_BRod_INio_XX tokipona_XX ddg_lang_mapcg}tjd|D]N}|js|jdrt t |ddrd|d}|j |Pdj|S)zMTo avoid a redirect, the !bang directives in the query string are quoted.z(\s+)!N' )resplitstrip startswithrrappendjoin)r3_qvals r1quote_ddg_bangsrYsrBxx%(yy{  >># 8NCG#DcU!*C #  88B<r0ct|dk\rd|d<yt|}tj|dtj}|d}t |d<d|d<d|d <d |d <d |d <d|d<|d}|j ds|d|d|jd|d<|d}||d<t|d<d|d<|ddk(rd|d<nxt||}|r||d<ntdd|dd |dd!"|djd#rd|d<yd|d$<d%|d&<d'|d(<d)|d*<d+|dd,z d-zz}|dz|d.<||d/<|d0k(rd0|d1<n ||d1<||d2d1<tj t|d3d}|r ||d4<||d2d4<d5|dd6<t|dd<tjd7|dtjd8|dtjd9|d2y):Niurlsearxng_localer8r9documentzSec-Fetch-DestnavigatezSec-Fetch-Modez same-originzSec-Fetch-Sitez?1zSec-Fetch-Userzhttps://html.duckduckgo.com/RefererzAccept-Language,-z;q=0.7dataqPOSTmethodpagenoryrAb)r3r5vqdrzVQD missed (page: z , locale: )suspended_timemessagezh nextParamszd.jsapijsonolv dcswt-wtklcookies time_rangedfz!application/x-www-form-urlencodedz Content-Typezparam headers: %szparam data: %szparam cookies: %s)lenrtraits get_region all_localer+rBupperr rErrr)strrCrD) r3r5 eng_regionr8ui_langrroffsett_ranges r1requestris 5zSu  E "E'' JYG-GL *G  *G  -G  $G 7GI%&G ;;( )(/y'!GMMO;LF%S!" &>DDIF5MF8 h1S E&1 DK . ,VH-=,>jP`IaHbbcd  " # . .t 4!F5M \U S S vh'!+r11aZT S WT T ",y$"&&s6,+?'@"EGT ")y$(KF9n%#*F9i  LL$fY&78 LL!6&>2 LL$fY&78r0domc,tt|dS)ziIn case of CAPTCHA ddg response its own *not a Robot* dialog and is not redirected to a CAPTCHA page.z//form[@id='challenge-form'])boolr )rs r1is_ddg_captchars  3 >? @@r0resprc t}|jdk(r|Stjj |j }|j }t|r#tdd|djddt|d}t|rF|d}t|d d}t|dd }t|t||j t|d D]p}t|d }t|ddg} |j|j j#t%|xsdt|ddt%| xsdrd} t%t|| j'} | rCd| vr?d| vr;d| vr7|j|j j)| t|dd|S)Ni/rz CAPTCHA (rrrrz//input[@name="vqd"]/..z//input[@name="vqd"]/@valuer)r3r4r5z6//div[@id="links"]/div[contains(@class, "web-result")]z.//h2/az).//a[contains(@class, "result__snippet")]rAz .//h2/a/@href)titlercontentz //div[@id="zero_click_abstract"]zYour IP address iszYour user agent:z URL Decoded:z(//div[@id="zero_click_abstract"]/a/@href)answerr)r status_codelxmlhtml fromstringtext search_paramsrrrBr rrr?r addtypes MainResultrr~Answer) rresdocr5formform_vqdr div_result_title_contentzero_click_info_xpath zero_clicks r1responsers /C 3 ))  tyy )C   Fc)iPVW]P^PbPbcgPhOiijDkll c4 5D 4yAwd$AB1EVF^C()h-%% !&^_  J 2&z3^`acef  II "6*0bz?;A>$X.4" !    ?j.CDEKKMJJ. j 0 * ,  II  !'-WYZ[   Jr0 engine_traitsc ddlm}ddlm}d|_|dd}|j s t dt|jd d }tj|}|jD]\}}|dk(rd|_tj|}|d k(r.|s*|jd \} } | d z| jz} tj t"j$j'|} |j,j| } | r| |k7rt+d| d| d|||j,| <i|j.d<t|jdd}||} | jD]\} }| dk(r t0j| | }|d k(r( |dk(rEtj t"j$j'| } | |j.d| <stj2t"j$j'|} |j4j| } | r| | k7rt+d| d| d| | |j4| <y#t"j($rt+d|d|d|dY,wxYw#t"j($rt+d|d| dYEwxYw)aFetch languages & regions from DuckDuckGo. SearXNG's ``all`` locale maps DuckDuckGo's "All regions" (``wt-wt``). DuckDuckGo's language "Browsers preferred language" (``wt_WT``) makes no sense in a SearXNG request since SearXNG's ``all`` will not add a ``Accept-Language`` HTTP header. The value in ``engine_traits.all_locale`` is ``wt-wt`` (the region). Beside regions DuckDuckGo also defines its languages by region codes. By example these are the english languages in DuckDuckGo: - en_US - en_AU - en_CA - en_GB The function :py:obj:`get_ddg_lang` evaluates DuckDuckGo's language from SearXNG's locale. r)rB)js_obj_str_to_pythonrz:https://duckduckgo.com/dist/util/u.7669f071a13a7daa57cb.js)timeoutz+Response from DuckDuckGo regions is not OK.zregions:z,snippetLengthsrQr_zERROR: z (z) -> z is unknown by babelzCONFLICT: babel z --> z, rJz languages:z,regionswt_WTzERROR: language z) is unknown by babelN) searx.networkrB searx.utilsrrok RuntimeErrorr rrloadsitemsr[r}rr region_tagbabelLocaleparseUnknownLocaleErrorprintregionsrLrv language_tag languages)rrBrrjs_codereng_tagnameregion eng_territoryeng_langsxng_tagconflictr babel_tags r1 fetch_traitsr s."0 'M KUV WD 77HII499j*;,>y,IJH !**..x8 8#8XVW ,4 )35%''  gvV W  D''  4R S  s+ 2I2-A J!72J!2(JJ!%K  K )en_US);__doc__rr|typingtr lxml.htmlrsearxrsearx.enginelibrsearx.enginelib.traitsrsearx.exceptionsrsearx.external_bangrrsearx.result_typesr rr r r r rr TYPE_CHECKINGsearx.extended_typesrsearx.search.processorsrrdictrr__annotations__rlistrrrr r)r*r+r2intr?rErNr[rvrrrrrr/r0r1rs dN  '/88,??243  tCtO#E* DI*D D?22+.sSRU"Vc3hV  &#';1391S1.1T1    *"K"K"K"K 4Z "KL            T#s(^ %  ]%  ]%  ] %   ] %  ]%  ]%  ]%  ]%  T%   ]!% " ]#% $ ]%% & ]'% ( ])% * ]+% , ]-% . ]/% 0       I%  d38n%P 3 3  ^93^9^94^9BA A5?5}5p`5 `5r0