iUdZddlmZddlZddlmZddlZddlZddlZ ddl m Z m Z ddl m Z ddl mZmZddlmZmZmZmZmZmZmZmZmZmZe j6d Z daej:dzed <d aee jBd z Z" d ej:fdZ#ded ejHdzfdZ%dZ&dZ'dejPfdZ)y)ah Bot protection / IP rate limitation. The intention of rate limitation is to limit suspicious requests from an IP. The motivation behind this is the fact that SearXNG passes through requests from bots and is thus classified as a bot itself. As a result, the SearXNG engine then receives a CAPTCHA or is blocked by the search engine (the origin) in some other way. To avoid blocking, the requests from bots to SearXNG must also be blocked, this is the task of the limiter. To perform this task, the limiter uses the methods from the :ref:`botdetection`: - Analysis of the HTTP header in the request / :ref:`botdetection probe headers` can be easily bypassed. - Block and pass lists in which IPs are listed / :ref:`botdetection ip_lists` are hard to maintain, since the IPs of bots are not all known and change over the time. - Detection & dynamically :ref:`botdetection rate limit` of bots based on the behavior of the requests. For dynamically changeable IP lists a Valkey database is needed. The prerequisite for IP based methods is the correct determination of the IP of the client. The IP of the client is determined via the X-Forwarded-For_ HTTP header. .. attention:: A correct setup of the HTTP request headers ``X-Forwarded-For`` and ``X-Real-IP`` is essential to be able to assign a request to an IP correctly: - `NGINX RequestHeader`_ - `Apache RequestHeader`_ .. _X-Forwarded-For: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For .. _NGINX RequestHeader: https://docs.searxng.org/admin/installation-nginx.html#nginx-s-searxng-site .. _Apache RequestHeader: https://docs.searxng.org/admin/installation-apache.html#apache-s-searxng-site Enable Limiter ============== To enable the limiter activate: .. code:: yaml server: ... limiter: true # rate limit the number of request on the instance, block some bots and set the valkey-url connection. Check the value, it depends on your valkey DB (see :ref:`settings valkey`), by example: .. code:: yaml valkey: url: valkey://localhost:6379/0 Configure Limiter ================= The methods of :ref:`botdetection` the limiter uses are configured in a local file ``/etc/searxng/limiter.toml``. The defaults are shown in limiter.toml_ / Don't copy all values to your local configuration, just enable what you need by overwriting the defaults. For instance to activate the ``link_token`` method in the :ref:`botdetection.ip_limit` you only need to set this option to ``true``: .. code:: toml [botdetection.ip_limit] link_token = true .. _limiter.toml: ``limiter.toml`` ================ In this file the limiter finds the configuration of the :ref:`botdetection`: - :ref:`botdetection ip_lists` - :ref:`botdetection rate limit` - :ref:`botdetection probe headers` .. kernel-include:: $SOURCEDIR/limiter.toml :code: toml Implementation ============== ) ip_addressN)Path)loggervalkeydb) botdetection) SXNG_Request sxng_request) config http_accepthttp_accept_encodinghttp_accept_languagehttp_user_agenthttp_sec_fetchip_limitip_lists get_network dump_requestlimiterCFGF limiter.tomlreturnc(tddlm}|jxs t ddz }t j jt|tjjatjjt|tS)z/Returns SearXNG's global limiter configuration.)settings_loaderz /etc/searxngr) rrget_user_cfg_folderrr Config from_tomlLIMITER_CFG_SCHEMAsearxcompatLIMITER_CFG_DEPRECATEDlimiter_fix_cfg)rcfg_files /root/searxng/searx/limiter.pyget_cfgr&sf {%#779QT.=QUccmm%%&8(ELLDgDgh $$S(3 Jrequestct}t|j}t||}|jdk(ry|j ryt j||\}}|r"tjd|j|yt j||\}}|r;tjd|j|tjd|zdfStfD]O}|j!|||}|tj"d|j$d|dt't(|cS|jd k(rrt*t,t.tt0t2fD]O}|j!|||}|tj"d|j$d|dt't(|cStj"d |dt't(y) Nz/healthzzPASS %s: matched PASSLIST - %sz BLOCK %s: matched BLOCKLIST - %szIP is on BLOCKLIST - %sizNOT OK (z): z: %sz/searchzOK )r&r remote_addrrpath is_link_localrpass_iprwarning compressedblock_iperrorflask make_responserfilter_requestdebug__name__rr r r r rr)r(cfgreal_ipnetworkmatchmsgfuncvals r%r4r4s )C,,-G'3'G||z!!!'3/JE3 79K9KSQ""7C0JE3  79K9KSQ""$=$CS#IJJ !!'7C8 ? LL8DMM?#gYdC\R^E_ `J ||y       D%%gwr^s[z!  ;     # V]]T  (^**^;6  >L>X->->-E>B(  $EKK$r'