iAUdZgdZddlZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlmZddlmZddlmZej*dZeeej2edzfZej8ed <Gd d ej<Zej@Gd d Z!GddejDZ#GddejHe#Z%y)z}Implementation of caching solutions. - :py:obj:`searx.cache.ExpireCache` and its :py:obj:`searx.cache.ExpireCacheCfg` ---- )ExpireCacheCfgExpireCacheStats ExpireCacheExpireCacheSQLiteN)Iterator)sqlitedb)logger) get_settingcache CacheRowTypeceZdZUdZeed< dZeed< dZeed< dZ eed< d Z eed < d Z e jd ed < edjZeed< dZy)rz/Configuration of a :py:obj:`ExpireCache` cache.namedb_urli( MAX_VALUE_LENi: MAXHOLD_TIMEiMAINTENANCE_PERIODauto)roffMAINTENANCE_MODEzserver.secret_keypasswordc|jsOtjtjzdt j |jdz|_yy)N sxng_cache_z.db)rtempfile gettempdirosseprnormalize_namerselfs /root/searxng/searx/cache.py __post_init__zExpireCacheCfg.__post_init__LsI{{"--/"&&8[IcIcdhdmdmInHoor;ssDKN)__name__ __module__ __qualname____doc__str__annotations__rrintrrrtypingLiteralr encoderbytesr"r#r!rr#s9 IFC>#M3"-(L#(U%%7=fnn]3< ""56==?He?tr#rc4eZdZUdZeeeefed< dZ y)rz@Dataclass which provides information on the status of the cache. cached_itemscd}d}g}|jjD]\}}|dz }|s|jd|dd$|D]s\}}}d} |r.tjj |j d} |dz }|jd|dd| d |d d t |jd |d u|jd ||jd|dj|S)Nr[20sz] emptyrz%Y-%m-%d %H:%M:%Sz]  12z --> () zNumber of contexts: znumber of key/value pairs:  ) r1itemsappenddatetime fromtimestampstrftimetyper$join) r c_ctxc_kvlinesctx_namekv_listkeyvalueexpire valid_untils r!reportzExpireCacheStats.reportes#!%!2!2!8!8!: s Hg QJE q#g67&- s"UF "*"3"3"A"A&"I"R"RSf"gK  q#b Qs2h&QUV[Q\QeQePffhinhoopqr  s s  +E734 24&9:yyr#N) r$r%r&r'dictr(listr r)rJr/r#r!rrRs$JsD..//  r#rc eZdZUdZeed<dZeed<ejddede jde dzdedzd e f d Zejdded e jdedzd e jfd Zejdd e de d e fdZejd efdZeded dfdZeded efdZde jd efdZded e jfdZdeezd efdZy)rzUAbstract base class for the implementation of a key/value cache with expire date.cfg hash_tokenNrFrGrHctxreturncy)aJSet *key* to *value*. To set a timeout on key use argument ``expire`` (in sec.). If expire is unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`. After the timeout has expired, the key will automatically be deleted. The ``ctx`` argument specifies the context of the ``key``. A key is only unique in its context. The concrete implementations of this abstraction determine how the context is mapped in the connected database. In SQL databases, for example, the context is a DB table or in a Key/Value DB it could be a prefix for the key. If the context is not specified (the default is ``None``) then a default context should be used, e.g. a default table for SQL databases or a default prefix in a Key/Value DB. Nr/)r rFrGrHrPs r!setzExpireCache.setr#defaultcy)z@Return *value* of *key*. If key is unset, ``None`` is returned.Nr/)r rFrUrPs r!getzExpireCache.getrTr#forcetruncatecy)a4Performs maintenance on the cache. ``force``: Maintenance should be carried out even if the maintenance interval has not yet been reached. ``truncate``: Truncate the entire cache, which is necessary, for example, if the password has changed. Nr/)r rXrYs r! maintenancezExpireCache.maintenancerTr#cy)zgReturns a :py:obj:`ExpireCacheStats`, which provides information about the status of the cache.Nr/rs r!statezExpireCache.staterTr#rct|S)zFactory to build a caching instance. .. note:: Currently, only the SQLite adapter is available, but other database types could be implemented in the future, e.g. a Valkey (Redis) adapter. )r)rNs r! build_cachezExpireCache.build_caches!%%r#rcdtjztjz}dj|Dcgc] }||vs| c}Scc}w)zReturns a normalized name that can be used as a file name or as a SQL table name (is used, for example, to normalize the context name).z-_.r)string ascii_lettersdigitsr@)r_validcs r!rzExpireCache.normalize_names@ --- =ww47a1;7887s A A c0tj|}|SN)pickledumps)r rGdumps r! serializezExpireCache.serializesll5) r#c0tj|}|Srg)rhloads)r rGobjs r! deserializezExpireCache.deserializesll5! r#ct|tr t|d}tj||j j zd}|jS)a Creates a hash of the argument ``name``. The hash value is formed from the ``name`` combined with the :py:obj:`password `. Can be used, for example, to make the ``key`` stored in the DB unreadable for third parties.zutf-8)encodingsha256) digestmod) isinstancer(r.hmacnewrNr hexdigest)r rms r! secret_hashzExpireCache.secret_hashsE dC 0D HHTDHH--- B{{}r#rgNNFF)r$r%r&r'rr)rOr(abcabstractmethodr+Anyr*boolrSrWr[rr] staticmethodr_rr.rkroryr/r#r!rr|s "J" s 6:: sTz d  ^b  & OsOVZZOS4ZOSYS]S]OO       $     *'** & &,? & &9S9S99vzze6:: e  r#rc eZdZUdZdZeed<iZee e fed<dZ e ed<de ffd Z d e jd effd Zd"d ed ed efdZde d efdZed ee fdZdee fdZed efdZd#de dej2dedzde dzd ef dZ d#deede dzd efdZ d#deede dzd eeee ffdZd$de dej2de dzd ej2fdZde d e ee ej2ffd Z!d e"fd!Z#xZ$S)%raCache that manages key/value pairs in a SQLite DB. The DB model in the SQLite DB is implemented in abstract class :py:obj:`SQLiteAppl `. The following configurations are required / supported: - :py:obj:`ExpireCacheCfg.db_url` - :py:obj:`ExpireCacheCfg.MAXHOLD_TIME` - :py:obj:`ExpireCacheCfg.MAINTENANCE_PERIOD` - :py:obj:`ExpireCacheCfg.MAINTENANCE_MODE` r3 DB_SCHEMADDL_CREATE_TABLESz CACHE-TABLECACHE_TABLE_PREFIXrNc||_|jdk(rtjdt||jy)zdAn instance of the SQLite expire cache is build up from a :py:obj:`config `.z:memory:z/don't use SQLite DB in :memory: in production!!N)rNrlogcriticalsuper__init__)r rN __class__s r!rzExpireCacheSQLite.__init__s6$' :: # LLJ K $r#connrQct||}|sytj|jj j }|j|j}||k7re|*tjd|jj|jdd|jj|j|y)NFz2[%s] hash token changed: truncate all cache tablesT)rXrY)rinithashlibrrrNrrw propertiesrOrwarningrr[rS)r rret_valrvoldrs r!rzExpireCacheSQLite.inits',t$nnTXX../99;oodoo. #: PRVRZRZR_R_`   4$  7 OO   5r#rXrYc,|s+ttj|jkry|jj dd|r|j |j yttj}|j5}|j D]:}|jd|d|f}tjd|j|< dddjd|jy#1swY+xYw) NFLAST_MAINTENANCErT DELETE FROM z WHERE expire < ?z3deleted %s keys from table %s (expire date reached)zPRAGMA wal_checkpoint(TRUNCATE)) r*timenext_maintenance_timerrStruncate_tables table_namesconnectexecuterdebugrowcountclose)r rXrYrHrtableress r!r[zExpireCacheSQLite.maintenancesTYY[)D,F,FF .3   !1!1 2TYY[! \\^ ft)) fll\%8I#JVIV OQTQ]Q]_de f f 67  f fs A D  Drc||jvrytjd|djd|dddd|jj d d g}d |d |d }|j 5}|j||j|dddj|jj|jd||y#1swYExYw)z|Create DB ``table`` if it has not yet been created, no recreates are initiated if the table already exists. Fz;key/value table '%s' NOT exists in DB -> create DB table ..r9zCREATE TABLE IF NOT EXISTS z (z key TEXT,z value BLOB,z6 expire INTEGER DEFAULT (strftime('%s', 'now') + z),zPRIMARY KEY (key))z(CREATE INDEX IF NOT EXISTS index_expire_z ON z (expire);N-T) rrinfor@rNrrrrrrSr)r r sql_table sql_indexrs r! create_tablezExpireCacheSQLite.create_table s D$$ $ NPUVII-eWB7$$HI^I^H__ab$   ?ugT%PYZ \\^ $t LL # LL # $  t667q@%H  $ $s 1#CC!cd|jd}|jj|jxsg}|Dcgc]}|d c}Scc}w)z3List of key/value tables already created in the DB.z.SELECT value FROM properties WHERE name LIKE 'z%%'r)rDBrfetchall)r sqlrowsrs r!rzExpireCacheSQLite.table_names;sR?t?V?V>WWZ[wws#,,.4""#!###s Arctjddj||j5}|D]}|j d| dddj y#1swYxYw)Nztruncate table: %s,rT)rrr@rrr)r rrrs r!rz!ExpireCacheSQLite.truncate_tablesBsj &(=> \\^ 5t$ 5 |E734 5 5   5 5s A,,A5c|jj|jjdt t j zS)z2Returns (unix epoch) time of the next maintenance.r)rNrrm_timer*rrs r!rz'ExpireCacheSQLite.next_maintenance_timeJs;xx**T__-C-CDVX[\`\e\e\gXh-iiir#NrFrGrHrPc 8|j|||fg|\}}|rAtjd|jj||t |t|S|D].}tj d|jj||0t|S)aSet key/value in DB table given by argument ``ctx``. If expire is unset the default is taken from :py:obj:`ExpireCacheCfg.MAXHOLD_TIME`. If ``ctx`` argument is ``None`` (the default), a table name is generated from the :py:obj:`ExpireCacheCfg.name`. If DB table does not exists, it will be created (on demand) by :py:obj:`self.create_table `. )rPz2%s -- %s: key '%s' updated or inserted (%s errors) %s -- %s: %s)_setmanyrrrNrlenerrorr)r rFrGrHrPre err_msg_listmsgs r!rSzExpireCacheSQLite.setRs--#uf)=(>C-H< IIJDHHMM[^`cehiuev wAw$ C .$((--cB CAwr#opt_listc ptj}|j||\}}tj}|D].}tjd|jj ||0tj d|jj ||t|||z t||S)zEfficient bootload of the cache from a list of options. The list contains tuples with the arguments described in :py:obj:`ExpireCacheSQLite.set`.)rrPrzI%s -- %s: %s/%s key/value pairs updated or inserted in %s sec (%s errors))rrrrrNrrr)r rrP_startrer_endrs r!setmanyzExpireCacheSQLite.setmanybs--s-C<yy{ ?C IIndhhmmS# > ? W HHMM  M 6M   r#c H|}|j|}|s%|j|jj}|j |d|d}g}g}|D]\}} } |j | } t | |jjkDr$|j|d|dt | d_| s|jj} ttj| z} || | | | f} |j| |sd|fS|r2|j5|jj||dddn;|j5} | j||ddd jt ||fS#1swYxYw#1swY2xYw)Nz INSERT INTO zU (key, value, expire) VALUES (?, ?, ?) ON CONFLICT DO UPDATE SET value=?, expire=?)rGz.key='z,' - serialized value too big to cache (len: r8r)r[rrNrrrkrrr;rr*rr executemanyrr)r rrPr table_namesql_strsql_rowsrrF_valrHrGsql_argsrs r!rzExpireCacheSQLite._setmany{s   ,,TXX]];J *%:,'+ ,   #% !) & Cv>>>5E5zDHH222##ugVC58dehineodppr$st..%.FUFE6:H OOH % &l? "  7##GX6 7 7 44  (3 4 JJL8}l** 7 7 4 4sF F FF!rUc&|}|j|s%|j|jj}||jvr|Sd|d}|j j ||fj}||S|j|dS)aGet value of ``key`` from table given by argument ``ctx``. If ``ctx`` argument is ``None`` (the default), a table name is generated from the :py:obj:`ExpireCacheCfg.name`. If ``key`` not exists (in table), the ``default`` value is returned. zSELECT value FROM z WHERE key = ?r) r[rrNrrrrfetchonero)r rFrUrPrrrows r!rWzExpireCacheSQLite.gets '' 6E (( (N"5'8ggoocC6*335 ;NA''r#c#K|}|j|s%|j|jj}||jvr?|j j d|D]}|d|j|dfyyw)zIterate over key/value pairs from table given by argument ``ctx``. If ``ctx`` argument is ``None`` (the default), a table name is generated from the :py:obj:`ExpireCacheCfg.name`.zSELECT key, value FROM rr3N)r[rrNrrrrro)r rPrrs r!pairszExpireCacheSQLite.pairss '' 6E D$$ $ww)@'HI 7!fd..s1v666 7 %sBB c i}|jD]Y}g||<|jjd|D]1}||j|d|j |d|df3[t |S)NzSELECT key, value, expire FROM rr3)r1)rrrr;ror)r r1rrs r!r]zExpireCacheSQLite.states68 %% WE"$L ww)H'PQ WU#**CFD4D4DSV4LcRSf+UV W W \::r#r{rgrz)%r$r%r&r'rr*r)rrKr(rrrsqlite3 Connectionrrr[rpropertyrLrrrr+r~rSr rtuplerrWrrrr] __classcell__)rs@r!rrs Is)+tCH~*++%N% ++  $<#$6$T#Y$$ 49jsjjs6::sTzd ^b&|$4Z 89+|$9+4Z9+ sDI~  9+v(s(VZZ(S4Z(SYS]S](. 7 7%VZZ*@!A 7;';r#r)&r'__all__r|collections.abcr dataclassesr<rrurrhrrarrr+msgspecsearxrr r getChildrrr(r~r*r TypeAliasr)Structr dataclassrABCr SQLiteApplrr/r#r!rs U $   foog!&sFJJd 'B!C fC,tW^^,t^ & & & RW#''WtK;++[K;r#