178 lines
25 KiB
Plaintext
178 lines
25 KiB
Plaintext
|
Ë
|
|||
|
|
JÀjÖ] ã óz — d Z ddlmZmZmZmZ ddlZddlmZ g d¢Z G d„ d« Z
|
|||
|
|
G d „ d
|
|||
|
|
« Z G d „ d e « Z y)
|
|||
|
|
zlModule containing a memory memory manager which provides a sliding window on a number of memory mapped filesé )Ú MapWindowÚ MapRegionÚ
|
|||
|
|
MapRegionListÚ is_64_bité N)Úreduce)ÚStaticWindowMapManagerÚSlidingWindowMapManagerÚ WindowCursorc ó¢ — e Zd ZdZdZdd„Zd„ Zd„ Zd„ Zd„ Z d „ Z
|
|||
|
|
d
|
|||
|
|
„ Z d „ Z d d „Z
|
|||
|
|
d
|
|||
|
|
„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zd„ Zy) r aã
|
|||
|
|
Pointer into the mapped region of the memory manager, keeping the map
|
|||
|
|
alive until it is destroyed and no other client uses it.
|
|||
|
|
|
|||
|
|
Cursors should not be created manually, but are instead returned by the SlidingWindowMapManager
|
|||
|
|
|
|||
|
|
**Note:**: The current implementation is suited for static and sliding window managers, but it also means
|
|||
|
|
that it must be suited for the somewhat quite different sliding manager. It could be improved, but
|
|||
|
|
I see no real need to do so.©Ú_managerÚ_rlistÚ_regionÚ_ofsÚ_sizeNc óJ — || _ || _ d | _ d| _ d| _ y )Nr r
|
|||
|
|
)ÚselfÚmanagerÚregionss úM/home/antigravity/intellecton/venv/lib/python3.12/site-packages/smmap/mman.pyÚ__init__zWindowCursor.__init__% s% € Ø ˆŒ
|
|||
|
|
Ø ˆŒ ؈Œ ؈Œ ؈�
|
|||
|
|
ó c ó$ — | j « y ©N©Ú_destroy©r s r Ú__del__zWindowCursor.__del__, ó € Ø �
|
|||
|
|
‰
|
|||
|
|
�r c ó — | S r © r s r Ú __enter__zWindowCursor.__enter__/ s € ؈ r c ó$ — | j « y r r )r Úexc_typeÚ exc_valueÚ tracebacks r Ú__exit__zWindowCursor.__exit__2 r r c ó — | j « | j �X t | j « dk( r>| j j j | j j
|
|||
|
|
« « yyy# t t f$ r Y yw xY w)z&Destruction code to decrement countersNr ) Ú unuse_regionr Úlenr Ú_fdictÚpopÚ
|
|||
|
|
path_or_fdÚ TypeErrorÚKeyErrorr s r r zWindowCursor._destroy5 sx € à ×ÑÔà �;‰;Ð "ð
|
|||
|
|
|
|||
|
|
Ü�t—{‘{Ó# qÒ(à—M‘M×(Ñ(×,Ñ,¨T¯[©[×-CÑ-CÓ-EÕFð )ð #øô œxÐ(ò
|
|||
|
|
ñ
|
|||
|
|
ð
|
|||
|
|
ús žAA6 Á6BÂBc óp — |j | _ t |j « |j « | _ |j | _ |j | _ |j
|
|||
|
|
| _ | j D ] }|j
|
|||
|
|
« Œ | j �| j j
|
|||
|
|
« yy)z>Copy all data from rhs into this instance, handles usage countN)r Útyper r r r Úincrement_client_count)r ÚrhsÚregions r Ú
|
|||
|
|
_copy_fromzWindowCursor._copy_fromJ s‰ € àŸ ™ ˆŒ
|
|||
|
|
Ø&”d˜3Ÿ:™:Ó& s§z¡zÓ2ˆŒ Ø—{‘{ˆŒ Ø—H‘HˆŒ Ø—Y‘YˆŒ
|
|||
|
|
à—k‘kò ,ˆFØ × )Ñ )Õ +ð ,ð �<‰<Ð #Ø �L‰L× /Ñ /Õ 1ð $r c óH — t | « « }|j | « |S )zcopy module interface)r2 r6 )r Úcpys r Ú__copy__zWindowCursor.__copy__Y s € àŒd�4‹j‹lˆØ �‰�tÔ Øˆ
|
|||
|
|
r c óF — | j « | j |« y)z™Assign rhs to this instance. This is required in order to get a real copy.
|
|||
|
|
Alternatively, you can copy an existing instance using the copy moduleN)r r6 )r r4 s r ÚassignzWindowCursor.assign` s € ð
|
|||
|
|
�
|
|||
|
|
‰
|
|||
|
|
ŒØ �‰˜Õ r c ó0 — d}| j }| j j « }t |xs ||j « xs |« }| j
|
|||
|
|
�.| j
|
|||
|
|
j
|
|||
|
|
|« rd}n| j « ||k\ r| S |r>|j | j |||d« | _ | j
|
|||
|
|
j « || j
|
|||
|
|
j z
|
|||
|
|
| _ t || j
|
|||
|
|
j « |z
|
|||
|
|
« | _
|
|||
|
|
| S )a$ Assure we point to a window which allows access to the given offset into the file
|
|||
|
|
|
|||
|
|
:param offset: absolute offset in bytes into the file
|
|||
|
|
:param size: amount of bytes to map. If 0, all available bytes will be mapped
|
|||
|
|
:param flags: additional flags to be given to os.open in case a file handle is initially opened
|
|||
|
|
for mapping. Has no effect if a region can actually be reused.
|
|||
|
|
:return: this instance - it should be queried for whether it points to a valid memory region.
|
|||
|
|
This is not the case if the mapping failed because we reached the end of the file
|
|||
|
|
|
|||
|
|
**Note:**: The size actually mapped may be smaller than the given size. If that is the case,
|
|||
|
|
either the file has reached its end, or the map was created between two existing regionsTF)r r Ú file_sizeÚminÚ window_sizer Ú includes_ofsr* Ú_obtain_regionr3 Ú_br Úofs_endr )r ÚoffsetÚsizeÚflagsÚ need_regionÚmanÚfsizes r Ú
|
|||
|
|
use_regionzWindowCursor.use_regionf sê € ð ˆ Ø�m‰mˆØ— ‘ ×%Ñ%Ó'ˆÜ�4’=˜5 #§/¡/Ó"3Ò"<°uÓ=ˆà �<‰<Ð #Ø�|‰|×(Ñ(¨Ô0Ø #‘ à×!Ñ!Ô#ð
|
|||
|
|
�UŠ?؈Kñ Ø ×-Ñ-¨d¯k©k¸6À4ÈÐPUÓVˆDŒLØ �L‰L× /Ñ /Ô 1ð ˜TŸ\™\Ÿ_™_Ñ,ˆŒ ܘ˜tŸ|™|× 3Ñ 3Ó 5¸Ñ >Ó?ˆŒ
|
|||
|
|
àˆ r c ó` — | j �| j j d« d| _ y)a/ Unuse the current region. Does nothing if we have no current region
|
|||
|
|
|
|||
|
|
**Note:** the cursor unuses the region automatically upon destruction. It is recommended
|
|||
|
|
to un-use the region once you are done reading from it in persistent cursors as it
|
|||
|
|
helps to free up resource more quicklyNéÿÿÿÿ)r r3 r s r r* zWindowCursor.unuse_regionŽ s' € ð �<‰<Ð #Ø �L‰L× /Ñ /°Ô 3؈� r c ó� — t | j j « « | j | j | j z S )aÊ Return a buffer object which allows access to our memory region from our offset
|
|||
|
|
to the window size. Please note that it might be smaller than you requested when calling use_region()
|
|||
|
|
|
|||
|
|
**Note:** You can only obtain a buffer if this instance is_valid() !
|
|||
|
|
|
|||
|
|
**Note:** buffers should not be cached passed the duration of your access as it will
|
|||
|
|
prevent resources from being freed even though they might not be accounted for anymore !)Ú
|
|||
|
|
memoryviewr Úbufferr r r s r rO zWindowCursor.bufferš s4 € ô ˜$Ÿ,™,×-Ñ-Ó/Ó0°·±¸4¿9¹9ÀTÇZÁZÑ;OÐPÐPr c ó6 — | j j « S )a
|
|||
|
|
:return: the underlying raw memory map. Please not that the offset and size is likely to be different
|
|||
|
|
to what you set as offset and size. Use it only if you are sure about the region it maps, which is the whole
|
|||
|
|
file in case of StaticWindowMapManager)r Úmapr s r rQ zWindowCursor.map¤ s € ð
|
|||
|
|
�|‰|× Ñ Ó!Ð!r c ó — | j duS )z2:return: True if we have a valid and usable regionN©r r s r Úis_validzWindowCursor.is_valid« s € à�|‰| 4Ð'Ð'r c ó — | j duS )z?:return: True if we are associated with a specific file alreadyN)r r s r Ú
|
|||
|
|
is_associatedzWindowCursor.is_associated¯ s € à�{‰{ $Ð&Ð&r c óH — | j j | j z S )zh:return: offset to the first byte pointed to by our cursor
|
|||
|
|
|
|||
|
|
**Note:** only if is_valid() is True)r rB r r s r Ú ofs_beginzWindowCursor.ofs_begin³ s € ð �|‰|�‰ §¡Ñ*Ð*r c ób — | j j | j z | j z S )z3:return: offset to one past the last available byte©r rB r r r s r rC zWindowCursor.ofs_end¹ s$ € ð �|‰|�‰ §¡Ñ*¨T¯Z©ZÑ7Ð7r c ó — | j S )z$:return: amount of bytes we point to)r r s r rE zWindowCursor.size¾ s € à�z‰zÐr c ó — | j S )z›:return: our mapped region, or None if nothing is mapped yet
|
|||
|
|
:raise AssertionError: if we have no current region. This is only useful for debuggingrS r s r r5 zWindowCursor.region s € ð �|‰|Ðr c óÀ — | j j | j z |cxk xr3 | j j | j z | j z k S c S )zš:return: True if the given absolute offset is contained in the cursors
|
|||
|
|
current region
|
|||
|
|
|
|||
|
|
**Note:** cursor must be valid for this to workrZ )r Úofss r r@ zWindowCursor.includes_ofsÇ sH € ð — ‘ —‘ $§)¡)Ñ+°Ö`°t·|±|·±ÈÏÉÑ7RÐUY×U_ÑU_Ñ7_Ñ`Ð`Ñ`Ð`r c ó6 — | j j « S )z$:return: size of the underlying file)r r= r s r r= zWindowCursor.file_sizeÏ s € à�{‰{×$Ñ$Ó&Ð&r c ó6 — | j j « S )z>:return: path or file descriptor of the underlying mapped file)r r. r s r r. zWindowCursor.path_or_fdÓ s € à�{‰{×%Ñ%Ó'Ð'r c óœ — t | j j « t « r t d« ‚| j j « S )ze:return: path of the underlying mapped file
|
|||
|
|
:raise ValueError: if attached path is not a pathz>Path queried although mapping was applied to a file descriptor)Ú
|
|||
|
|
isinstancer r. ÚintÚ
|
|||
|
|
ValueErrorr s r ÚpathzWindowCursor.path× s: € ô �d—k‘k×,Ñ,Ó.´Ô 4Ü Ð ]Ó^Ð ^à�{‰{×%Ñ%Ó'Ð'r c óœ — t | j j « t « r t d« ‚| j j « S )zÈ:return: file descriptor used to create the underlying mapping.
|
|||
|
|
|
|||
|
|
**Note:** it is not required to be valid anymore
|
|||
|
|
:raise ValueError: if the mapping was not created by a file descriptorz@File descriptor queried although mapping was generated from path)rb r r. Ústrrd r s r ÚfdzWindowCursor.fdß s: € ô
|
|||
|
|
�d—k‘k×,Ñ,Ó.´Ô 4Ü Ð _Ó`Ð `à�{‰{×%Ñ%Ó'Ð'r )NN)r r r ) Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__Ú __slots__r r r# r( r r6 r9 r; rJ r* rO rQ rT rV rX rC rE r5 r@ r= r. re rh r" r r r r s„ „ ñ$ð€Ióòòòòò* 2ò ò ó &òP òQò"ò(ò'ò+ò 8ò
|
|||
|
|
ò ò
|
|||
|
|
aò'ò(ò(ó(r r c ó� — e Zd ZdZg d¢ZeZeZe Z
|
|||
|
|
e Z dZ
|
|||
|
|
ddej fd„Zd„ Zd„ Zd„ Zd „ Zd
|
|||
|
|
„ Zd „ Zd „ Zd
|
|||
|
|
„ Zd„ Zd„ Zd„ Zy)r aß Provides a manager which will produce single size cursors that are allowed
|
|||
|
|
to always map the whole file.
|
|||
|
|
|
|||
|
|
Clients must be written to specifically know that they are accessing their data
|
|||
|
|
through a StaticWindowMapManager, as they otherwise have to deal with their window size.
|
|||
|
|
|
|||
|
|
These clients would have to use a SlidingWindowMapBuffer to hide this fact.
|
|||
|
|
|
|||
|
|
This type will always use a maximum window size, and optimize certain methods to
|
|||
|
|
accommodate this fact)r, Ú _window_sizeÚ_max_memory_sizeÚ_max_handle_countÚ _memory_sizeÚ
|
|||
|
|
_handle_counti r c ó — t « | _ || _ || _ || _ d| _ d| _ |dk r"d}t « rd}|| j z | _ |dk( r#d}t « rd}|| j z | _ yy)a£ initialize the manager with the given parameters.
|
|||
|
|
:param window_size: if -1, a default window size will be chosen depending on
|
|||
|
|
the operating system's architecture. It will internally be quantified to a multiple of the page size
|
|||
|
|
If 0, the window may have any size, which basically results in mapping the whole file at one
|
|||
|
|
:param max_memory_size: maximum amount of memory we may map at once before releasing mapped regions.
|
|||
|
|
If 0, a viable default will be set depending on the system's architecture.
|
|||
|
|
It is a soft limit that is tried to be kept, but nothing bad happens if we have to over-allocate
|
|||
|
|
:param max_open_handles: if not maxint, limit the amount of open file handles to the given number.
|
|||
|
|
Otherwise the amount is only limited by the system itself. If a system or soft limit is hit,
|
|||
|
|
the manager will free as many handles as possibler é@ i i N) Údictr, ro rp rq rr rs r Ú _MB_in_bytes)r r? Úmax_memory_sizeÚmax_open_handlesÚcoeffs r r z StaticWindowMapManager.__init__ s� € ô “fˆŒ Ø 'ˆÔØ /ˆÔ Ø!1ˆÔ Ø ˆÔØ ˆÔà ˜Š?؈EÜŒ{Ø �à %¨×(9Ñ(9Ñ 9ˆDÔ ð ˜aÒ ØˆEÜŒ{Ø �à$)¨D×,=Ñ,=Ñ$=ˆDÕ !ð r c ó — d}|dk( s | j |z | j kD ræd}d}| j j « D ]= }|D ]6 }|j « dk( sŒ|�|j
|
|||
|
|
|j
|
|||
|
|
k sŒ3|}|}Œ8 Œ? |€ |S |dz
|
|||
|
|
}||j
|
|||
|
|
|« = |j d« | xj |j « z c_ | xj dz c_ |dk( rŒÉ| j |z | j kD rŒæ|S )am Unmap the region which was least-recently used and has no client
|
|||
|
|
:param size: size of the region we want to map next (assuming its not already mapped partially or full
|
|||
|
|
if 0, we try to free any available region
|
|||
|
|
:return: Amount of freed regions
|
|||
|
|
|
|||
|
|
.. Note::
|
|||
|
|
We don't raise exceptions anymore, in order to keep the system working, allowing temporary overallocation.
|
|||
|
|
If the system runs out of memory, it will tell.
|
|||
|
|
|
|||
|
|
.. TODO::
|
|||
|
|
implement a case where all unusued regions are discarded efficiently.
|
|||
|
|
Currently its only brute force
|
|||
|
|
r Nr rL )
|
|||
|
|
rr rp r, ÚvaluesÚ client_countÚ_ucÚindexr3 rE rs )r rE Ú num_foundÚ
|
|||
|
|
lru_regionÚlru_listr r5 s r Ú_collect_lru_regionz*StaticWindowMapManager._collect_lru_region/ s € ð ˆ Ø�qŠy˜d× /Ñ /°$Ñ 6¸×9NÑ9NÒ NØ ˆJ؈HØ Ÿ;™;×-Ñ-Ó/ò
|
|||
|
|
+�Ø %ò +�Fà ×+Ñ+Ó-°Ó2Ø 'Ð /°6·:±:À
|
|||
|
|
ÇÁÓ3NØ%+˜
|
|||
|
|
Ø#*™ñ +ð
|
|||
|
|
+ð Ð!Øð Ðð
|
|||
|
|
|
|||
|
|
˜‰NˆIؘŸ ™ ¨
|
|||
|
|
Ó3Ð4Ø × -Ñ -¨bÔ 1Ø × Ò § ¡ Ó!2Ñ 2Õ Ø × Ò !Ñ #Õ ð- �q‹y˜d× /Ñ /°$Ñ 6¸×9NÑ9NÓ Nð0 Ðr c ó — | j |z | j kD r| j |« d}|rt |« dk( sJ ‚|d }nz | j |j « dt j |« }| xj dz
|
|||
|
|
c_
|
|||
|
|
| xj |j « z
|
|||
|
|
c_ |j |« |j |« sJ ‚|S # t $ r, |r‚ | j d« | j ||||d« cY S w xY w)z¸Utility to create a new region - for more information on the parameters,
|
|||
|
|
see MapCursor.use_region.
|
|||
|
|
:param a: A regions (a)rray
|
|||
|
|
:return: The newly created regionNr r T)rr rp rƒ r+ Ú MapRegionClsr. ÚsysÚmaxsizeÚ ExceptionrA rs rE Úappendr@ )r ÚarD rE rF Ú is_recursiveÚrs r rA z%StaticWindowMapManager._obtain_regionX s € ð
|
|||
|
|
× Ñ ˜tÑ # d×&;Ñ&;Ò ;Ø × $Ñ $ TÔ *ð
|
|||
|
|
ˆÙ Ü�q“6˜Q’;Ð �;Ø�!‘‰Að
|
|||
|
|
|
|||
|
|
IØ×%Ñ% a§l¡l£n°a¼¿¹ÀeÓL�ð
|
|||
|
|
× Ò !Ñ #Õ Ø × Ò §¡£Ñ )Õ Ø
|
|||
|
|
�H‰H�QŒKð �~‰~˜fÔ%Ð%Ð%؈øô) ò
|
|||
|
|
Iñ
|
|||
|
|
ð à×(Ñ(¨Ô+Ø×*Ñ*¨1¨f°d¸EÀ4ÓHÒHð
|
|||
|
|
Iús Á 0C Ã2D Ä D c ó — | j j |« }|€ | j |« }|| j |< | j | |« S )a’
|
|||
|
|
:return: a cursor pointing to the given path or file descriptor.
|
|||
|
|
It can be used to map new regions of the file into memory
|
|||
|
|
|
|||
|
|
**Note:** if a file descriptor is given, it is assumed to be open and valid,
|
|||
|
|
but may be closed afterwards. To refer to the same file, you may reuse
|
|||
|
|
your existing file descriptor, but keep in mind that new windows can only
|
|||
|
|
be mapped as long as it stays valid. This is why the using actual file paths
|
|||
|
|
are preferred unless you plan to keep the file descriptor open.
|
|||
|
|
|
|||
|
|
**Note:** file descriptors are problematic as they are not necessarily unique, as two
|
|||
|
|
different files opened and closed in succession might have the same file descriptor id.
|
|||
|
|
|
|||
|
|
**Note:** Using file descriptors directly is faster once new windows are mapped as it
|
|||
|
|
prevents the file to be opened again just for the purpose of mapping it.)r, ÚgetÚMapRegionListClsÚWindowCursorCls)r r. r s r Ú make_cursorz"StaticWindowMapManager.make_cursor� sL € ð —+‘+—/‘/ *Ó-ˆØ ˆ?Ø×+Ñ+¨JÓ7ˆGØ&-ˆD�K‰K˜
|
|||
|
|
Ñ #à×#Ñ# D¨'Ó2Ð2r c ó$ — | j d« S )z]Collect all available free-to-collect mapped regions
|
|||
|
|
:return: Amount of freed handlesr )rƒ r s r Úcollectz StaticWindowMapManager.collect˜ s € ð ×'Ñ'¨Ó*Ð*r c ó — | j S )zO:return: amount of file handles in use. Each mapped region uses one file handle)rs r s r Únum_file_handlesz'StaticWindowMapManager.num_file_handles� s € à×!Ñ!Ð!r c ó\ — t d„ d„ | j j « D « d« S )z$Amount of opened files in the systemc ó — | |z S r r" )ÚxÚys r ú<lambda>z7StaticWindowMapManager.num_open_files.<locals>.<lambda>£ s
|
|||
|
|
€ 1 q¡5€ r c 3 ó> K — | ] }t |« d kD sŒd–— Œ yw)r r N)r+ )Ú.0Úrlists r ú <genexpr>z8StaticWindowMapManager.num_open_files.<locals>.<genexpr>£ s è ø€ Ò*a°ÔRUÐV[ÓR\Ð_`ÓR`¬1Ñ*aùs ‚ – r )r r, r| r s r Únum_open_filesz%StaticWindowMapManager.num_open_files¡ s& € äÑ(Ñ*a¸$¿+¹+×:LÑ:LÓ:NÔ*aÐcdÓeÐer c ó — | j S )z8:return: size of each window when allocating new regions)ro r s r r? z"StaticWindowMapManager.window_size¥ ó € à× Ñ Ð r c ó — | j S )z2:return: amount of bytes currently mapped in total)rr r s r Úmapped_memory_sizez)StaticWindowMapManager.mapped_memory_size© r¡ r c ó — | j S )z5:return: maximum amount of handles we may have opened)rq r s r Úmax_file_handlesz'StaticWindowMapManager.max_file_handles s € à×%Ñ%Ð%r c ó — | j S )z1:return: maximum amount of memory we may allocate)rp r s r Úmax_mapped_memory_sizez-StaticWindowMapManager.max_mapped_memory_size± s € à×$Ñ$Ð$r c óÒ — t j dk7 ryd}| j j « D ]3 \ }}|j |« sŒ|D ] }|j « |dz
|
|||
|
|
}Œ Œ5 |S )a× ONLY AVAILABLE ON WINDOWS
|
|||
|
|
On windows removing files is not allowed if anybody still has it opened.
|
|||
|
|
If this process is ourselves, and if the whole process uses this memory
|