![Object-specific allocators
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | <----- Python memory (under PyMem manager's control) ------> | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | <------ Virtual memory allocated for the python process -------> |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
__________________________________ __________________________________
[ ] [ ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |](https://image.slidesharecdn.com/pygc-170614135419/85/python-2-320.jpg)
![where store variable ?
run-time Stack
list
dict
int
heap
[1 ,2, 3]
{“n”: “1”}
1](https://image.slidesharecdn.com/pygc-170614135419/85/python-9-320.jpg)
![why ?
In [1]: a = 123
In [2]: b = 123
In [3]: a is b
Out[3]: True
In [4]: a = 1000
In [5]: b = 1000
In [6]: a is b
Out[6]: False
In [7]: a = 'n'
In [8]: b = 'n'
In [9]: a is b
Out[9]: True
In [10]: a = "python"
In [11]: b = "python"
In [12]: a is b
Out[12]: True](https://image.slidesharecdn.com/pygc-170614135419/85/python-10-320.jpg)
![why ?
In [1]: def go(var):
...: print id(var)
…:
In [2]: id(a)
Out[2]: 4401335072
In [3]: go(a)
4401335072
In [10]: a = b = 'nima'
In [11]: b = a
In [12]: a is b
Out[12]: True
In [13]: b = 'hehe'
In [14]: a is b
Out[14]: False
只有引用 ?](https://image.slidesharecdn.com/pygc-170614135419/85/python-11-320.jpg)
![ref count
x = 300
y = x
z = [x, y]
X
ref += 1
300
y
Z
ref += 1
ref += 2
References -> 4 ！](https://image.slidesharecdn.com/pygc-170614135419/85/python-17-320.jpg)
![container objects
a = [10, 10, 11]
b = a
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
11](https://image.slidesharecdn.com/pygc-170614135419/85/python-24-320.jpg)
![copy.copy
a = [10, 10, [10, 11] ]
b = copy.copy(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
10
ref
PyListObject
10
11](https://image.slidesharecdn.com/pygc-170614135419/85/python-25-320.jpg)
![copy.deepcopy
a = [10, [ 10, 11 ] ]
b = copy.deep(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
ref
PyListObject
10
11
PyListObject
10
11](https://image.slidesharecdn.com/pygc-170614135419/85/python-26-320.jpg)
![diy gc
import gc
import sys
gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK)
a=[]
b=[]
a.append(b)
print 'a refcount:',sys.getrefcount(a) # 2
print 'b refcount:',sys.getrefcount(b) # 3
del a
del b
print gc.collect() # 0](https://image.slidesharecdn.com/pygc-170614135419/85/python-27-320.jpg)
python高级内存管理
- 2. Object-specific allocators _____ ______ ______ ________ [ int ] [ dict ] [ list ] ... [ string ] Python core | +3 | <----- Object-specific memory -----> | <-- Non-object memory --> | _______________________________ | | [ Python's object allocator ] | | +2 | ####### Object memory ####### | <------ Internal buffers ------> | ______________________________________________________________ | [ Python's raw memory allocator (PyMem_ API) ] | +1 | <----- Python memory (under PyMem manager's control) ------> | | __________________________________________________________________ [ Underlying general-purpose allocator (ex: C library malloc) ] 0 | <------ Virtual memory allocated for the python process -------> | ========================================================================= _______________________________________________________________________ [ OS-specific Virtual Memory Manager (VMM) ] -1 | <--- Kernel dynamic storage allocation & management (page-based) ---> | __________________________________ __________________________________ [ ] [ ] -2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
- 3. * Request in bytes Size of allocated block Size class idx * ---------------------------------------------------------------- * 1-8 8 0 * 9-16 16 1 * 17-24 24 2 * 25-32 32 3 * 33-40 40 4 * 41-48 48 5 * 49-56 56 6 * 57-64 64 7 * ... ... ... * 497-504 504 62 * 505-512 512 63 * * */
- 5. method posix malloc python memory pool object buffer pool
- 6. Arena FeeePool Pool Pool Pool Pool Headers No BLock Arena malloc heap & pool Process stack heap bss init data text UserPool 1-8 … … 249 - 256 Pool Free Block Free Block Use Block
- 7. userdpool design UserdPools 1-8 9-16 17-24 … … 249-256 Pool Header Free Block Free Block Header 分配 回收 … Pool Free Block Free Block Use Block 同⼀个Pool下Block⼀样长 单Pool为4kb Block及Pool都为单链表
- 8. free pool desgin FeeePool Pool Pool Pool … Pool Headers No BLock Pool Headers No BLock Pool为4kb⼤小 Pool清理Headers
- 9. where store variable ? run-time Stack list dict int heap [1 ,2, 3] {“n”: “1”} 1
- 10. why ? In [1]: a = 123 In [2]: b = 123 In [3]: a is b Out[3]: True In [4]: a = 1000 In [5]: b = 1000 In [6]: a is b Out[6]: False In [7]: a = 'n' In [8]: b = 'n' In [9]: a is b Out[9]: True In [10]: a = "python" In [11]: b = "python" In [12]: a is b Out[12]: True
- 11. why ? In [1]: def go(var): ...: print id(var) …: In [2]: id(a) Out[2]: 4401335072 In [3]: go(a) 4401335072 In [10]: a = b = 'nima' In [11]: b = a In [12]: a is b Out[12]: True In [13]: b = 'hehe' In [14]: a is b Out[14]: False 只有引用 ?
- 12. python objects stored in memory? names names object Python Has Names, Not Variables ! ! !
- 13. 整数对象池 -5 -4 … 0 … 256 257 小整数 ⼤整数 … … -5 -4 … … 257 … … var_1 var_2 … … -5 -4 … … 257 … … var_3 var_4 the same addr ! not the same addr ! 28 bytes 解释器初始化
- 14. 整数对象池 Block List Free List PyIntBlock PyIntBlock PyIntBlock PyIntBlock 不会归还给 Arena和os ! ! !
- 15. 字符对象池 a b c d … … … var_1 var_2 the same addr ! 单个字符38 bytes 由解释器初始化
- 16. 字符串对象池 aa en cao oh woyao buyao kuai feile var_1 0 1 2 3 … var_2 hash存储变量 共用地址 记录引用计数 ref
- 17. ref count x = 300 y = x z = [x, y] X ref += 1 300 y Z ref += 1 ref += 2 References -> 4 ！
- 18. What does del do? x = 300 y = x del x X ref -= 1 300 y References -> 1！The del statement doesn’t delete objects. • removes that name as a reference to that object • reduces the ref count by 1
- 19. ref count case def go(): w = 300 go() a = “fuc . ” del a b = “en, a” b = None ref count +1 w is out of scope; ref count -1 del a; ref count -1 重新赋值; ref count -1
- 20. cyclical refclass Node: def __init__(self, va): self.va = va def next(self, next): self.next = next mid = Node(‘root’) left = Node(‘left’) right = Node(‘right’) mid(left) left.next(right) right.next(left) Mid rightleft if del mid node: how ?
- 21. mark & sweep gc root b a w c K G R
- 22. 分代回收 node node node node node node node node node node node node node node node node node node node
- 23. 可变 vs 不可变 (obj) string int tuple list dict
- 24. container objects a = [10, 10, 11] b = a PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 11
- 25. copy.copy a = [10, 10, [10, 11] ] b = copy.copy(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 ref PyListObject Type list rc 1 items size … … 10 10 ref PyListObject 10 11
- 26. copy.deepcopy a = [10, [ 10, 11 ] ] b = copy.deep(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 ref PyListObject Type list rc 1 items size … … 10 ref PyListObject 10 11 PyListObject 10 11
- 27. diy gc import gc import sys gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK) a=[] b=[] a.append(b) print 'a refcount:',sys.getrefcount(a) # 2 print 'b refcount:',sys.getrefcount(b) # 3 del a del b print gc.collect() # 0
- 28. Garbage Collector Optimize memory bound 可以降低threshold来时间换空间 cpu bound 提⾼threshold来空间换时间 暂停gc, 引⼊master worker设计
- 29. 引用计数 跟 gil 的影响 ? gc 是否是原⼦ ? gc的 stop the world现象 ? … Q & A
- 30. “ END ” – xiaorui.cc