SlideShare a Scribd company logo
Porting to Python 3

        Lennart Regebro
   EuroPython 2010, Birmingham
Python 3 is here
Loads of goodies!
● print is a function
● All strings are unicode


● Binary data handled by new bytes type


● dict.keys/items/values() are generators


● Integers are all long


● 3/2 == 1.5


● The standard library is reorganized


● New easter egg
Choosing a strategy
Only support Python 3
● For end-user software and systems
● Run 2to3 once


● Then fix the code until it works.


● Done!
Separate branches
● For stable python modules
● If you only do bugfixes


● Make a branch for the old version


● Run 2to3 once


● Fix until it works


● Bug fix both branches
How to distribute
● Separate package names on pypi
● Add a warning and error in the setup.py:


  “For Python 2 support, please use version
  3.x of MyCoolModule”
● Only release binary eggs (bdist_egg)


● Include both src2/ and src3/ and have

  different setup() parameters for different
  Python versions.
Continuous conversion with 2to3
● If you need to support Python 2 and Python
  3 and the code is changing a lot
● Keep developing in Python 2


● Convert to Python 3 for testing and install


● Only one source distribution


● Compatibility hacks may be needed
Solution: Distribute!
● Very little work to set up 2to3 support
● Runs 2to3 on testing and installing:


     – Only one source tree
     – Only one distribution
Using Distribute


setup(name='py3examples',
    ...
    test_suite='py3example.tests.test_suite',
    use_2to3=True,
    convert_2to3_doctests=['py3example/README.txt'],
)
Single code base; no conversion
● If 2to3 for some reason is infeasible
● Support Python 2 and Python 3 from the

  same code without 2to3
● No distribution worries!


● Loads of compatibility hacks


● Fun, but ugly!
When?
● For end-user systems: When you feel like it
● For modules: As soon as you can


● I.e. when your dependencies do
Preparing for the port
Get rid of warnings
● Run under Python 2.6 with -3
● Fix all deprecation warnings
Use // instead of /
In Python 2:
  >>> 3/2
  1
In Python 3:
  >>> 3/2
  1.5
In Python 2.2 and later:
  >>> 3//2
  1
Prepare for bytes
● Use separate variables for binary data
● Add 'b' and 't' to file flags
Write tests


Increasing your test coverage will
   simplify porting and increase
      confidence in the port
Porting
Porting with 2to3



2to3 -w .
2to3 -w -d .
2to3 -w -d README.txt
Common Problems
Comparisons
● cmp() is gone
● cmp = lambda a, b: (a > b) - (a < b)
Bytes vs Strings vs Unicode


 The unicode type is now called string

        The string type is gone

  The bytes type handles binary data
Bytes:
● Used for binary data
>>> file = open('maybe_a.gif', 'rb')
>>> file.read(6)
b'GIF89'

>>> b'GIF89'[2]
70
Solutions
>>> import sys
>>> if sys.version < '3':
... def b(x):
...        return x
... else:
... import codecs
... def b(x):
...       return codecs.latin_1_encode(x)[0]

>>> b('binaryx67data')
Still behaves differently
>>> data = open('a.gif', 'rb').read()
>>> data[2]

Python 2: 'F'
Python 3: 70
Wrapper class for Python 2
class bites(str):
   def __new__(cls, value):
      if isinstance(value[0], int):
          # It's a list of integers
          value = ''.join([chr(x) for x in value])
      return super(bites, cls).__new__(cls, value)

   def itemint(self, index):
      return ord(self[index])

   def iterint(self):
      for x in self:
          yield ord(x)
Wrapper class for Python 3
class bites(bytes):
   def __new__(cls, value):
      if isinstance(value, str):
          # It's a unicode string:
          value = value.encode('ISO-8859-1')
      return super(bites, cls).__new__(cls, value)

   def itemint(self, x):
      return self[x]

   def iterint(self):
      for x in self:
          yield x
Usage
>>> from bites import bites
>>> bites('GIF89a').itemint(2)
70

>>> for x in bites('GIF89a').iterint():
... print(x),
71 73 70 56 57 97
Mixed/unknown files
●   Open in binary mode, and decode


    >>> data = open('maybe_a.gif', 'rb').read()
    >>> if data[:6] != bites('GIF89a'):
    ... str = data.decode('Latin-1')
Unorderable types
● Many built in types that were comparable
  now are not
● __cmp__ is no longer supported
Use “Rich comparison methods”
● __lt__
● __le__


● __eq__


● __ge__


● __gt__


● __ne__
Doctests
● Binary data now is a b'' literal
● Unicode data no longer has the u''


● Classes and types have often changed

  names or at least __repr__
● Exceptions are formated differently as a

  result, and exception matching doesn't
  work under both versions.
Test with ==
Go from:

>>> myfunction()
u'Foobar'

To:

>>> myfunction() == u'Foobar'
True
Capture exceptions
>>> try:
...    dosomethingthatfails()
...    print “ExpectedException not raised”
... except ExpectedException:
...      print “Success”
Success
Without 2to3
Workaround for stdlib reorg
try:
       from StringIO import StringIO
except ImportError:
       from io import StringIO
Workaround for except:
●   Python 2:
        except Exception, e:
●   Python 3:
        except Exception as e:
●   Both:
        except Exception:
           e = sys.exc_info[1]
Workaround for print()
●   For simple print cases:
         print(“Something”) works everywhere.
         print(“Something”, “else”) doesn't.
●   Make your own print function!
Modernize your code
Don't float ints
Python 2:
 >>> float(3)/2
 1.5

Python 2.2 and later:
 >>> from __future__ import division
 >>> 3/2
 1.5
Sorting
●   Old way:
       – Needs lists
       – Slow
       – Comparison methods

●   New way:
       – Sort any iterable
       – Fast
       – Key methods = trivial (often lambdas)
Old way
>>> def lastnamefirst(a, b):
...    s = "%s, %s" % (a.lastname, a.firstname)
...    o = "%s, %s" % (b.lastname, b.firstname)
...    return cmp(s, o)
>>> thelist = [Person('Donald', 'Duck'), Person('Paul', 'Anka')]
>>> thelist.sort(cmp=lastnamefirst)
>>> thelist
[Paul Anka, Donald Duck]
Exponential slowness
● The cmp function compares pairs
● Averages (from Jarret Hardie):


     – 4 items: 6 calls (1.5 per item)
     – 10 items: 22 calls (2.2 per item)
     – 100 items: 528 calls (5.28 per item)
     – 40,000 items: 342,541 calls (8.56 per item)
New way
>>> def lastnamefirst(a):
... return "%s, %s" % (a.lastname, a.firstname)
>>> thetuple = (Person('Donald', 'Duck'), Person('Paul', 'Anka'),)
>>> sorted(thetuple, key=lastnamefirst)
[Paul Anka, Donald Duck]
Fast!



The method is called exactly one time per item
Use iterator methods


                  Instead of
      {}.keys(), {}.values(), {}.items()
                      Use
{}.iterkeys(), {}.itervalues(), {}.iteritems()
Use iterator methods
● In Python 3, {}.keys(), {}.values(),
  {}.items() are iterators
● 2to3 wraps them in list(), so d.values()

  becomes list(d.values())
● By explicitly using iterators in Python 2

  code, you avoid the list()
Put an in in it


●   Old: adict.has_key(foo)

●   New: foo in adict
C-code
● There are API changes
● But you can handle them with ifdefs.
2 * 3 = Six
● A module to help with compatibility
● Several compatibility methods:


       – b()
       – print_()
       – etc...

●   Tons of helpful constants:
       – integer_types
       – string_types
       – etc...
That's it!
Comparison Example
class ComparableMixin(object):
  def __lt__(self, other): return self._compare(other, (-1,))
  def __le__(self, other): return self._compare(other, (-1, 0,))
  def __eq__(self, other): return self._compare(other, (0,))
  def __ge__(self, other): return self._compare(other, (0, 1,))
  def __gt__(self, other): return self._compare(other, (1,))
  def __ne__(self, other): return self._compare(other, (-1, 1,))
Comparison Example
class Orderable(ComparableMixin):
  def __init__(self, firstname, lastname):
     self.firstname = firstname
     self.lastname = lastname
   def _compare(self, other, truthvalues):
     try:
        s = "%s, %s" % (self.lastname, self.firstname)
        o = "%s, %s" % (other.lastname, other.firstname)
        value = (s > o) - (s < o)
        return value in truthvalues
     except AttributeError:
        return NotImplemented

More Related Content

PPTX
Python programming
PDF
python codes
PDF
Python Cheat Sheet
PDF
Learn 90% of Python in 90 Minutes
PDF
Matlab and Python: Basic Operations
PPTX
Basics of Python programming (part 2)
PDF
Functions in python
ODP
Python course Day 1
Python programming
python codes
Python Cheat Sheet
Learn 90% of Python in 90 Minutes
Matlab and Python: Basic Operations
Basics of Python programming (part 2)
Functions in python
Python course Day 1

What's hot (20)

PPTX
Introduction to the basics of Python programming (part 1)
ODP
PDF
Python
PPTX
Introduction to the basics of Python programming (part 3)
PDF
Python in 90 minutes
PPTX
Learn python in 20 minutes
PDF
PDF
Python Puzzlers - 2016 Edition
PPTX
SQL Server Select Topics
PDF
Python Basics
PPTX
Python language data types
PDF
Python Programming: Data Structure
PDF
Python 2.5 reference card (2009)
PPTX
13. Java text processing
PDF
Python Functions (PyAtl Beginners Night)
PPTX
FUNCTIONS IN PYTHON. CBSE +2 COMPUTER SCIENCE
PPTX
Python 3.6 Features 20161207
PPTX
Python programing
PPTX
Programming in Python
PPT
Introduction to Python Language and Data Types
Introduction to the basics of Python programming (part 1)
Python
Introduction to the basics of Python programming (part 3)
Python in 90 minutes
Learn python in 20 minutes
Python Puzzlers - 2016 Edition
SQL Server Select Topics
Python Basics
Python language data types
Python Programming: Data Structure
Python 2.5 reference card (2009)
13. Java text processing
Python Functions (PyAtl Beginners Night)
FUNCTIONS IN PYTHON. CBSE +2 COMPUTER SCIENCE
Python 3.6 Features 20161207
Python programing
Programming in Python
Introduction to Python Language and Data Types
Ad

Similar to Porting to Python 3 (20)

PDF
Porting to Python 3
PPT
Os Vanrossum
PDF
Pycon taiwan 2018_claudiu_popa
PDF
Lecture1_cis4930.pdf
PDF
Python lecture 05
PDF
What's new in Python 3.11
PDF
Mastering Python 3 I/O (Version 2)
PPTX
Learning python
PPTX
Learning python
PPTX
Learning python
PPTX
Learning python
PPTX
Learning python
PPTX
Learning python
PPTX
Learning python
PDF
Intro to Python
PPT
Python Evolution
PDF
SunPy: Python for solar physics
KEY
State of Python (2010)
PPT
Python basics to advanced in on ppt is available
Porting to Python 3
Os Vanrossum
Pycon taiwan 2018_claudiu_popa
Lecture1_cis4930.pdf
Python lecture 05
What's new in Python 3.11
Mastering Python 3 I/O (Version 2)
Learning python
Learning python
Learning python
Learning python
Learning python
Learning python
Learning python
Intro to Python
Python Evolution
SunPy: Python for solar physics
State of Python (2010)
Python basics to advanced in on ppt is available
Ad

Recently uploaded (20)

PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
Approach and Philosophy of On baking technology
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
Network Security Unit 5.pdf for BCA BBA.
PPT
Teaching material agriculture food technology
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Empathic Computing: Creating Shared Understanding
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Advanced IT Governance
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
PDF
cuic standard and advanced reporting.pdf
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
PPTX
MYSQL Presentation for SQL database connectivity
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Per capita expenditure prediction using model stacking based on satellite ima...
Dropbox Q2 2025 Financial Results & Investor Presentation
Approach and Philosophy of On baking technology
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Reach Out and Touch Someone: Haptics and Empathic Computing
Network Security Unit 5.pdf for BCA BBA.
Teaching material agriculture food technology
Understanding_Digital_Forensics_Presentation.pptx
Advanced methodologies resolving dimensionality complications for autism neur...
Empathic Computing: Creating Shared Understanding
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Advanced IT Governance
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
cuic standard and advanced reporting.pdf
20250228 LYD VKU AI Blended-Learning.pptx
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
MYSQL Presentation for SQL database connectivity
NewMind AI Weekly Chronicles - August'25 Week I
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf

Porting to Python 3

  • 1. Porting to Python 3 Lennart Regebro EuroPython 2010, Birmingham
  • 2. Python 3 is here
  • 3. Loads of goodies! ● print is a function ● All strings are unicode ● Binary data handled by new bytes type ● dict.keys/items/values() are generators ● Integers are all long ● 3/2 == 1.5 ● The standard library is reorganized ● New easter egg
  • 5. Only support Python 3 ● For end-user software and systems ● Run 2to3 once ● Then fix the code until it works. ● Done!
  • 6. Separate branches ● For stable python modules ● If you only do bugfixes ● Make a branch for the old version ● Run 2to3 once ● Fix until it works ● Bug fix both branches
  • 7. How to distribute ● Separate package names on pypi ● Add a warning and error in the setup.py: “For Python 2 support, please use version 3.x of MyCoolModule” ● Only release binary eggs (bdist_egg) ● Include both src2/ and src3/ and have different setup() parameters for different Python versions.
  • 8. Continuous conversion with 2to3 ● If you need to support Python 2 and Python 3 and the code is changing a lot ● Keep developing in Python 2 ● Convert to Python 3 for testing and install ● Only one source distribution ● Compatibility hacks may be needed
  • 9. Solution: Distribute! ● Very little work to set up 2to3 support ● Runs 2to3 on testing and installing: – Only one source tree – Only one distribution
  • 10. Using Distribute setup(name='py3examples', ... test_suite='py3example.tests.test_suite', use_2to3=True, convert_2to3_doctests=['py3example/README.txt'], )
  • 11. Single code base; no conversion ● If 2to3 for some reason is infeasible ● Support Python 2 and Python 3 from the same code without 2to3 ● No distribution worries! ● Loads of compatibility hacks ● Fun, but ugly!
  • 12. When? ● For end-user systems: When you feel like it ● For modules: As soon as you can ● I.e. when your dependencies do
  • 14. Get rid of warnings ● Run under Python 2.6 with -3 ● Fix all deprecation warnings
  • 15. Use // instead of / In Python 2: >>> 3/2 1 In Python 3: >>> 3/2 1.5 In Python 2.2 and later: >>> 3//2 1
  • 16. Prepare for bytes ● Use separate variables for binary data ● Add 'b' and 't' to file flags
  • 17. Write tests Increasing your test coverage will simplify porting and increase confidence in the port
  • 19. Porting with 2to3 2to3 -w . 2to3 -w -d . 2to3 -w -d README.txt
  • 21. Comparisons ● cmp() is gone ● cmp = lambda a, b: (a > b) - (a < b)
  • 22. Bytes vs Strings vs Unicode The unicode type is now called string The string type is gone The bytes type handles binary data
  • 23. Bytes: ● Used for binary data >>> file = open('maybe_a.gif', 'rb') >>> file.read(6) b'GIF89' >>> b'GIF89'[2] 70
  • 24. Solutions >>> import sys >>> if sys.version < '3': ... def b(x): ... return x ... else: ... import codecs ... def b(x): ... return codecs.latin_1_encode(x)[0] >>> b('binaryx67data')
  • 25. Still behaves differently >>> data = open('a.gif', 'rb').read() >>> data[2] Python 2: 'F' Python 3: 70
  • 26. Wrapper class for Python 2 class bites(str): def __new__(cls, value): if isinstance(value[0], int): # It's a list of integers value = ''.join([chr(x) for x in value]) return super(bites, cls).__new__(cls, value) def itemint(self, index): return ord(self[index]) def iterint(self): for x in self: yield ord(x)
  • 27. Wrapper class for Python 3 class bites(bytes): def __new__(cls, value): if isinstance(value, str): # It's a unicode string: value = value.encode('ISO-8859-1') return super(bites, cls).__new__(cls, value) def itemint(self, x): return self[x] def iterint(self): for x in self: yield x
  • 28. Usage >>> from bites import bites >>> bites('GIF89a').itemint(2) 70 >>> for x in bites('GIF89a').iterint(): ... print(x), 71 73 70 56 57 97
  • 29. Mixed/unknown files ● Open in binary mode, and decode >>> data = open('maybe_a.gif', 'rb').read() >>> if data[:6] != bites('GIF89a'): ... str = data.decode('Latin-1')
  • 30. Unorderable types ● Many built in types that were comparable now are not ● __cmp__ is no longer supported
  • 31. Use “Rich comparison methods” ● __lt__ ● __le__ ● __eq__ ● __ge__ ● __gt__ ● __ne__
  • 32. Doctests ● Binary data now is a b'' literal ● Unicode data no longer has the u'' ● Classes and types have often changed names or at least __repr__ ● Exceptions are formated differently as a result, and exception matching doesn't work under both versions.
  • 33. Test with == Go from: >>> myfunction() u'Foobar' To: >>> myfunction() == u'Foobar' True
  • 34. Capture exceptions >>> try: ... dosomethingthatfails() ... print “ExpectedException not raised” ... except ExpectedException: ... print “Success” Success
  • 36. Workaround for stdlib reorg try: from StringIO import StringIO except ImportError: from io import StringIO
  • 37. Workaround for except: ● Python 2: except Exception, e: ● Python 3: except Exception as e: ● Both: except Exception: e = sys.exc_info[1]
  • 38. Workaround for print() ● For simple print cases: print(“Something”) works everywhere. print(“Something”, “else”) doesn't. ● Make your own print function!
  • 40. Don't float ints Python 2: >>> float(3)/2 1.5 Python 2.2 and later: >>> from __future__ import division >>> 3/2 1.5
  • 41. Sorting ● Old way: – Needs lists – Slow – Comparison methods ● New way: – Sort any iterable – Fast – Key methods = trivial (often lambdas)
  • 42. Old way >>> def lastnamefirst(a, b): ... s = "%s, %s" % (a.lastname, a.firstname) ... o = "%s, %s" % (b.lastname, b.firstname) ... return cmp(s, o) >>> thelist = [Person('Donald', 'Duck'), Person('Paul', 'Anka')] >>> thelist.sort(cmp=lastnamefirst) >>> thelist [Paul Anka, Donald Duck]
  • 43. Exponential slowness ● The cmp function compares pairs ● Averages (from Jarret Hardie): – 4 items: 6 calls (1.5 per item) – 10 items: 22 calls (2.2 per item) – 100 items: 528 calls (5.28 per item) – 40,000 items: 342,541 calls (8.56 per item)
  • 44. New way >>> def lastnamefirst(a): ... return "%s, %s" % (a.lastname, a.firstname) >>> thetuple = (Person('Donald', 'Duck'), Person('Paul', 'Anka'),) >>> sorted(thetuple, key=lastnamefirst) [Paul Anka, Donald Duck]
  • 45. Fast! The method is called exactly one time per item
  • 46. Use iterator methods Instead of {}.keys(), {}.values(), {}.items() Use {}.iterkeys(), {}.itervalues(), {}.iteritems()
  • 47. Use iterator methods ● In Python 3, {}.keys(), {}.values(), {}.items() are iterators ● 2to3 wraps them in list(), so d.values() becomes list(d.values()) ● By explicitly using iterators in Python 2 code, you avoid the list()
  • 48. Put an in in it ● Old: adict.has_key(foo) ● New: foo in adict
  • 49. C-code ● There are API changes ● But you can handle them with ifdefs.
  • 50. 2 * 3 = Six ● A module to help with compatibility ● Several compatibility methods: – b() – print_() – etc... ● Tons of helpful constants: – integer_types – string_types – etc...
  • 52. Comparison Example class ComparableMixin(object): def __lt__(self, other): return self._compare(other, (-1,)) def __le__(self, other): return self._compare(other, (-1, 0,)) def __eq__(self, other): return self._compare(other, (0,)) def __ge__(self, other): return self._compare(other, (0, 1,)) def __gt__(self, other): return self._compare(other, (1,)) def __ne__(self, other): return self._compare(other, (-1, 1,))
  • 53. Comparison Example class Orderable(ComparableMixin): def __init__(self, firstname, lastname): self.firstname = firstname self.lastname = lastname def _compare(self, other, truthvalues): try: s = "%s, %s" % (self.lastname, self.firstname) o = "%s, %s" % (other.lastname, other.firstname) value = (s > o) - (s < o) return value in truthvalues except AttributeError: return NotImplemented