Iterators & generators: practical uses in memory management

Iterators & Generators
Practical Uses in Memory Management

Reach out
● Software Engineer
● Systems Administrator
● All around nerd
@aramonc

Memory Management in PHP
● Garbage collected
○ Reference counts
○ Happens after variable is unset, leave function
scope, or script ends, GC buffer is full
● All values are stored in hash tables of ZVAL structs
● Arrays are dynamically allocated by doubling the
size
● Much better in PHP 7 than 5.6

What are Iterators and Generators
Iterators are:
● An object that can be iterated over using a loop
● PHP has several built in the SPL
● Can create additional ones
● Special ones are the recursive & filter iterators
Generators are:
● A special kind of custom iterator
● Keywords & interfaces that allow you to define an
Iterator without having to create a class that
implements the interface

https://github.com/aramonc/ig_talk

Generating many values: The object
<?php
namespace ArcIgTalk;
class User
{
public $_id;
public $firstName;
public $lastName;
public $email;
public $password;
public $phoneNumber;
}

Generating many values: Generate some values
<?php
...
for ($i = 0; $i < 1000000; $i++) {
$user = new User();
$user->_id = new ObjectID();
$user->email = $faker->email;
$user->password = $faker->password(12, 32);
$user->firstName = $faker->firstName();
$user->lastName = $faker->lastName;
$user->phoneNumber = $faker->phoneNumber;
$users[] = $user;
if ($i % 5000 === 0) {
$collection->insertMany($users);
$users = [];
}
}

Generating many values: Generate some values

Processing values: Paginated Provider
<?php
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): array
{
$cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]);
$users = [];
foreach ($cursor as $data) {
$users[] = $this->createUserEntity($data->getArrayCopy());
}
return $users;
}
}

Processing values: Simple approach
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 10000; $up++) {
foreach ($provider->getPagedList($up) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);

Processing values: Simple approach

Processing values: Why does it take so long?
● 3 nested loops
○ for ($up = 0; $up < 10000; $up++)
○ foreach ($provider->getPagedList($up) as $user)
○ foreach ($cursor as $data)
● O(n^3) or n*m*p

Processing values: What is going on?
<?php
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): array
{
$cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]);
$users = [];
$users[] = $this->createUserEntity($data->getArrayCopy());
}
return $users;
}
}

<?php
for ($up = 0; $up < 10000; $up++) {
foreach ($provider->getPagedList($up) as $user) {
}
}
fclose($emailLog);

Processing values: Make it faster
<?php
for ($up = 0; $up < 1000; $up++) {
foreach ($provider->getPagedList($up, 1000) as $user) {
}
}
fclose($emailLog);

<?php
for ($up = 0; $up < 1000; $up++) {
}
}
fclose($emailLog);

<?php
for ($up = 0; $up < 100; $up++) {
}
}
fclose($emailLog);

Processing values: Make it better
<?php
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): Generator
{
$cursor = $this->collection->find([],[ 'limit' => $limit,'skip' => $page * $limit]);
/** @var $data BSONDocument */
yield $this->createUserEntity($data->getArrayCopy());
}
}
}

Processing values: Make it better

Processing values: It’s a bit slower
● 3 nested loops
○ for ($up = 0; $up < 10000; $up++)
○ foreach ($provider->getPagedList($up) as $user)
○ foreach ($cursor as $data)
● Effectively only 2
● O(n^2) or n*m
● Extra time for YIELD to go back & forth

We can do more! Let’s filter the list!

Filter values: Abstract the pagination
<?php
class Service
{
const LIMIT = 100000;
public function getList(int $limit): Generator
{
$pages = ceil($limit / self::LIMIT);
$innerLimit = $pages > 1 ? self::LIMIT : $limit;
for ($page = 0; $page < $pages; $page++) {
foreach ($this->provider->getPagedList($page, $innerLimit) as $user) {
yield $user;
}
}
}
}

Filter values: The filter
<?php
...
function filterGmailUsers(User $user): bool
{
$emailParts = explode('@', $user->email);
return $emailParts[1] === 'gmail.com';
}
$filtered = new CallbackFilterIterator(
$service->getList(1000000),
'filterGmailUsers'
);
foreach ($filtered as $user) {
}

http://bit.ly/2tgw6z7

Iterators & generators: practical uses in memory management

More Related Content

What's hot (20)

Similar to Iterators & generators: practical uses in memory management (20)

More from Adrian Cardenas (7)

Recently uploaded (20)

Iterators & generators: practical uses in memory management