(Ab)Using the MetaCPAN API for Fun and Profit
Download as KEY, PDF3 likes2,336 views
The document discusses using the MetaCPAN API to retrieve and filter data from the MetaCPAN database. It provides examples of using the API to fetch author data for a specific author, search for authors by profile or country, and filter results. It also covers more advanced techniques like using Boolean filters, sorting results, and pagination via the scrolling API. Additional resources and tips are provided, such as caching responses and enabling compression.
![{
dir => "id/M/MS/MSTROUT",
email => ["perl-stuff@trout.me.uk"],
gravatar_url => "https://secure.gravatar.com/avatar/...",
name => "Matt S Trout",
pauseid => "MSTROUT",
website => ["http://www.trout.me.uk/"],
}](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-24-320.jpg)
![{
_shards => { failed => 0, successful => 5, total => 5 },
hits => {
hits => [
{
_id => "KHAMPTON",
_index => "cpan_v1",
_score => 1,
_source => {
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
},
_type => "author",
},
],
max_score => 1,
total => 9780,
},
timed_out => bless(do{(my $o = 0)}, "JSON::XS::Boolean"),
took => 1,
}](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-30-320.jpg)
![my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 1,
},
);
# dump $result->{hits}->{hits}->[0]->{_source};](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-31-320.jpg)
![{
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton
@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
}](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-32-320.jpg)
![my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } }
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-37-320.jpg)
![my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-38-320.jpg)
![my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);
# “missing” isn’t really helpful in this search
# just an example of how you might use it](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-39-320.jpg)
![my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-40-320.jpg)
![my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
fields => [ 'pauseid', 'country' ],
},
);](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-41-320.jpg)
![my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
sort => [ { 'author.pauseid' => 'ASC' } ],
},
);](https://image.slidesharecdn.com/abusingmetacpan-120615151940-phpapp01/85/Ab-Using-the-MetaCPAN-API-for-Fun-and-Profit-42-320.jpg)
(Ab)Using the MetaCPAN API for Fun and Profit
- 1. (Ab)Using the MetaCPAN API for Fun and Profit Olaf Alders (OALDERS) @wundercounter
- 2. Architecture • Built on ElasticSearch • Uses Catalyst as a thin wrapper • You don’t need to know this
- 5. iCPAN - iPad
- 6. Android
- 17. What can we build?
- 18. What can we build? • Do something with Github
- 19. What can we build? • Do something with Github • Get a list of all CPAN authors who have enabled the “hireable” flag in their Github profiles
- 21. Let’s Get Started • We want to fetch some data
- 22. Let’s Get Started • We want to fetch some data • We’ll use Sawyer’s MetaCPAN::API
- 23. #!/usr/bin/env perl use strict; use warnings; use MetaCPAN::API; my $mcpan = MetaCPAN::API->new(); my $author = $mcpan->author('MSTROUT');
- 24. { dir => "id/M/MS/MSTROUT", email => ["perl-stuff@trout.me.uk"], gravatar_url => "https://secure.gravatar.com/avatar/...", name => "Matt S Trout", pauseid => "MSTROUT", website => ["http://www.trout.me.uk/"], }
- 28. my $author = $mcpan->author('MSTROUT');
- 29. my $result = $mcpan->post( 'author', { query => { match_all => {} }, size => 1, }, );
- 30. { _shards => { failed => 0, successful => 5, total => 5 }, hits => { hits => [ { _id => "KHAMPTON", _index => "cpan_v1", _score => 1, _source => { city => "Los Angeles", country => "US", dir => "id/K/KH/KHAMPTON", email => ["khampton@totalcinema.com", "kip.hampton@tamarou.com"], gravatar_url => "http://www.gravatar.com/avatar/...", name => "Kip Hampton", pauseid => "KHAMPTON", profile => [ { id => "ubu", name => "coderwall" }, { id => "ubu", name => "github" }, { id => "kiphampton", name => "twitter" }, ], region => "CA", updated => "2011-07-22T20:42:06", website => ["http://totalcinema.com/"], }, _type => "author", }, ], max_score => 1, total => 9780, }, timed_out => bless(do{(my $o = 0)}, "JSON::XS::Boolean"), took => 1, }
- 31. my $result = $mcpan->post( 'author', { query => { match_all => {} }, size => 1, }, ); # dump $result->{hits}->{hits}->[0]->{_source};
- 32. { city => "Los Angeles", country => "US", dir => "id/K/KH/KHAMPTON", email => ["khampton@totalcinema.com", "kip.hampton @tamarou.com"], gravatar_url => "http://www.gravatar.com/avatar/...", name => "Kip Hampton", pauseid => "KHAMPTON", profile => [ { id => "ubu", name => "coderwall" }, { id => "ubu", name => "github" }, { id => "kiphampton", name => "twitter" }, ], region => "CA", updated => "2011-07-22T20:42:06", website => ["http://totalcinema.com/"], }
- 33. my $result = $mcpan->post( 'author', { query => { match_all => {} }, size => 100, }, );
- 34. my $filter = { { term => { 'author.profile.name' => 'stackoverflow', } }, }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, }, );
- 35. use Pithub; my $p = Pithub->new; AUTHOR: foreach my $author ( @{ $result->{hits}->{hits} } ) { foreach my $profile ( @{ $author->{_source}->{profile} } ) { if ( $profile->{name} eq 'github' ) { my $username = $profile->{id}; $username =~ s{https?://github.com/(w*)/?}{$1}i; next AUTHOR if !$username; if ( $p->users->get( user => $username )->content->{hireable} ) { # do something... } next AUTHOR; } } }
- 36. Getting fancy
- 37. my $filter = { and => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } } ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, }, );
- 38. my $filter = { and => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } }, { exists => { 'field' => 'author.region' } }, ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, }, );
- 39. my $filter = { and => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } }, { exists => { 'field' => 'author.region' } }, { missing => { 'field' => 'author.location' } }, ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, }, ); # “missing” isn’t really helpful in this search # just an example of how you might use it
- 40. my $filter = { or => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } }, { exists => { 'field' => 'author.region' } }, { missing => { 'field' => 'author.location' } }, ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, }, );
- 41. my $filter = { or => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } }, { exists => { 'field' => 'author.region' } }, { missing => { 'field' => 'author.location' } }, ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, fields => [ 'pauseid', 'country' ], }, );
- 42. my $filter = { or => [ { term => { 'author.profile.name' => 'github', } }, { term => { 'author.country' => 'US', } }, { exists => { 'field' => 'author.region' } }, { missing => { 'field' => 'author.location' } }, ] }; my $result = $mcpan->post( 'author', { query => { match_all => {} }, filter => $filter, size => 100, sort => [ { 'author.pauseid' => 'ASC' } ], }, );
- 43. Getting Help • #metacpan or irc.perl.org • https://metacpan.org/about/resources
- 44. Resources • https://github.com/CPAN-API/cpan- api/wiki/Beta-API-docs • http://www.slideshare.net/ clintongormley/terms-of-endearment- the-elasticsearch-query-dsl-explained
- 45. Bonus Slides
- 48. Convenience Endpoints • /author/DOY • /distribution/Moose • /release/Moose • /module/Moose • /pod/Moose
- 50. Exporting Pod • /pod/Moose?content-type=text/html (default) • /pod/Moose?content-type=text/plain • /pod/Moose?content-type=text/x-pod • /pod/Moose?content-type=text/x-markdown
- 52. The (real) Endpoints • /author
- 53. The (real) Endpoints • /author • /distribution
- 54. The (real) Endpoints • /author • /distribution • /favorite
- 55. The (real) Endpoints • /author • /distribution • /favorite • /rating
- 56. The (real) Endpoints • /author • /distribution • /favorite • /rating • /release
- 57. The (real) Endpoints • /author • /distribution • /favorite • /rating • /release • /file
- 58. The (real) Endpoints • /author • /distribution • /favorite • /rating • /release • /file
- 60. Using a cache use HTTP::Tiny::Mech; use MetaCPAN::API; use WWW::Mechanize::Cached; my $mcpan = MetaCPAN::API->new( ua => HTTP::Tiny::Mech->new( mechua => WWW::Mechanize::Cached->new() ) );
- 62. Enable Compression • use WWW::Mechanize::Gzip • use WWW::Mechanize::Cached::Gzip • Or set the appropriate request header
- 63. Use the scrolling API • The scrolling API allows you to iterate over an arbitrary number of results • Be aware that when you scroll, your docs will come back unsorted
Editor's Notes
- #2: show of hands: \n1) have used the metacpan search site \n2) use it as their default search site \n3) have worked with the API\n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: CPAN visualization tool\n
- #12: \n
- #13: \n
- #14: \n
- #15: Exports Pod into a format you can import right into your Kindle app.\n
- #16: \n
- #17: Drop-in replacement for Perldoc. Read documentation for modules which you haven’t even installed. Genius.\n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: You can see that for the email and website fields, we allow you to provide a list rather than a single value. Now, for our example we need an author’s Github profile. Matt Trout does not provide this, so he’s a bad test case for our script.\n
- #24: Things like StackOverflow, Twitter and Github usernames are all provided by authors voluntarily after logging in to MetaCPAN. In order to see what the profiles look like in a data structure, we need to find an author who has filled these fields.\n
- #25: You can see from Mo’s example here that he has filled out some of his profile information. He’s a good test case. Note the MetaCPAN explorer link on the bottom left corner. These links can also be found on the module and release pages.\n
- #26: This is a great way to explore the various endpoints of the API and practice crafting queries by hand. However, today we’re just concerned with the /author endpoint.\n
- #27: \n
- #28: \n
- #29: You can see here that since we’re no longer using a convenience endpoint, the output is a little busier. What we generally care about here is the list provided inside of hits->{hits}. In each list item, we care about _source and _source->{profile} in particular.\n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n