SlideShare a Scribd company logo

thinking in key value stores

Bhasker Kode, profile picture
Bhasker Kode

The document discusses the crawler infrastructure at Hover.in, detailing its utilization of key-value stores and parallel computing to enhance efficiency, reliability, and speed. It elaborates on the implementation of flow control, metadata storage using Tokyo Cabinet, and in-memory caching using Erlang's Mnesia as part of its data handling strategy. Furthermore, it covers various use cases, advantages of in-memory data structures, and key-value store features that support the platform's operations.

TechnologyNews & Politics
1 of 38
Downloaded 134 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores
thinking in key value stores

More Related Content

PDF
Handling not so big data
SATOSHI TAGOMORI
 
PDF
Fluentd at Bay Area Kubernetes Meetup
Sadayuki Furuhashi
 
PDF
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
PPT
2011 06-30-hadoop-summit v5
Samuel Rash
 
PDF
SF Big Analytics meetup : Hoodie From Uber
Chester Chen
 
PPTX
Hadoop operations-2015-hadoop-summit-san-jose-v5
Chris Nauroth
 
PDF
Overview of data analytics service: Treasure Data Service
SATOSHI TAGOMORI
 
PDF
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Shirshanka Das
 
Handling not so big data
SATOSHI TAGOMORI
 
Fluentd at Bay Area Kubernetes Meetup
Sadayuki Furuhashi
 
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
2011 06-30-hadoop-summit v5
Samuel Rash
 
SF Big Analytics meetup : Hoodie From Uber
Chester Chen
 
Hadoop operations-2015-hadoop-summit-san-jose-v5
Chris Nauroth
 
Overview of data analytics service: Treasure Data Service
SATOSHI TAGOMORI
 
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Shirshanka Das
 

What's hot (20)

PPTX
Functional Comparison and Performance Evaluation of Streaming Frameworks
Huafeng Wang
 
PDF
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit
 
PPTX
January 2011 HUG: Pig Presentation
Yahoo Developer Network
 
PDF
DOD 2016 - Rafał Kuć - Building a Resilient Log Aggregation Pipeline Using El...
PROIDEA
 
PDF
Prestogres internals
Sadayuki Furuhashi
 
PPTX
Hadoop, Hive, Spark and Object Stores
Steve Loughran
 
PDF
The Future of Apache Storm
DataWorks Summit/Hadoop Summit
 
PPTX
A Container-based Sizing Framework for Apache Hadoop/Spark Clusters
DataWorks Summit/Hadoop Summit
 
PPTX
Building a Scalable Web Crawler with Hadoop
Hadoop User Group
 
PDF
Presto in the cloud
Qubole
 
PDF
Presto - Hadoop Conference Japan 2014
Sadayuki Furuhashi
 
PPTX
January 2011 HUG: Howl Presentation
Yahoo Developer Network
 
PDF
Big data: Loading your data with flume and sqoop
Christophe Marchal
 
PPTX
Powering a Virtual Power Station with Big Data
DataWorks Summit/Hadoop Summit
 
PDF
Treasure Data and AWS - Developers.io 2015
N Masahiro
 
PDF
Hdfs high availability
Hadoop User Group
 
PDF
Distributed Stream Processing on Fluentd / #fluentd
SATOSHI TAGOMORI
 
PPT
Spark stream - Kafka
Dori Waldman
 
PPTX
Achieving 100k Queries per Hour on Hive on Tez
DataWorks Summit/Hadoop Summit
 
PDF
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
Functional Comparison and Performance Evaluation of Streaming Frameworks
Huafeng Wang
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit
 
January 2011 HUG: Pig Presentation
Yahoo Developer Network
 
DOD 2016 - Rafał Kuć - Building a Resilient Log Aggregation Pipeline Using El...
PROIDEA
 
Prestogres internals
Sadayuki Furuhashi
 
Hadoop, Hive, Spark and Object Stores
Steve Loughran
 
The Future of Apache Storm
DataWorks Summit/Hadoop Summit
 
A Container-based Sizing Framework for Apache Hadoop/Spark Clusters
DataWorks Summit/Hadoop Summit
 
Building a Scalable Web Crawler with Hadoop
Hadoop User Group
 
Presto in the cloud
Qubole
 
Presto - Hadoop Conference Japan 2014
Sadayuki Furuhashi
 
January 2011 HUG: Howl Presentation
Yahoo Developer Network
 
Big data: Loading your data with flume and sqoop
Christophe Marchal
 
Powering a Virtual Power Station with Big Data
DataWorks Summit/Hadoop Summit
 
Treasure Data and AWS - Developers.io 2015
N Masahiro
 
Hdfs high availability
Hadoop User Group
 
Distributed Stream Processing on Fluentd / #fluentd
SATOSHI TAGOMORI
 
Spark stream - Kafka
Dori Waldman
 
Achieving 100k Queries per Hour on Hive on Tez
DataWorks Summit/Hadoop Summit
 
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
Ad

Viewers also liked (20)

PDF
Hubbub deck short
Yurij Riphyak
 
PPTX
Technology Advancement Core: Our Process
Technology Advancement Core
 
PDF
Pebs14 hubbub bike summit
Birgit Hess
 
PPTX
Scientist and inventor
Lee W.
 
PPTX
Bbr security
BIRENDER SINGH RAWAT
 
PDF
Data Networks: Next-Generation Optical Access toward 10 Gb/s Everywhere
Xi'an Jiaotong-Liverpool University
 
PDF
Short-Run Digital Book Printing
RISO printer
 
DOCX
Thermal bimorph valve operated microthruster.
SAI SIVA
 
PPTX
Powerpoint
kristi_macleod
 
PPTX
Secure develpment 2014
Ariel Evans
 
PPT
Power point presentation
saki-t
 
PPTX
SHMcloud vision
Mark Kerzner
 
PPTX
A brief introduction to Spintronics
Ram Kumar K R
 
PPTX
National river linking project
Raj_Anand
 
PPTX
Project Oxyzen
chetanmistry21
 
PPTX
Project oxygen
linkoravi
 
PPTX
Shaan-E-Awadh
Reliance Retail ltd
 
PPTX
Hyderabadi and Andhra Cuisine
shweta_1712
 
PPTX
Presentation on smartphone by Apu
Ashiqul Islam Apu
 
PPTX
E-books Are Not the Future of Books
Xavier Badosa
 
Hubbub deck short
Yurij Riphyak
 
Technology Advancement Core: Our Process
Technology Advancement Core
 
Pebs14 hubbub bike summit
Birgit Hess
 
Scientist and inventor
Lee W.
 
Bbr security
BIRENDER SINGH RAWAT
 
Data Networks: Next-Generation Optical Access toward 10 Gb/s Everywhere
Xi'an Jiaotong-Liverpool University
 
Short-Run Digital Book Printing
RISO printer
 
Thermal bimorph valve operated microthruster.
SAI SIVA
 
Powerpoint
kristi_macleod
 
Secure develpment 2014
Ariel Evans
 
Power point presentation
saki-t
 
SHMcloud vision
Mark Kerzner
 
A brief introduction to Spintronics
Ram Kumar K R
 
National river linking project
Raj_Anand
 
Project Oxyzen
chetanmistry21
 
Project oxygen
linkoravi
 
Shaan-E-Awadh
Reliance Retail ltd
 
Hyderabadi and Andhra Cuisine
shweta_1712
 
Presentation on smartphone by Apu
Ashiqul Islam Apu
 
E-books Are Not the Future of Books
Xavier Badosa
 
Ad

Similar to thinking in key value stores (20)

PDF
Scaleable PHP Applications in Kubernetes
Robert Lemke
 
PPTX
Infrastructure as code, using Terraform
Harkamal Singh
 
PPT
21 Www Web Services
royans
 
PPT
Apache Traffic Server
supertom
 
PDF
lessons from managing a pulsar cluster
Shivji Kumar Jha
 
PDF
Building a Feature Store around Dataframes and Apache Spark
Databricks
 
PDF
Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx
Lex Avstreikh
 
PDF
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Provectus
 
PDF
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
PPTX
Technology Stack Discussion
Zaiyang Li
 
PDF
Scaling PHP apps
Matteo Moretti
 
PDF
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
HostedbyConfluent
 
PPTX
Advanced JavaScript
Mahmoud Tolba
 
PDF
Presentation on Japanese doc sprint
Go Chiba
 
PDF
Headless approach for offloading heavy tasks in Magento
Sander Mangel
 
PDF
Globo.com & Varnish
lokama
 
PPT
Caching By Nyros Developer
Nyros Technologies
 
PDF
Presemtation Tier Optimizations
Anup Hariharan Nair
 
PDF
Kubernetes for the PHP developer
Paul Czarkowski
 
PDF
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Lucidworks
 
Scaleable PHP Applications in Kubernetes
Robert Lemke
 
Infrastructure as code, using Terraform
Harkamal Singh
 
21 Www Web Services
royans
 
Apache Traffic Server
supertom
 
lessons from managing a pulsar cluster
Shivji Kumar Jha
 
Building a Feature Store around Dataframes and Apache Spark
Databricks
 
Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx
Lex Avstreikh
 
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Provectus
 
Fluentd Overview, Now and Then
SATOSHI TAGOMORI
 
Technology Stack Discussion
Zaiyang Li
 
Scaling PHP apps
Matteo Moretti
 
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
HostedbyConfluent
 
Advanced JavaScript
Mahmoud Tolba
 
Presentation on Japanese doc sprint
Go Chiba
 
Headless approach for offloading heavy tasks in Magento
Sander Mangel
 
Globo.com & Varnish
lokama
 
Caching By Nyros Developer
Nyros Technologies
 
Presemtation Tier Optimizations
Anup Hariharan Nair
 
Kubernetes for the PHP developer
Paul Czarkowski
 
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Lucidworks
 

More from Bhasker Kode (9)

PDF
Kafka & Storm - FifthElephant 2015 by @bhaskerkode, Helpshift
Bhasker Kode
 
PDF
Instrumenting Go (Gopherconindia Lightning talk by Bhasker Kode)
Bhasker Kode
 
PDF
Recursion & Erlang, FunctionalConf 14, Bangalore
Bhasker Kode
 
PDF
Parsing binaries and protocols with erlang
Bhasker Kode
 
PDF
hover.in at CUFP 2009
Bhasker Kode
 
PDF
in-memory capacity planning, Erlang Factory London 09
Bhasker Kode
 
ODP
erlang at hover.in , Devcamp Blr 09
Bhasker Kode
 
PPT
end user programming & yahoo pipes
Bhasker Kode
 
PPT
Overview of End-user Programming | Talk at DesignCamp Bangalore
Bhasker Kode
 
Kafka & Storm - FifthElephant 2015 by @bhaskerkode, Helpshift
Bhasker Kode
 
Instrumenting Go (Gopherconindia Lightning talk by Bhasker Kode)
Bhasker Kode
 
Recursion & Erlang, FunctionalConf 14, Bangalore
Bhasker Kode
 
Parsing binaries and protocols with erlang
Bhasker Kode
 
hover.in at CUFP 2009
Bhasker Kode
 
in-memory capacity planning, Erlang Factory London 09
Bhasker Kode
 
erlang at hover.in , Devcamp Blr 09
Bhasker Kode
 
end user programming & yahoo pipes
Bhasker Kode
 
Overview of End-user Programming | Talk at DesignCamp Bangalore
Bhasker Kode
 

Recently uploaded (20)

PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PDF
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
Microsoft Dynamics
 
PPTX
Chapter 5: Probability Theory and Statistics
RandyFin
 
PPTX
Modernising the Digital Integration Hub
Daniel Toomey
 
PDF
August Patch Tuesday
Ivanti
 
PPTX
The various Industrial Revolutions .pptx
bandileshozi3
 
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
PDF
Unlock new opportunities with location data.pdf
Precisely
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
PPTX
observCloud-Native Containerability and monitoring.pptx
anupsingh76937
 
PDF
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
PDF
sustainability-14-14877-v2.pddhzftheheeeee
VenkateshGovindaraja9
 
PPTX
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
PPT
Module 1.ppt Iot fundamentals and Architecture
nanditha7766
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
PDF
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
Microsoft Dynamics
 
Chapter 5: Probability Theory and Statistics
RandyFin
 
Modernising the Digital Integration Hub
Daniel Toomey
 
August Patch Tuesday
Ivanti
 
The various Industrial Revolutions .pptx
bandileshozi3
 
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
Unlock new opportunities with location data.pdf
Precisely
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
observCloud-Native Containerability and monitoring.pptx
anupsingh76937
 
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
sustainability-14-14877-v2.pddhzftheheeeee
VenkateshGovindaraja9
 
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
Module 1.ppt Iot fundamentals and Architecture
nanditha7766
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954