Databases Basics and Spacial Matrix - Discussig Geographic Potentials of Databases.

Editor's Notes

• #2: So, Good Evening all, So let’s dive directly into databases. Many of the faces are relatively new to me. So I will just introduce myself.
• #3: So We divide the presentation into two horizons. Different Databases around us and A quick but boring introduction to GeoSpatial Arrangement of Data. - means data types while handling latitude and longitudes, distances, areas and lines, and Querying in them.
• #4: Before that, let’s put a question for the developers here:“ Assume you are a developer at LinkedIN, (come on yaar, we don’t need to pay tax for that, at least now) and you are assigned to share a Tim Cook’s post to all his 3rd and 4th degree connections in his geographical area! What will be your “data structure” and “query” to find those users? - This is the problem we are trying to tackle here. --- So before that, as Developer, what is our core responsibility?? To manage soft copy of data, Right? I hope so! Do someone have any other opinion? (Yeah! I respect that as well.) When the world is running, our big 3 headaches starts with ‘V’. They are: Volume, Velocity and Variety of data. So in order to control them, we don’t have a magic spell like Harry Potter uses. So we need engineering. The Primary Step will be classifying and storing different type of datas into different formats, Where we can perform operations on them. So There we forms …  Next: Common Data Types
• #5: … Data Types. So, if we need to operate it efficiently; as you know, we need them to be in an optimal structure; at least this topic will be boring for some of us. These are Boolean to store true or false, we can store dates times, numbers, files, and majorly text datas. These things are self explanatory. So we can skip to next topic, How Datas are stored in Relational DB. Next: How data is stored in Relational DB.
• #6: Hope everyone is aware of term “Relational Database”. For those haven’t, Here is the most simplified definition: - This is an application, something like photoshop, google chrome, but primary purpose is to get connected by other programs, to store, modify and retrieve data efficiently. You can see a sample table above and a csv in the right side. We can imagine our data we insert into database will be stored in such a manner. --- So for those who already have an idea, Have you ever wondered, how these datas has been stored in db? This is out of our scope, yet in one line, assume for the left dataset, We have | title - max 70 chars | qty - 4 chars (smallint) | area name - maybe max 20 chars | bay-name - may be 6 chars Net it will take around 100 chars space for each row. So regardless of data is there or not, the space is holded there and can access a data, with some simple calculations, if we know the starting address of table and size of any row and delta of space for each column. Form the row.Now you can think of select * from table where areaname=’Warehouse’; can be executed over your custom script. Next Slide: This is known as Relational DB!
• #7: And Now, This is self explanatory! As usual, A tabular storage of data. So, Relational DB is one of the advanced database techniques from 1970s. And is really useful when we have a structured data, or when we want to have a clean data in structured form! I'm Just skipping this slide as you will be familiar with this type of databases..This brings us to the question,... do we really need a structure to store data??? Well, its upto us. And here we have another type of database to hep us, which is NOSQL Database!
• #8: I know, at least few of you expected an example of MongoDB in this slide! Just Kidding! But I put this intentionally, to change the perspective. And Of Course! MongoDB is a good example! Here we are discussing about key value databases, which holds a is a major share of nosql database. So again, what is Database?, Databases are some applications or “files” which helps us to store and retrieve data “in-its-ease”. This can be persistent or might have some other purpose some times.When Relational database have a Structured Querying Language to fetch data, No SQL’s do not tends to follow a tabulated storage structure and they have their own querying pattern. Here the example shows how to interact with Redis. I know many of you might be familiar with Redis as a cache application or some as in memory db ? Redis focuses on performance and store data in memory. So what’s the benefit? Fetching from memory will be around at least 6 times faster than DB Storage. Retrieval of data, which makes it extremely efficient in non persistent datasets. Let’s Dive into some usages other than caching: Twitter uses redis to create your timeline - Twitter expects around 3 lakhs read per second and 6K writes per second. Instead of creating a timeline on my request, Twitter handles it in more efficient way. When each of my friend tweets, twitter will add that post id into my timeline, which is getting constructed in redis against my user id. So when I enter to redis, So on my request, Pre constructed post just have to get loaded. OK COOL! Next: So Do you remember our first question about LinkedIN Post? Let’s partially solve it next..
• #9: Here Comes, Graph DataBases! When a Relational DB uses tables to store data, and Redis stores data into a location mapped with a key, A Graph Database uses something called "nodes" and "edges" to represent data and relationships between them. Neo4j is one of the prominent player in this segment. You can see some some Neo4J Queries to create 3 users and define a relation “FRIEND” between u1-u2, u2-u3 and u3-u1. If we ignore the case of u3-u1, u2 is the first level friend of u1 and u3 is the second level friend of u2. It brings up a mapping, u3 is one of the 2nd degree connection for u1. Thus a Graph DB can efficiently store and retrieve data using relations. Now let’s just imagine how this can be implemented in Google Maps “in its initial days”.
• #10: When we go to depth, we can store the datas: kakkanad, palarivattom, vyttila and Palachuvadu with distance between them as weight in the relation distance, and time between them as the relation time. Kakkanad x Palarivattom (4 kms) Kakkanad x Padamugal (3 kms) Palarivattom x Vyttila (4 kms) Padamugal x Vyttila (4 kms) Now we can track all possible ways between kakkanad and vyttila. (two ways will be via palarivattom and via vyttila.) (Of Course, google have much efficient way, but for an MVP, this will be enough.) USAGE EXAMPLE: As we discussed first, Making user networks in social medias like LinkedIN / Twitter. Store Places to generate navigation in google maps / System.
• #11: This is one of the less efficient Databases, for its features, Its my personal opinion, but horizontally scalable database to hold enormous volume of data. Explain the Usage of Amazon DynamoDB at Gateway Aggregator Project. We have more variants in it like column Databases, One of the prominent is HBase, Database created directly over Hadoop in Big Data Network. I feel lucky to work on even that. USAGE EXAMPLE: - Store Chat History (You can use a unique key for chat room, to access all the chats).
• #12: Search Database are in mostly :in-memory: databases without promising much persistence, but provide extremely high efficiency, potentially in filtering searching, ordering and complex analytics. USAGE EXAMPLE: Product Search in Ecommerce AutoSuggestion. Summary Calculation. Live Analytics I brought you an interesting case study here: Let’s enlarge the screen shot of flipkart in the top corner.
• #13: Sorry, If you look into this today, you will not be able to find it, flipkart has discontinued this feature, due to extreme load. I believe they have performed a good cost cutting by eliminating this minute feature. Maybe this feature isn’t performing well in their ROI conversion matrix. I took it from an old version of flipkart.com at Web Archives. > Each and everything starting from color of a button in Flipkart Tech is a chapter for another presentation, and extremely recommendable for UIUX engineers, Developers, BA and Tech leads. You can read more at blog.flipkart.tech Back to our topic. When you choose a category these things dynamically updates product count in each filtering. I have personally computed them efficiently using elasticsearch for a project and these are amazingly interesting. But if we proceed to use a non matching db like Relational for this project, they will kill the business itself. We are eating up our own time, so let’s sweep into next slide.
• #14: Time Series Databases. These store datas against time as keys and capable to handle huge writes.In the example, you can see “Prometheus”!! I guess Devops Team knows it! In one of the previous projects, We used it to store and analyze K8S pods log in a fintech domain; So that we can analyze errors and api specific logs and much more with simple UI and queries;Before proceeding; Just discuss read intensive and write intensive databases;So There are much more; Which will be out of our scope! Column DB is one of them!  USAGE EXAMPLE : Server logs, hardware / IOT logs e.t.c. It’s been so long in the first part, and now we moving on to the second part of the slide. Anyone who have lost the bus in between, it’s the last stop for you to re-enter.
• #15: So coming to Geospatial Arrangement of data. Let’s again take a small problem; assume you are a developer at Airbnb or Booking.com, and You are assigned to locate nearby hotels from my location. What will be your query?
• #16: So before we start, we need to be in the same page on some information. In order to point out a particular location in map, we need a Geographic Information System. We use latitude and longitude for them. This is a standard. And this standard is known as SRID 4326. It plans the earth surface into a mesh of horizontal and vertical lines named (latitudes and longitudes). Another popular one is “what-three-words”; it do not even uses latitude and longitude. This uses words.
• #17: And here is some information about srid and its standardization. I can share the slide if someone wants. Let’s skip this theoryfor now.
• #18: So coming to technology, as we discussed in the above slides, we have different data types to store each type of data. And now we unlocked some new data types. We want to mark POINTS, LINES, OR A POLYGON in a map. And we need to store them in Our DATABASES.
• #19: So here are some common data types related to GEOLOGICAL INFORMATION. Again POINT, LINESTRING and POLYGON are self explanatory. GEOMETRY is like a data type which holds any of the above data types. Additionally against each of them, we may require to plot multiple points or multiple lines as a part of same field, such as a road or street lights in a street. For that we also have some additional data types to store a collection of data..
• #20: So this is how we create a table, with a geolocation point which stores a point.And while inserting data, we can simply pass oru point as string and our database utility functions will convert them for us. I know these are a bit complex. Just understand the working rather than trying to read complex function names. Function names can be referred always.
• #21: Here is a query to search for 3 nearest hotels which is within 5000 meters radius from my location [POINT(76.2673 9.9312)]. This also returns a air way distance between my point and that hotel’s location in meters.
• #22: And we can even use some polygons to find all hotels inside it. For that we can pass the polygon and the location field with a contains function.