Introduction to Datastore
1 like1,381 views
The document provides an introduction to Google App Engine's datastore, which is a schema-less persistence system defined by entities with immutable keys and mutable properties. It contrasts traditional relational databases with the datastore's architecture, highlighting its scalability and operational principles, including transactions and indexing via a system known as Bigtable. Additionally, it covers how to use the datastore with JPA and details on entity management, queries, and transactional operations.
Introduction to Datastore
- 1. Introducion to Datastore Assoc.Prof. Dr.Thanachart Numnonda Asst.Prof. Thanisa Kruawaisayawan Mini Master of Java Technology KMITL July 2010
- 2. Agenda What is DataStore? Using DataStore JPA in DataStore
- 4. What is Datastore? Google App Engine Datastore is a schema-less persistence system, whose fundamental persistence unit is called Entity, c omposed by an immutable Key and a collection of mutable pr operties. Entities can be created, updated, deleted, loaded by key and queried for properties values. DataStore is consistent and transactional, with support to current transaction.
- 5. The DataStore The Datastore is not a relational database nor a façade. Relational database technology doesn’t scale horizontally – Connection pools, shared caching are a problem The Datastore is one of many public APIs used for accessing Google’s
- 8. The DataStore : Operations Transactions and Index are based on MegaTable. File persistence it's done with Google File System (GFS). It's distributed by Chubby, a lock service for loosely- coupled distributed systems.
- 9. BigTable BigTable is a compressed, high performance, and proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs Currently not distributed or used outside of Google. BigTable development began in 2004. and is now used by a number of Google application Google Earth, Google Map, Gmail, Youtube, etc..
- 10. BigTable : Design BigTable is a fast and extremely large-scale DBMS. It is a sparse, distributed multi-dimensional sorted map, sharing characteristics of both row-oriented and column- oriented databases. sparse because only "not null" values are persisted distributed in Google cloud persistent on Google File System multidimensional in columns values ordered lexicographically by key
- 11. BigTable : Design Tables are optimized for GFS by being split into multiple tablets - segments of the table. BigTable is designed to scale into the petabyte. Each table has multiple dimensions (one of which is a feld for time, allowing for versioning and garbage collection). It allows an infnite number of rows and columns.
- 12. Google File System GFS is a proprietary distributed fle system developed by Google. It is designed to provide effcient, reliable access to data using large clusters of commodity hardware. GFS grew out of an earlier Google effort, BigFiles, developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.
- 13. Using DataStore
- 14. DataStore Operations Datastore operations are defned around entities (data models) which are objects with one or more properties Types: string, user, Boolean, and so on Entities may be recursive or self-referential Entity relationships are one-to-many or many-to-many. Entities may be fxed or grow as needed.
- 15. DataStore Storage Model Every entity is of a particular kind Entities in a kind need not have the same properties One entity may have different “columns” from another in the same kind! Unique IDs are automatically assigned unless the user defnes a key_name
- 16. Compare DataStore with Others
- 17. DataStore Storage Model Basic unit of storage is an Entity consisting of Kind (table) Key (primary key) Entity Group (partition) 0..N typed Properties (columns)
- 18. Datastore Quotas Each call to Datastore counts towards the quota The amount of data cannot exceed the billable Includes properties and keys but not the indices CPU and Datastore CPU time quotas apply
- 19. Using the Datastore Applications may access the Datastore using the JDO or the JPA classes. The JDO and JPA classes are abstracted using the DataNucleus API Open source Not very popular Support for Java standards Poor documentation
- 20. JPA in DataStore
- 21. Setting Up JPA The JPA and datastore JARs must be in the app's war/WEB-INF/lib/ directory. A confguration fle named persistence.xml must be in the app's war/WEB-INF/classes/META-INF/ directory, A confguration fle tells JPA to use the App Engine datastore. The appengine-api.jar must also be in the war/WEB- INF/lib/ directory.
- 22. persistence.xml: Example <?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?> <persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" <persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd"> http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd"> <persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL"> <persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL"> <provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider <provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider </provider> </provider> <non-jta-data-source/> <non-jta-data-source/> <properties> <properties> <property name="datanucleus.ConnectionURL" value="appengine"/> <property name="datanucleus.ConnectionURL" value="appengine"/> <property name="datanucleus.NontransactionalRead" value="true"/> <property name="datanucleus.NontransactionalRead" value="true"/> <property name="datanucleus.NontransactionalWrite" value="true"/> <property name="datanucleus.NontransactionalWrite" value="true"/> </properties> </properties> </persistence-unit> </persistence-unit> </persistence> </persistence>
- 23. Getting an EntityManager Instance An app interacts with JPA using an instance of the EntityManager. import javax.persistence.EntityManagerFactory; import javax.persistence.EntityManagerFactory; import javax.persistence.Persistence; import javax.persistence.Persistence; public class EMF {{ public class EMF private static final EntityManagerFactory emfInstance == private static final EntityManagerFactory emfInstance Persistence.createEntityManagerFactory("transactions-optional"); Persistence.createEntityManagerFactory("transactions-optional"); public static EntityManagerFactory get() {{ public static EntityManagerFactory get() return emfInstance; return emfInstance; }} }}
- 24. Entity Class : Example @Entity @Entity public class GuestList implements Serializable {{ public class GuestList implements Serializable …… @Id @Id private String id; private String id; @Basic @Basic private User author; private User author; private String content; private String content; @Temporal(javax.persistence.TemporalType.DATE) @Temporal(javax.persistence.TemporalType.DATE) private Date visitDate; private Date visitDate; …… // Getter and Setter methods // Getter and Setter methods }}
- 25. Queries and Indices A query operates on every entity of a given kind. Specify zero or more sort orders Specify zero or more flters on property values Indices are defned in the App Engine confguration fles Results are fetched directly from these indices; no indices are created on the fly WEB-INF/datastore-indexes.xml - non-standard fles Normalization is not recommended Optimization techniques for RDBMSs may result in poor Datastore performance!
- 26. Query : Example EntityManager em == EMF.get().createEntityManager(); EntityManager em EMF.get().createEntityManager(); try {{ try Query query == em.createQuery("SELECT oo FROM GuestList AS o"); Query query em.createQuery("SELECT FROM GuestList AS o"); @SuppressWarnings("unchecked") @SuppressWarnings("unchecked") List<GuestList> results == (List<GuestList>) query.getResultList(); List<GuestList> results (List<GuestList>) query.getResultList(); for (Object obj :: results) {{ for (Object obj results) GuestList guest == (GuestList) obj; GuestList guest (GuestList) obj; String nickname == guest.getAuthor().getNickname(); String nickname guest.getAuthor().getNickname(); out.println(nickname ++ "" "" ++ guest.getId()); out.println(nickname guest.getId()); }} }} catch(Exception ex) {{ catch(Exception ex) out.println(ex); out.println(ex); }}
- 27. Entity Relationships Models association between entities. There are four types of relationship multiplicities: @OneToOne @OneToMany @ManyToOne Supports unidirectional as well as bidirectional relationships Unidirectional relationship: Entity A references B, but B doesn't reference A.
- 28. Example : ManyToOne Mapping
- 29. Example : OneToMany Mapping
- 30. Transactions and Entity Groups Transaction = Group of Datastore operations that either succeed or fail Entity groups are required because all grouped entities are stored in the same Datastore node An entity may be either created or modifed once per transaction Transactions may fail if a different user or process tries an update in the same group at the same time Users decide whether to retry or roll the transaction back
- 31. Transaction in JPA : Example Book book == em.find(Book.class, "9780596156732"); Book book em.find(Book.class, "9780596156732"); BookReview bookReview == new BookReview(); BookReview bookReview new BookReview(); bookReview.rating == 5; bookReview.rating 5; book.getBookReviews().add(bookReview); book.getBookReviews().add(bookReview); Transaction txn == em.getTransaction(); Transaction txn em.getTransaction(); txn.begin(); txn.begin(); try {{ try book == em.merge(book); book em.merge(book); txn.commit(); txn.commit(); }} finally {{ finally if (txn.isActive()) {{ if (txn.isActive()) txn.rollback(); txn.rollback(); }} }}
- 32. Unsupported Features of JPA Owned many-to-many relationships, and unowned relationships. "Join" queries. Aggregation queries (group by, having, sum, avg, max, min) Polymorphic queries.
- 33. Resources Google App Engine for Java HOWTO, Andrew Lombardi, Mar 2010 The Softer Side Of Schemas, Max Ross, May 2009 Official Google App Engine Tutorial, http://code.google.com/appengine/docs/java/gettingstarted/ Programming Google App Engine, Don Sanderson, O'Reilly, 2010
- 34. Thank you thananum@gmail.com twitter.com/thanachart www.facebook.com/thanachart www.thaijavadev.com