Offline first: application data and synchronization

Editor's Notes

• #2: TODO: Chat example should explain choosing of sync structure for different fields (title, messages, participants) Tell them how important to understand approaches to not work around hacks and bugs (my experience during SCollab development)
• #5: Current approach of mobile app (web, hybrid, native) development takes into account that a client has connection to server or sometimes can have connection breaks and a user can wait until the connection is restored. We live in a disconnected & battery powered world, but our technology and best practices are a leftover from the always connected & steadily powered past.
• #6: But this paradigm is not complete because: connection might blink device might not detect connection status change immediately connection might be unstable (request handled but not communicated to the client back) So, to bring super-stable experience the client should be super-smart and the server should understand it.
• #9: Field workers TV, internet, cell providers, doctor visits, energy providers, plumbers, etc. Store observation Technical audit Car, house, utilities assets audit Medical observation
• #11: TODO: Insert a picture of mobile device and cell/radio tower
• #12: TODO: insert a diagram “start” of offline-first app branches - User Interface - User Experience - Offline Data - Connectivity - Synchronization - Troubleshooting
• #13: TODO: insert a diagram “start” of offline-first app branches - User Interface - User Experience - Offline Data - Connectivity - Synchronization - Troubleshooting
• #14: TODO: Insert a picture of mobile device and cell/radio tower
• #15: ## Application itself - scripts, - styles - media resources ## Static meta-data - configs and settings - e.g. how to connect to services, client-server time difference, - layout declaration - permissions - translations ## Business data - dictionaries - fetched data - entered data (aka User data) ## System runtime data - logs (action, error, profiling) - session data (app state, offline authentication) - transaction states (data sync, app updates)
• #16: ## Application itself - scripts, - styles - media resources
• #17: ## Static meta-data - configs and settings - e.g. how to connect to services, client-server time difference, - layout declaration - permissions - translations
• #18: ## Business data - dictionaries - fetched data - entered data (aka User data)
• #19: ## System runtime data - logs (action, error, profiling) - session data (app state, offline authentication) - transaction states (data sync, app updates)
• #20: Data synchronization is the process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time. It is fundamental to a wide variety of applications, including file synchronization and mobile device synchronization
• #21: One-way sync: data is only synced from the server to the apps (e.g. news app where content is synced from the backend CMS to the apps) or data is synced from the apps to a server (e.g. logging/analytics). Two-way sync: data is synced in two directions, from an app to a backend and back. E.g. a user is logged in and can manage his own data on a website and in an app (assuming user cannot be logged in on 2 devices at the same time). Multi-way sync: data is synced from multiple devices to a server/backend and back. This also means that data from one device is synced to the server and from the server to other devices (e.g. collaboration apps...).
• #22: One-way sync: data is only synced from the server to the apps (e.g. news app where content is synced from the backend CMS to the apps) or data is synced from the apps to a server (e.g. logging/analytics). Two-way sync: data is synced in two directions, from an app to a backend and back. E.g. a user is logged in and can manage his own data on a website and in an app (assuming user cannot be logged in on 2 devices at the same time). Multi-way sync: data is synced from multiple devices to a server/backend and back. This also means that data from one device is synced to the server and from the server to other devices (e.g. collaboration apps...).
• #23: Traditional pessimistic replication systems try to guarantee from the beginning that all of the replicas are identical to each other, as if there was only a single copy of the data all along. Optimistic replication does away with this in favor of eventual consistency, meaning that replicas are guaranteed to converge only when the system has been quiesced for a period of time. As a result there is no longer a need to wait for all of the copies to be synchronized when updating data, which helps concurrency and parallelism. The trade-off is that different replicas may require explicit reconciliation later on, which might then prove difficult or even insoluble.
• #24: Informally, eventual consistency means that replicas eventually reach the same value if clients stop submitting updates. Eventually consistent systems accept local updates without remote synchronization, improving performance and scalability by sacrificing strong consistency. Without remote synchronization, replicas concurrently hold different values which are expected to converge over time. Convergence is complicated by conflicts which arise when merging values between replicas. A conflict is a combination of concurrent updates which may be individually correct, but taken together violate some system invariant. A conflict is a combination of concurrent updates which may be individually correct, but taken together violate some system invariant. Conventional conflict-resolution schemes involve state roll-back, full consensus, or even user interaction.
• #25: An optimistic replication algorithm consists of five elements: Operation submission: Users submit operations at independent sites. Propagation: Each site shares the operations it knows about with the rest of the system. Scheduling: Each site decides on an order for the operations it knows about. Conflict resolution: If there are any conflicts among the operations a site has scheduled, it must modify them in some way. Commitment: The sites agree on a final schedule and conflict resolution result, and the operations are made permanent. There are two strategies for propagation: state transfer, where sites propagate a representation of the current state, and operation transfer, where sites propagate the operations that were performed (essentially, a list of instructions on how to reach the new state). Scheduling and conflict resolution can either be syntactic or semantic. Syntactic systems rely on general information, such as when or where an operation was submitted. Semantic systems are able to make use of application-specific information to make smarter decisions. Note that state transfer systems generally have no information about the semantics of the data being transferred, and so they have to use syntactic scheduling and conflict resolution.
• #26: Strong eventual consistency is a property of some eventually-consistent systems: replicas that have received and applied the same set of updates must immediately have equivalent state. There is no conflict arbitration process, because conflicts do not exist in strongly-consistent systems. CRDTs are used to achieve strong eventual consistency in a distributed system.
• #29: State-based CRDTs are called convergent replicated data types, or CvRDTs. In contrast to CmRDTs, CvRDTs send their full local state to other replicas. CvRDTs have the following local interface:
• #30: query - reads the state of the replica, with no side effects update - writes to the replica state in accordance with certain restrictions merge - merges local state with the state of some remote replica The merge function must be commutative, associative, and idempotent. It provides a join for any pair of replica states, so the set of all states forms a semilattice. The updatefunction must monotonically increase the internal state, according to the same partial order rules as the semilattice.
• #31: A common strategy in CRDT development is to stick multiple primitive CRDTs together to make a more complex CRDT. In this case, two increment-only counters were combined to create a CvRDT supporting both increment and decrement operations. Note that the CvRDT's internal state must increase monotonically, even though its external state as exposed throughquery can return to previous values.
• #32: The grow-only set is a CvRDT implementing a set which only allows adds. Since it is impossible for adds and removes to commute (one must take precedence over the other), any CvRDT supporting both add and remove operations must pick and choose its semantics.
• #33: Two grow-only set CvRDTs are combined to create the 2P-set CvRDT. With the addition of a "tombstone" set, elements can be added and also removed. Once removed, an element cannot be re-added; that is, once an element e is in the tombstone set, query will never again return True for that element. The 2P-set uses "remove-wins" semantics, so remove(e) takes precedence over add(e).