What linq is about
- 1. What LINQ is aboutor, Playingwith LINQ, MongoDB, and someold new patterns for developers.or, How to injectfunctionalparadigms in imperativelanguagesor, parallelism made easy.WebWorkersCamp, 03/07/2010Pierre Couzy – Microsoft France
- 2. Who are you ?Worked in the Microsoft space for 20 yearsSpeaker, trainer, developer, architect, …Primarilyinterested in Web topicsIdentity, architecture, WS-*, interop, cloud, …Recentinterop/OSS activityWorked on drupal for Sql server projectWorks on automateddrupaltesting on WindowsSpeaker at major PHP eventsDrupalCon, PHP Forum, Symfony liveblog.couzy.com / pierre.couzy@microsoft.com
- 3. AgendaLINQMore LINQLINQ for MongoDBGoingParallelGoingDistributedQ&ADisclosure: This session is heavily Microsoft-Biased (which is to be expected, I work there)
- 4. LINQ : Why ?C# is an imperativelanguageWhichisniceExcept for a few thingsSplitting data manipulation betweenyour code and your data storeCoding data manipulationApplying new manipulation patterns to existing data
- 6. Whatwe’dlikeisExpress more intent, lessimplementationEasy compositionStrongtypingThis issubject to hot debateBe as declarative as possible within the limitsimposed by C#
- 7. From C# to LINQ
- 8. Things to noticeLazyevaluationIf westepintothis code, wesee LINQ triggers itsmagic on demandEasy compositionIf weadd new clauses after the LINQ expression isbuilt but beforeit’sconsumed, wesimplymerge the additions to the existing expressionSo far, we’veworked on objectsWhichis how manynaive LINQ implementationswork : synchronously pipelining methods calls on collectionsLINQ has an interesting twistExpression trees
- 9. Expression Trees and IQueryableLet’stake the sameexample and introduceAsQueryable()Uponexecution, wecan notice the compiler didn’teatour LINQ expressionIt’sstill in there, using a tree format (an AbstractSymbolTree)That treewillberesolvedatruntimeIn the Linq to Objects case, wegetexactly the samebehaviour. The iteration pattern maystillbeyield-based, but thisisleft to IQueryable’simplementationdetails.
- 10. Playingwith expression treesLet’stackleLinq To SQLMy expression treeisnowpushed to the database layerIf I want to block the flow to SQL, I just have to insert a new node in the tree.Whichallows me to balance thingsUsing C# patterns and methodswhen I wantthemPushing to the databasewhatitdoes best
- 11. IQueryable goodiesSo, if I wantmythings to be LINQ aware, I caneitherBuildsomemethods and let the IEnumerable<T> do its pure objectmagic (I stillgetlazyeval)Or, playwithIQueryableAnd do myowninterpretation of the treeLet’sseeMongoDB in LINQ
- 12. Strongly-typed interaction when querying and updating collections.Improved interface to send common Mongo commands (creating indices, getting all the existing dbs and collections, etc.).Ultra-fast de/serialization of BSON to .Net CLR types and back.LINQ-to-MongoSupport for Mono
- 13. LINQ PatternsHaving a LINQ Expression isnice: youbasicallydelegate the execution to someoneelse’s codePlus, your LINQ Expressions have every good aspect youusuallyexpectfromfunctionalprogramming (ie, they are closures).So, they are perfectlysuited to parallelisationAll you have to do isaddanothernode to the expression treeDemo : AsParallel()
- 14. WritingMapReduce in LINQ public static IQueryable<R> MapReduce<S,M,K,R>(this IQueryable<S> source, Expression<Func<S,IEnumerable<M>>> mapper, Expression<Func<M,K>> keySelector, Expression<Func<K,IEnumerable<M>,R>> reducer){ return source.SelectMany(mapper).GroupBy(keySelector, reducer);}Notice that the callingsyntax : Source.MapReduce(lMapper, lKeySelector, lReducer)issimply a new nodethatcanbeinserted in a larger expression tree.This MapReducemethodcanrun on N coresusing PLINQ, or on X machines (and N cores per machine) usingDryadLINQ
- 15. Collection<T> collection;boolIsLegal(Key k); string Hash(Key);var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};DryadLINQ = LINQ + DryadVertexcodeQueryplan(Dryad job)DatacollectionC#C#C#C#results
- 16. Parallel executionvar logentries = from line in logswhere !line.StartsWith("#")select new LogEntry(line);var user = from access in logentrieswhere access.user.EndsWith(@"\ulfar")select access;var accesses =from access in usergroup access by access.page into pagesselect new UserPageCount("ulfar", pages.Key, pages.Count());var htmAccesses =from access in accesseswhere access.page.EndsWith(".htm")orderby access.count descendingselect access;
- 17. Thanks.