Improving Performance and Flexibility of Content Listings Using Criteria API

Improving Performance and Flexibility
of Content Listings Using Criteria API
Nils Breunese

Public Broadcaster since 1926 The Netherlands

Online since 1994 Open-source CMS released in 1997

Using Magnolia since 2010 Still migrating websites

Tens of thousands of pages Multiple sites like that

Thanks for the warning… Even 10 seconds would be way too long
WARN info.magnolia.module.cache.filter.CacheFilter
-- The following URL took longer than 10 seconds
(63969 ms) to render. This might cause timeout
exceptions on other requests to the same URI.

Overview models Standard Templating Kit

Tracking back from the template newsOverview.ftl
(...)
[#assign pager = model.pager] 
[#assign newsList = cmsfn.asContentMapList(pager.pageItems)!]
(...)

Constructing the pager AbstractItemListModel
public STKPager getPager() throws RepositoryException { 
(...) 
return new STKPager(currentPageLink, getItems(), content); 
}

Four step pipeline AbstractItemListModel
public Collection<Node> getItems() throws RepositoryException { 
List<Node> itemsList = search(); 
 
this.filter(itemsList); 
this.sort(itemsList); 
itemsList = this.shrink(itemsList); 
 
return itemsList; 
}
1
2
3
4

Step 1a: Constructing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) {
(...)
StringBuffer sql = new StringBuffer(
"select * from nt:base where jcr:path like '"
+ path + "/%'");
(...add 'mgnl:template=' clauses...)
(...add 'ORDER BY' clauses...) 
return getWrappedNodesFromQuery(sql.toString(),
repository, maxResultSize);
} maxResultSize == Integer.MAX_VALUE

Step 1b: Executing the query TemplateCategoryUtil
public static List<Node> getContentListByTemplateNames(...) {
(...) 
NodeIterator items = QueryUtil.search(
repository,
sql.toString(),
Query.SQL,
NodeTypes.Content.NAME);
}

Step 2: Filtering the item list STKDateContentUtil
public static void filterDateContentList(...) { 
CollectionUtils.filter(itemsList, new Predicate() { 
@Override 
public boolean evaluate(Object object) { 
(...)
return date.after(minDate) && date.before(maxDate); 
} 
}); 
}

Step 3: Time to sort STKDateContentUtil
public static void sortDateContentList(...) { 
Collections.sort(itemsList, new Comparator<Node>() { 
@Override 
public int compare(Node c1, Node c2) { 
(...) 
if (StringUtils.equals(sortDirection, ASCENDING)) { 
return date2.compareTo(date1); 
} 
return date1.compareTo(date2); 
} 
}); 
}

Step 4: Shrinking the list STKTemplatingFunctions
public List<Node> cutList(List<Node> itemsList, final int maxResults) { 
if (itemsList.size() > maxResults) { 
return itemsList.subList(0, maxResults); 
} 
return itemsList; 
}
NewsOverviewModel passes Integer.MAX_VALUE,
so shrink does effectively nothing in this case

Step 5: Get the items from the pager STKPager
public Collection getPageItems() { 
 
Collection subList = items; 
int offset = getOffset(); 
if (count > 0) { 
int limit = maxResultsPerPage + offset; 
if (items.size() < limit) { 
limit = count; 
} 
subList = ((List) items).subList(offset, limit); 
 
} 
return subList; 
}
maxResultsPerPage is
typically something like 20

When this becomes a problem We have multiple sites like this
select * from nt:base
where jcr:path like '/siteX/news/%' AND
mgnl:template = 'standard-templating-kit:pages/stkNews'
20000 pages under website:/siteX/news
Four step pipeline returns STKPager with 20000 items (page nodes)
[#assign model.pager]
[#assign newsList = cmsfn.asContentMapList(pager.pageItems)!]
STKPager returns list with 20 page nodes
19980 Node objects created, but not rendered

A query could do all steps at once JCR queries are pretty ﬂexible

Everything in a single JCR query Only 20 nodes returned
SELECT * FROM nt:base
WHERE jcr:path LIKE '/siteX/news/%' AND
mgnl:template = 'standard-templating-kit:pages/stkNews'
AND jcr:created < cast('2016-06-07T00:00:00.000Z' AS DATE)
ORDER BY date ASCENDING
LIMIT 20 OFFSET 20
Search
Filter
Sort
Paging

Criteria API For those familiar with Hibernate/JPA
Criteria criteria = JCRCriteriaFactory.createCriteria() 
.setBasePath("/siteX/news") 
.add(Restrictions.eq(
"@mgnl:template", "standard-templating-kit:pages/stkNews")) 
.add(Restrictions.betweenDates("@jcr:created", minDate, maxDate)) 
.addOrder(Order.asc("date")) 
.setPaging(20, 1);
 
ResultIterator<...> items = criteria.execute(session).getItems();
Sort
Paging
Filter
Search

Criteria API for Magnolia CMS Magnolia module by Openmind

jcr-criteria https://github.com/vpro/jcr-criteria

Custom pager Only a single page worth of items
public class VtkPager<T> extends STKPager {
 
private final List<? extends T> items;
private final int pageSize; 
private final int count; 
 
(...) 
 
@Override 
public List<? extends T> getPageItems() { 
return items; 
}
}

Use it in your model classes VtkContentListModel (vpro)
public abstract class VtkContentListModel ... { 
 
protected final VtkPager<ContentMap> pager; 
 
@Override 
public String execute() { 
pager = createPager(); 
return super.execute(); 
} 
 
protected abstract VtkPager<T> createPager();
(...)
}

Concrete Example VtkNewsOverviewModel (vpro)
@Override 
protected VtkPager<Node> createPager() {
(...) 
AdvancedResult result = JCRCriteriaFactory.createCriteria() 
.setBasePath(path) 
.add(Restrictions.in("@mgnl:template", templates)) 
.add(Restrictions.betweenDates("@jcr:created", minDate, maxDate))
.addOrder(Order.asc("date"))
.setPaging(itemsPerPage, pageNumberStartingFromOne)
.execute(session); 
List<Node> items = new ArrayList<>(); 
for (AdvancedResultItem item : result.getItems()) { 
items.add(item.getJCRNode()); 
} 
 
int count = result.getTotalSize(); 
 
return new VtkPager<>(link, items, content, itemsPerPage, count);
}

Still this. Was it all for nothing? :o(
WARN info.magnolia.module.cache.filter.CacheFilter
-- The following URL took longer than 10 seconds
(63969 ms) to render. This might cause timeout
exceptions on other requests to the same URI.

Example VtkNewsOverviewModel (vpro)
@Override 
protected VtkPager<Node> createPager() {
(...) 
AdvancedResult result = JCRCriteriaFactory.createCriteria() 
.setBasePath(path) 
.add(Restrictions.in("@mgnl:template", templates)) 
.add(Restrictions.betweenDates("@jcr:created", minDate, maxDate))
.addOrder(Order.asc("date"))
.setPaging(itemsPerPage, pageNumberStartingFromOne)
.execute(session); 
List<Node> items = new ArrayList<>(); 
for (AdvancedResultItem item : result.getItems()) { 
items.add(item.getJCRNode()); 
} 
 
int count = result.getTotalSize(); 
 
return new VtkPager<>(link, items, content, itemsPerPage, count);
}
This call takes 10-60+ seconds!

AdvancedResultImpl (jcr-criteria)
@Override 
public int getTotalSize() { 
if (totalResults == null) { 
int queryTotalSize = -1; 
try { // jcrQueryResult instanceof JackrabbitQueryResult) { 
Method m = jcrQueryResult.getClass().getMethod("getTotalSize"); 
queryTotalSize = (int) m.invoke(jcrQueryResult); 
} catch (InvocationTargetException | IllegalAccessException e) { 
LOG.error(e.getMessage(), e); 
} catch (NoSuchMethodException e) { 
 
} 
if (queryTotalSize == -1 && (itemsPerPage == 0 || applyLocalPaging)) { 
try { 
totalResults = (int) jcrQueryResult.getNodes().getSize(); 
} catch (RepositoryException e) { 
// ignore, the standard total size will be returned 
} 
} 
 
if (queryTotalSize == -1) { 
totalResults = queryCounter.getAsInt(); 
} else { 
totalResults = queryTotalSize; 
 
} 
} 
return totalResults; 
}
We end up here

jackrabbit-core 2.8.0
protected void getResults(long size) throws RepositoryException { 
(...) 
result = executeQuery(maxResultSize); // Lucene query
(...)
// Doesn’t use result.getSize(), call collectScoreNodes(...)
}
private void collectScoreNodes(...) { 
while (collector.size() < maxResults) { 
ScoreNode[] sn = hits.nextScoreNodes();
(...) 
// check access 
if (isAccessGranted(sn)) { 
collector.add(sn); 
} else { 
invalid++; 
} 
} 
} QueryResultImpl

It used to be fast! https://issues.apache.org/jira/browse/JCR-3858

jackrabbit-core 2.10.0+
protected void getResults(long size) throws RepositoryException {
(...)
if (sizeEstimate) { 
numResults = result.getSize(); // Use count from Lucene 
 
} else { 
// do things the Jackrabbit 2.8.0 way 
(...) 
}
(...)
}
QueryResultImpl

Enable Jackrabbit’s 'sizeEstimate' Jackrabbit 2.10+
<SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
(...)
<param name="sizeEstimate" value="true"/>
</SearchIndex>

Rendering times down to 1-2 seconds Bingo

Feel free to contact me
Nils Breunese

@breun

n.breunese@vpro.nl

Improving Performance and Flexibility of Content Listings Using Criteria API

More Related Content

What's hot (20)

More from Nils Breunese (6)

Recently uploaded (20)

Improving Performance and Flexibility of Content Listings Using Criteria API