Announcement Announcement Module
No announcement yet.
MongoDB MapReduce with Spring MVC Pagination Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • MongoDB MapReduce with Spring MVC Pagination

    Hail fellow, well met.

    I've been playing around with MongoDB lately, specifically its map/reduce functionality, and I was hoping someone out there might be able to lend a hand.

    I am trying to find a simple way to page through a results set from a Mongo data base. This page will be displayed on a jsp page using Spring MVC.

    From my previous tests, querying the database and forming a page are pretty straight forward:

    //Create page request and form a query
    PageRequest pageRequest = new PageRequest(currentPage, totalOnPage, Sort.Direction.DESC, "timeStamp");
    Query query = queryConstructor.formQuery(formData);
    //use above query to get results, stick those results in a page, add page to the model
    List<LogRecord> results= mongoTemplate.find(query, LogRecord.class)
    PageImpl<LogRecord> page = new PageImpl<LogRecord>(results, pageRequest, COLLECTION_NAME.count());
    model.add("logRedcords", page);
    This works without an issue: the results list is sorted in descending order according to each element's timeStamp field, each page has the correct number of elements, I can enter new values for currentPage and I get the expected result, etc, etc.

    The issue comes when I try to repeat this same approach using the mongoTemplate.mapReduce(..) function, as follows:

    PageRequest pageRequest = new PageRequest(currentPage, totalOnPage, Sort.Direction.DESC, "timeStamp");
    Query query = queryConstructor.formQuery(formData);
    //query.with(pageRequest); <- This breaks everything mayan 2012 style, so I'll skip over it for now.
    MapReduceResults<ThreadReport> results = mongoTemplate.mapReduce(query, COLLECTION_NAME, MAP_FUNCTION, REDUCE_FUNCTION, ThreadReport.class);
    List<ThreadReport> listResults= turnIntoList(results);
    PageImpl<ThreadReport> page = new PageImpl<ThreadReports>(listResults, pageRequest, COLLECTION_NAME.count());
    model.add("threadReports", page);
    This works (kind of). The page I add to the model has the ENTIRE results set, not just the first numberOnPage. The elements are not even in decending order according to their timeStamp field, or any field for that matter.

    The map and reduce functions work fine, they do everything I'm asking. The results set also is totally correct and everything I expect to be in there is.

    The problem is just splitting up the results via PageRequest and PageImpl (using the same approach as I did in the first example).

    The problem *could* be that I am unable to perform the query.with(pageRequest) call. I skipped over it because it crashes the whole program. I read in the documentation that you cannot perform Mongo's mapReduce(..) call and Mongo's skip() call in the same operation, which I only guess query.with(Pageable) uses behind the curtains.

    So, I came up with the following work around:
    Maintain the huge results list in some wrapper class.
    Sort that list based on my needs.
    When the user requests a new page, iterate from the bottomIndex = n*totalOnPage element to the topIndex = bottomIndex + (totalOnPage-1) element.
    Stick those elements in a new list
    Stick that new list in a PageImpl and stick the page in the model.

    What do you guys think? Am I working to hard? Does the framework have this functionality already built into it and I just haven't discovered it yet?

    Thanks a million. Hope to hear back before the end of the world, don't want to go out not knowing.

    P.S. I may/may not be a newb.

  • #2
    Shameless bump #1.

    I am only curious if there is a simple way to get the MapReduceResults object to work cleanly with a PageImpl<> object. The MapReduce(..) operation can take a Query object (which can be given an instance of Pageable, where a user can define a page size), however, performing a map reduce this way does not return an instance of a Page. Instead, it returns a MapReduceResults object that does not even have the specified number of results in the Pageable instance. The MapReduceResults will have M reduced objects, where M is the number of objects OUT OF the specified N that would have made up what ever page you are currently viewing.

    I have thus had to create some obnoxious work-arounds to get a total of N reduced objects and then put them in a PageImpl.

    I just have a feeling I am doing something wrong. Any advice would be greatly appreciated.