Announcement Announcement Module
Collapse
No announcement yet.
Problems with MongoItemReader Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Problems with MongoItemReader

    Hi all,

    I found a weird issue with MongoItemReader.
    This is the configuration I've for a step:

    Code:
        private Step addMnpInformation() throws Exception {
            return stepBuilders.get("addMnpInformation")
                    .chunk(50)
                    .reader(stagedCdrReader())
                    .processor(addMnpProcessor())
                    .writer(stagedCdrWriter())
                    .build();
        }
    and this is the reader
    Code:
        @Bean
        public MongoItemReader<StagedCdr> stagedCdrReader() throws Exception {
            MongoItemReader reader = new MongoItemReader<StagedCdr>();
            reader.setTemplate(stagingCdrTransactionManager);
            reader.setQuery("{cdrStatus:'PENDING'}");
            reader.setTargetType(StagedCdr.class);
            HashMap<String,Direction> sort = new HashMap<String, Direction>();
            sort.put("sequence", Direction.ASC);
            reader.setSort(sort);
            reader.setName("stagedCdrReader");
            reader.afterPropertiesSet();
    
            return reader;
        }
    It seems the reader is not reading every other page. The records I'm procesing has a sequential Id and I found it's reading records using a page of 50 (probably my chunk size) but in this way:

    from 0 to 49
    from 100 to 149
    from 200 to 249
    .
    .
    .

    to it's not reading
    from 50 to 99
    from 150 to 199
    .
    .
    .

    Did I missed something configuring the reader or is this a bug?

  • #2
    I forgot about this one.
    I discovered that the processor was changing the field cdrStatus to COMPLETED. That way, when the reader requested the new page and it's using limit and skip, the changed records of the first page where not included in the new page request so the 'new' first page was skipped and records not retrieved.

    Is this the expected behavior? In that case, if I need to mark records as COMPLETED, what should be the best approach?

    I also found that using skip is very slow when having few thousand documents in the collection. Is there a way to avoid this?

    Comment

    Working...
    X