Announcement Announcement Module
Collapse
No announcement yet.
SkipLimit - using same keys Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • SkipLimit - using same keys

    Hi,

    The scenario is reading from the database and spewing onto a file with skip limits to be configured. As a test 4 records available in the DB to spew out the file
    This is my simple job configuration
    Code:
    	<bean id="orderOutputJob" parent="simpleJob">
    		<property name="name" value="orderOutputJob" />
    		<property name="steps">
    			<list>
    				<bean id="step1" parent="skipLimitStep">
    					<property name="commitInterval" value="100" />
    					<property name="skipLimit" value="5"/> 
    					<property name="itemReader" ref="orderItemReader" />
    					<property name="itemWriter" ref="orderWriter"/>
    				</bean>
    			</list>
    		</property>
    	</bean>
    where skipLimitStep is a simple implementation of the SkipLimitStepFactoryBean

    Now, I got my mapper mapped to the orderReader and this is the code

    Code:
    public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
        	try{
                log.debug("Going to call the row map at rowcount"+rowNum);
                
                int columnCount = rs.getMetaData().getColumnCount();
                Object[] obj    = new Object[columnCount];
                for(int colIndex=1;colIndex<columnCount+1;colIndex++){
                	obj[colIndex-1] = rs.getObject(colIndex);
                }
                log.debug("Mapping completed successfully "+obj);
                return obj;
        	}catch(Exception ex){
        		log.warn("Error while mapping dataset row to object array",ex);
        		throw new WMException("BATCH-0005",ex,null);
        	}
        }
    As you can see, I return an object array of the values rather than a domain object coz it is a generic implementation....In my writer I forcefully coz an exception to test my limit handling..

    Now here's the strange thing..While the first and the third records get picked up successfully, the second and the third record do not get picked up with the key already
    exists
    Here are the 4 unique object arrays created
    Code:
    //Fine
    OrderMapper.mapRow(27) | Mapping completed successfully [Ljava.lang.Object;@97e3a5
    
    //Not Fine
    OrderMapper.mapRow(27) | Mapping completed successfully [Ljava.lang.Object;@a166bd
    ItemSkipPolicyItemHandler.read(145) | Skipping item on input, previously failed on output; key=[[Ljava.lang.Object;@97e3a5]
    
    //Fine
    OrderMapper.mapRow(27) | Mapping completed successfully [Ljava.lang.Object;@60029d
    
    //Not Fine
    OrderMapper.mapRow(27) | Mapping completed successfully [Ljava.lang.Object;@e41d4a
    ItemSkipPolicyItemHandler.read(145) | Skipping item on input, previously failed on output; key=[[Ljava.lang.Object;@97e3a5]
    Any help on this is appreciated..Am i missing any configuration

  • #2
    You'll need to either create an ItemKeyGenerator to uniquely identify records, perhaps using the row number, or you'll need to buffer the input. JdbcCursorItemReader does this already, what ItemReader are you using?

    Comment


    • #3
      I use the JDBCCursorItemReader which I have extended like this

      Code:
      public class OrderReader extends JdbcCursorItemReader implements StepExecutionListener

      Comment


      • #4
        Why did you implement a StepExecutionListener?
        Last edited by lucasward; Jun 3rd, 2008, 10:06 AM. Reason: changed extend to implement

        Comment


        • #5
          The reason being I override the beforeStep method to set the sql dynamically...As you have mentioned before usage of the preparedstatementsetter wouldnt work for me..Any problems due to this ?

          heres the code, with the comment for the dao

          Code:
          	public void beforeStep(StepExecution stepExecution) {
          		try{
          			log.debug("Invoking beforeStep in the Order reader to set the sql");
          			String sql = flatFileDao.getSql("KARVY"); // retrieve the sal
          			super.setSql(sql);
          		}catch(Exception ex){
          			log.warn("Error in execution of beforeStep of the output reader",ex);
          			throw new WMException("BATCH-0002",ex,null);
          		}
          	}

          Comment


          • #6
            Because the JdbcCursorItemReader buffers input, you should be able to use the comparison by reference, but you might want to double check. I would create a quick test ItemKeyGenerator and debug through it to see if the comparison is working.

            Comment


            • #7
              Is there any sample for ItemKeyGenerator ?

              Comment


              • #8
                It's just one method getKey(Object item). The default behaviour just returns the item itself. This is fine if Object.equals() has been overriden correctly. You should be able to simply debug through it to see if you're getting back an item with the same memory location.

                You set it on the factorybean. There's a simple setter for it.

                Comment


                • #9
                  Yes Lucas,

                  You were right. I implemented the ItemKeyGenerator and injected it into the Factory Bean.....It returns me the same instance for 2 and 4

                  Is this because I am returning an Object array from the mapRow() method. I am pretty sure that if I use a proper data object with equals() and hashcode(), it will work fine...but I do need to use an object array to keep things generic

                  Any ideas?

                  Tks..Vijay

                  Comment


                  • #10
                    Couldn't you create a wrapper object that contains a map?

                    Comment


                    • #11
                      Duh me..Thanks lucas..tat was pretty sraightforward to see

                      So it works fine now...skipping the required items and adding it to the file...


                      I got a delegate writer extends the StepListenerSupport as I need to populate the headers and footers for the output file.
                      At any error during itemRead or itemWrite I need to log into a table. Since my writer extends the StepListenerSupport shouldnt the methods onWriteError(Exception ex,Object item) be invoked if I have overriden it ? It doesnt seem to be invoking it. I overrode the other two methods too beforeWrite() and afterWrite() but to no avail .

                      Also, during my processing of records when I get an exception and add it as a skipped record, the process starts right from the beginning of the chunk and skips the record for processing in the subsequent chunk ? Is this a normal behaviour ?

                      Thanks again for all your patient replies.

                      Rgds...Vijay
                      Last edited by vijaynairis; Jun 5th, 2008, 12:45 AM.

                      Comment


                      • #12
                        Originally posted by vijaynairis View Post
                        I got a delegate writer extends the StepListenerSupport as I need to populate the headers and footers for the output file.
                        At any error during itemRead or itemWrite I need to log into a table. Since my writer extends the StepListenerSupport shouldnt the methods onWriteError(Exception ex,Object item) be invoked if I have overriden it ? It doesnt seem to be invoking it. I overrode the other two methods too beforeWrite() and afterWrite() but to no avail .
                        StepListenerSupport is only for a StepExecutionListener, before and after write are on the ItemWriteListener class, you'll need to implement that as well. Just having methods with those names won't work unless you've implemented the interface so that the step can pick it up.

                        Also, during my processing of records when I get an exception and add it as a skipped record, the process starts right from the beginning of the chunk and skips the record for processing in the subsequent chunk ? Is this a normal behaviour ?
                        That depends on where you're getting the exception from. Any exception thrown from an ItemWriter will cause the transaction to be rolled back, regardless of whether or not it calls a skip. (1.1 contains a new feature to let you control this though) If you throw an exception from the ItemReader however, there will be no rollback.

                        Comment


                        • #13
                          Hi,

                          Here's my class definition

                          Code:
                          public class OrderWriter implements  ItemWriter,ItemWriteListener,StepExecutionListener
                          but it still doesnt invoke any method other than the step listener methods

                          Thanks...Vijay

                          Comment


                          • #14
                            Did you set the listeners property in the step factory bean (I couldn't see that in you XML you posted earlier)?

                            Comment


                            • #15
                              Hi Dave,

                              Thanks for the solution. It has worked but has confused me to a significant extent.

                              While I did not have to register the StepListener which I have for my Reader and Writer, why would I have to do this for the other two listeners ?

                              Code:
                              public class OrderReader extends JdbcCursorItemReader implements StepExecutionListener
                              
                              public class OrderWriter implements  ItemWriter,StepExecutionListener
                              For both of these, the step listener methods do get invoked without me having to seperate into a separate listener and getting it registered.

                              Hope I am making sense

                              Tks..Vijay

                              Comment

                              Working...
                              X