Announcement Announcement Module
Collapse
No announcement yet.
Parsing Java log files Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Parsing Java log files

    We're migrating a batch process to Spring Batch 2. One of the steps parses several Java log files from different servers and extract stack trace block for identified exceptions.

    We're trying to use multiline pattern without success. Is there any suitable approach in order to achieve this behaviour?

    Thanks in advance.

  • #2
    You could create a wrapper that watches for the end of the stack trace. The delegate FFIR would use a PassThroughLineMapper.
    Code:
    public class LogItemReader implements ItemReader<List<String>> {
    	private FlatFileItemReader<String> delegate; // setter omitted
    	private String last = null;
    	private boolean done = false;
    
    	public List<String> read() throws Exception {
    		if (done) { // last item already returned
    			return null;
    		}
    
    		List<String> item = new ArrayList<String>();
    		if (last != null) {
    			item.add(last); // first line of item already read
    		}
    
    		for (String line = null; (line = this.delegate.read()) != null;) {
    			if (line.startsWith(" ") || item.isEmpty()) { // part same item
    				item.add(line);
    			}
    			else { // start of next item
    				last = line; // store first line of next item
    				return item;
    			}
    		}
    
    		// no more lines
    		done = true;
    		return item;
    	}

    Comment


    • #3
      It almost sounds like you need a regex reader or something similar, which wasn't what the FlatFilteItemReader was designed to do. Even the traditional 'multi-line' approach was really there to support lines with header - detail - footer, not the kind of parsing you're looking to do. I think you'll have to write a custom reader.

      Comment


      • #4
        Success

        Finally, we've implemented following schema.

        Code:
        <beans:bean id="multifileASReader" class="org.springframework.batch.item.file.MultiResourceItemReader" lazy-init="true">
            <beans:property name="resources" value="file://${loc.working.directory.appserver}/${appserver.file.1}.*" />
            <beans:property name="delegate" ref="proxyFlatFileASItemReader" />
        </beans:bean>
        
        <beans:bean id="proxyFlatFileASItemReader" class="foo.bar.LogItemReader">
            <beans:property name="delegate" ref="flatFileASItemReader" />
        </beans:bean>
        
        <beans:bean id="flatFileASItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
          <beans:property name="lineMapper">
            <beans:bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper" />
          </beans:property>
        </beans:bean>
        Code:
        public class LogItemReader 
            implements ItemReader<List<String>>, 
                       ResourceAwareItemReaderItemStream<List<String>> {
          
          public List<String> read() throws Exception, UnexpectedInputException, ParseException {
          
              // Logic...
        
          }
        
          public void setDelegate(FlatFileItemReader<String> delegate) {
            this.delegate = delegate;
          }
        
          public void setResource(Resource resource) {
            delegate.setResource(resource);
          }
        
          public void close() throws ItemStreamException {
            delegate.close();
          }
        
          public void open(ExecutionContext executionContext) throws ItemStreamException {
            delegate.open(executionContext);
          }
        
          public void update(ExecutionContext executionContext) throws ItemStreamException {
            delegate.update(executionContext);
          }
          
        }
        Maybe it's inelegant, but it works fine.

        Thanks both.

        Comment


        • #5
          Have you tested this in a restart scenario? I think it might be broken because you are doing a look ahead into the delegate, so when the framework calls update() it will sometimes (often) be in the wrong place.

          Comment


          • #6
            Dave, you're right.

            In order to allow restarts, update method must be modified. It's easy with our requirements, because we're always reading the whole file, but it could be dangerous in other scenarios.

            Code:
              public void update(ExecutionContext executionContext) throws ItemStreamException {
                executionContext.putInt(ecSupport.getKey(READ_COUNT), 0);
                delegate.update(executionContext);
              }

            Comment


            • #7
              You can either:

              a) set saveState=false to ensure that the reader always starts at the beginning, or
              b) save the values of "done" and "last" in update() and restore them in open().

              Comment

              Working...
              X