Announcement Announcement Module
Collapse
No announcement yet.
Processing flat-file in groups Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Processing flat-file in groups

    Hello, imagine the following situation:

    I have a flat-file with lot of transactions (one per line, in CVS format) ordered by city and province where they have been done.

    I need to process the input file and generate an output file with the amount of transactions per city and the amount of transactions per province.

    I want to read all the transactions in one city and THEN write a line in the output file.
    -------------------------------------------------------------
    Has spring batch a way to process a group of Items? Is my question clear?

    Thanks
    Neuquino

  • #2
    I may not well understand your needs but with not implementing your own ItemWriter and develop your custom write() method.

    If your input file is ordered is simple again. Just persist the current city name in some attribute of your writer and increase a counter if you need it. Then, when the current city is different from previous write a line in the target file.

    Comment


    • #3
      What you're trying to do can be categorized as multiple lines per item. What I mean is that multiple lines in your file make up an item, which is fairly common. You just need to aggregate the lines, similar to what the multiline sample job does. You keep reading until you have an entire city made up, then return that as the item from your ItemReader.

      Comment


      • #4
        OK, but how about single line input to multi-line output?

        I'm having trouble finding support for multi-line, flat file output for a single bean. For example for the following simple class:

        Code:
        public class Partner {
        	private long id;
        	private long mediaCode;
        	private Name name;
        	private Address address;
        
        	public Address getAddress() {
        		return address;
        	}
        	public void setAddress(Address address) {
        		this.address = address;
        	}
        	public long getId() {
        		return id;
        	}
        	public void setId(long id) {
        		this.id = id;
        	}
        	public long getMediaCode() {
        		return mediaCode;
        	}
        	public void setMediaCode(long mediaCode) {
        		this.mediaCode = mediaCode;
        	}
        	public Name getName() {
        		return name;
        	}
        	public void setName(Name name) {
        		this.name = name;
        	}
        	
        	public void setName(String firstName, String lastName) {
        		this.name = new Name(firstName, lastName);
        	}
        	
        	public void setAddress(String city, String state) {
        		this.address = new Address(city, state);
        	}
        
        	private static class Name {
        		private String firstName;
        		private String lastName;
        
        		private Name(String firstName, String lastName) {
        			this.firstName = firstName;
        			this.lastName = lastName;
        		}
        		public String getFirstName() {
        			return firstName;
        		}
        		public String getLastName() {
        			return lastName;
        		}
        	}
        	
        	private static class Address {
        		private String city;
        		private String state;
        
        		private Address(String city, String state) {
        			this.city = city;
        			this.state = state;
        		}
        		public String getCity() {
        			return city;
        		}
        		public String getState() {
        			return state;
        		}
        	}
        }
        How do I get a three output lines in a flat file - one for Partner id and mediaCode, one for the Name properties, and one for the Address properties? For example, output that looks like this:

        Code:
        PAR;21;7
        NAM;John;Doe
        ADR;Madison;WI
        Any help is appreciated.

        Comment


        • #5
          In the samples project, there is a job called multiline.xml as part of the iosample. This sample demonstrates how to read a multiline file and also how to write one. The MultiLineTradeItemWriter does the writing:

          Code:
          	private FlatFileItemWriter<String> delegate;
          
          	public void write(List<? extends Trade> items) throws Exception {
          		List<String> lines = new ArrayList<String>();
          		for (Trade t : items) {
          			lines.add("BEGIN");
          			lines.add("INFO," + t.getIsin() + "," + t.getCustomer());
          			lines.add("AMNT," + t.getQuantity() + "," + t.getPrice());
          			lines.add("END");
          		}
          		this.delegate.write(lines);
          	}

          Comment


          • #6
            Thank you for that incredibly fast response! On a Friday afternoon, too!

            I see the code now. I didn't expect that to be in "src/test/java" - I would expect it to be in "src/main/java".

            Just as an FYI, you can also use the FormatterLineAggregator and put "%n" conversions in the string, e.g.:

            Code:
            <property name="lineAggregator">
              <bean class ="org.springframework.batch.item.file.transform.FormatterLineAggregator">
            				<property name="fieldExtractor">
            					<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
            						<property name="names" value="id,mediaCode,name.firstName,name.lastName,address.city,address.state" />
            					</bean>
            				</property>
            				<property name="format" value="PAR%1$10d%2$10d%nNAM%3$30s%4$100s%nADR%5$70s%6$15s"></property>
            			</bean>
            But that's clearly not as flexible (or readable) as what you've shown.

            Comment

            Working...
            X