Announcement Announcement Module
Collapse
No announcement yet.
java heap problem in spring batch OutOfMemoryException ) Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • java heap problem in spring batch OutOfMemoryException )

    Hi,
    I am new to spring batch , iam facing problem with heap memory

    requirement: i have flat file (500 MB (1 million lines)) , i need to write this file data into gemfire catch, and i set java max heap size 1GB .


    Even i am geting OutOfMemoryException while i am procesing

    1) how max heap size it requires to process 1 GB flat file( it may increase)
    2) is it require flush or clean once wrien into cache ,if it how?
    3)is it any problem with m code below.

    kindly let know ans.




    code:
    ======
    <gfe:cache id="cacheWithXml" />
    <batch:job id="toCacheFile">

    <batch:step id="step1_1">
    <batch:tasklet transaction-manager="jobRepository-transactionManager">
    <batch:chunk reader="itemReader1" writer="DBToCache"
    commit-interval="1"/>
    </batch:tasklet>
    </batch:step>

    </batch:job>


    <bean id="DBToCache" class="com.FeedToCache">
    <property name="cache" ref="cacheWithXml" />
    </bean>

    cache.xml
    =======

    <cache>
    <disk-store name="ds" auto-compact="true" max-oplog-size="1024" queue-size="10000" time-interval="10">
    <disk-dirs>
    <disk-dir >c:/filecache</disk-dir>
    </disk-dirs>
    </disk-store>

    <region name="cacheRegion">
    <region-attributes data-policy="persistent-replicate" scope="local"
    refid="REPLICATE_PERSISTENT" statistics-enabled="true" disk-store-name="ds" disk-synchronous="false" >


    </region-attributes>
    </region>
    </cache>

    write metthod for cachig data(FeedToCache.java)
    ======================
    public void write(List<? extends FeedData> items) throws Exception {
    for (FeedData item : items) {
    if (item!= null)
    if(item.getAccountNumber()!=null)
    cacheRegion.put(item.getAccountNumber()+Math.rando m(),item);

    }

    }


    Thanks
    Anil.

  • #2
    Please use [ code][/code ] tags when posting code, that way it remains readable ...

    Post your configuration of the reader (maybe some code) and post the full stacktrace (remember use [ code][/code ] tags!!!!)

    Comment


    • #3
      The simple way to see how much memory your program needs to run.. is give it way more memory, watch how big your heap grows, then decide if you you can reduce memory usage

      Run with a 64-bit JVM, and set your XMX to 16 GB.... watch your heap usage with jVisualVM... and see how high you get. Then decide if its cheaper to just leave XMX there, or work on using less memory.

      Comment


      • #4
        my reader code:
        =================

        <bean id="itemReader1" class="org.springframework.batch.item.file.FlatFil eItemReader">
        <property name="resource" value="file:///C:/anil.FEED3" />
        <property name="encoding" value="ISO-8859-1" />
        <property name="lineMapper">
        <bean class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
        <property name="lineTokenizer" ref="accountTokenizer">
        </property>
        <property name="fieldSetMapper" >
        <bean class="com.FeedDataFieldSetMapper" />
        <property>
        </bean>
        </property>
        </bean>

        error log
        ========
        SystemFailure Watchdog: [warning] failure detected
        java.lang.OutOfMemoryError: Java heap space
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
        at java.lang.Class.getDeclaredMethod(Unknown Source)

        [error 2011/04/29 11:44:13.845 IST <UDP mcast receiver> tid=0x15] ThreadGroup:
        Uncaught Exception in thread UDP mcast receiver
        java.lang.OutOfMemoryError: Java heap space
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
        at java.lang.Class.getDeclaredMethod(Unknown Source)
        at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
        at java.io.ObjectStreamClass.access$1700(Unknown Source)
        at java.io.ObjectStreamClass$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(Unknown Source)
        at java.io.ObjectStreamClass.lookup(Unknown Source)
        at java.io.ObjectOutputStream.writeObject0(Unknown Source)
        at java.io.ObjectOutputStream.writeObject(Unknown Source)
        at com.gemstone.gemfire.internal.InternalDataSerializ er.basicWriteObject
        (InternalDataSerializer.java:1552)
        at com.gemstone.gemfire.DataSerializer.writeObject(Da taSerializer.java:3
        025)
        at com.gemstone.gemfire.internal.util.BlobHelper.seri alizeToBlob(BlobHel
        per.java:39)
        at com.gemstone.gemfire.internal.cache.EntryEventImpl .serialize(EntryEve
        ntImpl.java:1282)
        at com.gemstone.gemfire.internal.cache.DiskEntry$Help er.writeToDisk(Disk
        Entry.java:450)
        at com.gemstone.gemfire.internal.cache.DiskEntry$Help er.doAsyncFlush(Dis
        kEntry.java:1050)
        at com.gemstone.gemfire.internal.cache.DiskStoreImpl$ FlusherThread.run(D
        iskStoreImpl.java:1661)
        at java.lang.Thread.run(Unknown Source)
        at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
        at java.io.ObjectStreamClass.access$1700(Unknown Source)
        at java.io.ObjectStreamClass$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(Unknown Source)
        at java.io.ObjectStreamClass.lookup(Unknown Source)
        at java.io.ObjectOutputStream.writeObject0(Unknown Source)
        at java.io.ObjectOutputStream.writeObject(Unknown Source)
        at com.gemstone.gemfire.internal.InternalDataSerializ er.basicWriteObject
        (InternalDataSerializer.java:1552)
        at com.gemstone.gemfire.DataSerializer.writeObject(Da taSerializer.java:3
        025)
        at com.gemstone.gemfire.internal.util.BlobHelper.seri alizeToBlob(BlobHel
        per.java:39)
        at com.gemstone.gemfire.internal.cache.EntryEventImpl .serialize(EntryEve
        ntImpl.java:1282)
        420393 [main] ERROR org.springframework.batch.core.step.AbstractStep - Encounte
        red an error executing the step: class java.lang.OutOfMemoryError: Java heap space




        Here i am sending reader code and error log what i am getting . kindly check and respond me.

        Comment


        • #5
          Please as stated use [ code][/code ] tags so that we can actually (easily) read it...

          However judging from the stacktrace it is a gemfire issue. I'm not that familiar with gemfire so I cannot help you there. But it is either a bug or some misconfiguration on your side.

          Comment


          • #6
            Same issue i got for flat file writing . and even i tried with writing into queue also But i am facing same problem.

            Comment


            • #7
              Then it has to do with either your configuration, environment, fieldsetmapper... You are keeping objects somewhere (at least that is what it then looks like).

              Comment

              Working...
              X