Announcement Announcement Module
Collapse
No announcement yet.
Memory consumption in ApplicationContext 3.0.0 Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Memory consumption in ApplicationContext 3.0.0

    Hi,

    currently I'm trying to migrate my application from 2.5.6 to 3.0.0. Surprisingly I can see a significant higher memory consumption in 3.0.0. The DefaultListableBeanFactory uses 8.5 mb in spring 2.5.6 and about 90 mb in spring 3.0.0. As far as I can see, a large part of the memory consumption is due to the usage of CachingMetadataReaderFactory.

    Is it possible to SimpleMetadataReaderFactory instead of CachingMetadataReaderFactory and what are the performance consequences?

    Thanks in advance

    Uwe

  • #2
    Hmm I would consider this a bug, could you register a JIRA with a small test case to prove the issue (shouldn't be that hard).

    I'm not sure on how to switch to the SimpleMetadataReaderFactory, I suspect some dynamic discovery on which to use, not sure how to influence that.

    Comment


    • #3
      I've raised https://jira.springsource.org/browse/SPR-6775 We'll try to reproduce the problem but it would help to get some info or memory information about your application.

      Comment


      • #4
        Guys, I've committed a fix for this in the trunk, improving the internal cache of the reader with a limited, LRU.
        Ura, it would be great if you could check one of the nightly builds. Out of curiosity, how big is your application and how many classes do you scan? From my tests, 90 MB is roughly equivalent to 4k classes being scanned (give or take), let's 3K which is still a lot.

        Comment


        • #5
          Costin, I'll try the nightly build as soon as possible...

          XmlBeanDefinitionReader says:

          Code:
          Loaded 1257 bean definitions from location pattern [classpath*:META-INF/*-beans.xml]
          Loaded 72 bean definitions from location pattern [classpath*:META-INF/spring/*.xml]
          Loaded 3 bean definitions from location pattern [file:C:\Reposit.../database.xml]
          Loaded 1 bean definitions from location pattern [file:C:\Reposit.../*-beans.xml]
          ClassPathBeanDefinitionScanner says:

          Code:
          Scanning URL [jar:file:/C:/Reposit...]  2130 times
          The scanning for most of the classes is done by a component-scan element like this:

          Code:
          <context:component-scan base-package="de.varial.accounting.persistency,ce.varial.accounting.persistency,com.varial.accounting.persistency" resource-pattern="**/*Dao.class" />
          Most of the classes scanned have a structure like the example below:

          Code:
          @Repository(value="xyzDao")
          public final class XyzDao implements RowMapper<XyzContainer>, IXyzDao {
          
          	@Autowired
          	private final PrimaryKeyIncrementer incrementer = null;
          	
          	@Autowired
          	@Qualifier("dataSource")
          	private final DataSource dataSource = null; 
          
          	@Autowired
          	private final SQLExceptionTranslatorFactory translator = null;
          
          	@Autowired
          	private final StatementOptimizer optimizer = null;
          		
          	public XyzDao() {
          		..
          	}
          
          	@Override
              public  XyzContainer getInitializedInstance() {
          		..
              }
          
          	@Override
          	public  void getRows(A, B, C) {
          		..
          	}
          
          	@Override
          	public  void getRows(A, B, C, D) {
          		...
          	}
          
          	private boolean putData(A, B) {
          		...
          	}
          
          	@Override
          	public  XyzContainer insert(A, B) {
          		...
          	}
          
          	@Override
          	public  XyzContainer update(A, B) {
          		...
          	}
          
          	@Deprecated
          	@Override
          	public  XyzContainer updateIgnoreVersion(A, B) {
          		...
          	}
          
          	@Override
          	public  void updateAttributes(A, B) {
          		...
          	}
          
          	@Override
          	public  int updateAttributes(A, B, C) {
          		...
          	}
          
          	@Override
          	public void delete(A, B) {
          		...
          	}
          
          	@Override
          	public void delete(A, B, C) {
          		...
          	}
          
          	@Override
          	public int delete(A, B) {
          		...
          	}
          
          	private  PreparedStatement fillUpdateParameter(A, B) {
          		...
          	}
          
          	private  PreparedStatement fillInsertParameter(A, B) {
          		...
          	}
              
          	private  List<String> getContainerAsStrings(A, B) {
          		...
          	}
          
          	@Override
          	public  XyzContainer getRowByObjectID(A) {
          		...
          	}
          
          	@Override
          	public  XyzContainer[] getRows(A, B) {
          		...
          	}
          
          	private  XyzContainer makeContainer(A) {
          		...
          	}
          
          	@Override
          	public  long getRowCount(A) {
          		...
          	}
          
          	@Override
          	public  PersistencyDataSet getDataSet(A, B) {
          		...
          	}
          
          	@Override
          	public XyzContainer mapRow(A, B) {
          		...
          	}
          
          	private SQLExceptionTranslator getExceptionTranslator() {
          		...
          	}
          }

          Comment


          • #6
            Just an additional information...

            I've attached 2 ZIP files containing GIFs. The file Memory-2.5.6.gif shows the memory consumption using spring 2.5.6, the file Memory-3.0.0.gif shows the same information using spring 3.0.0. I'll do the same for the nightly build, mentioned in the posts before, alter on.

            Comment


            • #7
              You have a pretty big application - around 1500 definitions (let's assume one class per definition) and the scanner introspects over 2K classes so all in all, we're well over 3K classes analyzed which adds up to the figures I've seen from my tests.

              P.S. the latest snapshot (562) has been already published ~12h ago so it should be ready for download

              Comment


              • #8
                Costin, I did the test with 3.0.1.CI-562. The results are not good but much better compared to 3.0.0.RELEASE. In 3.0.0.RELEASE the memory consumption of CachingMetadataReaderFactory was about 81 mb, in 3.0.1.CI-562 the consumption is reduced to 18 mb (see the attached jpg).

                Comment


                • #9
                  Hmm - It should be less then that. I'm using YourKit and the integration test (which goes through 10k classes) doesn't even show on the profiler anymore and the memory usage never goes beyond 8MB. I'll try Eclipse MAT and maybe a more complicated class - maybe your classes are bigger then the one I use which might explain the difference in memory size. We currently use 256 entries so we could lower this number to 128 which should reduce the size to half of what it's now.

                  Comment


                  • #10
                    Out of curiosity, could you check the average size of your classes in actual bytes on the file system - I think this might be the most important aspect.

                    Comment


                    • #11
                      OK, here are some numbers in bytes..

                      Size of bytecode: arithmetic mean 25842, min 11920, max 61478, standard deviation 4935

                      Comment


                      • #12
                        Now I had a deeper look into the code...

                        As far as I can see the obfuscated class ...asm.ClassReader, which is indirectly referenced from the cache of CachingMetadataReaderFactgory, holds the bytecode of a bean class and an array of strings. This array holds, among other stuff, stringified method signatures defined in or called from the bean class.

                        For my application, I found this string array mostly larger then the bytecode array (see attachment) and short snippets below.

                        Code:
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;Lcom/varial/base/persistency/callback/IConsumer<Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;>;I)V
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;Lcom/varial/base/persistency/callback/IConsumer<Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;>;)V
                        (Ljava/util/List<Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;>;Lcom/varial/base/persistency/callback/IConsumer<Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;>;)Z
                        Ljava/lang/Object;Lorg/springframework/jdbc/core/RowMapper<Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;>;Lcom/varial/migrationprocess/persistency/migpwwag/IMigpwwagDao;
                        (Ljava/sql/Connection;Ljava/lang/String;[Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;I)Ljava/sql/PreparedStatement;
                        (Ljava/sql/Connection;Ljava/lang/String;[Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;)Ljava/sql/PreparedStatement;
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;)[Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;
                        (Ljava/lang/String;Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;)Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;Lcom/varial/base/persistency/callback/IConsumer;I)V
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;)Lcom/varial/base/persistency/PersistencyDataSet;
                        ([Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;Lcom/varial/base/persistency/callback/IConsumer;)V
                        (Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;[Ljava/lang/String;Lcom/varial/base/persistency/criterion/Criterion;)I
                        (Ljava/lang/String;[Lcom/varial/base/persistency/criterion/Order;Lcom/varial/base/persistency/criterion/Criterion;)Ljava/lang/String;
                        (Ljava/sql/PreparedStatement;Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;)Ljava/sql/PreparedStatement;
                        (Ljava/sql/Connection;Ljava/sql/ResultSet;Ljava/sql/PreparedStatement;Lorg/springframework/jdbc/core/RowMapper;)V
                        (Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;Z)Ljava/util/List<Ljava/lang/String;>;
                        (Ljava/lang/String;Ljava/lang/String;Ljava/sql/SQLException;)Lorg/springframework/dao/DataAccessException;
                        (Ljava/sql/ResultSet;I)Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;
                        (Ljava/lang/String;Lcom/varial/base/persistency/criterion/Criterion;)Ljava/lang/String;
                        (Ljava/sql/ResultSet;)Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;
                        (Ljava/lang/String;)Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;
                        (Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;Z)Ljava/util/List;
                        (Ljava/lang/String;Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;)V
                        (Lcom/varial/migrationprocess/persistency/migpwwag/MigpwwagContainer;[Ljava/lang/String;)V
                        (Ljavax/sql/DataSource;)Lorg/springframework/jdbc/support/SQLExceptionTranslator;

                        Comment


                        • #13
                          Hi ura,

                          Thanks for looking into this. I've fixed this problem already however I'd like to minimize the impact of the cache even further (not sure if that's possible but I'm looking into it).
                          For example, if the classes have a big number of methods (40) the amount of metadata can add up so I'm trying to compact any wasted space.

                          If compiling the trunk is an option, I can commit what I have right now assuming you can then update it on your side. Otherwise, I'll spend some more time and then fire up a nightly build to publish a new snapshot for you to try.

                          How does it sound?

                          Comment


                          • #14
                            Hi Costin,

                            firing up a nightly build after finishing your changes will be perfectly working for me. Pls give me a short message after doing the nightly build. I'll try the shapshot afterwards.

                            Thanks

                            Uwe

                            Comment


                            • #15
                              Uwe, I've improved the way information is stored inside the cache which should be overall, a lot more efficient now. When dealing with a LOT (hundreds) of huge classes (with plenty of annotated methods (more then 20-30)), the cache becomes big but in your case, it should drop to 20% of what it is right now, maybe more.
                              Let me know how it works for you - we might consider introducing a time based policy so that the cache expires in time to improve memory for long running applications.
                              If that doesn't work, then probably using a dedicated MetadataReader or the simple one is better though if the latter is used there would be _significant_ IO activity.

                              Anyway, build 563 has been started (http://build.springframework.org/bro...-TRUNKSNAPSHOT) - it should be ready in about 90 minutes from now.

                              Looking forward to your feedback!

                              Comment

                              Working...
                              X