Announcement Announcement Module
No announcement yet.
Webflow upgrade from 2.0.8 to 2.3.0 - 6% CPU Increase On Websphere Page Title Module
Move Remove Collapse
Conversation Detail Module
  • Filter
  • Time
  • Show
Clear All
new posts

  • Webflow upgrade from 2.0.8 to 2.3.0 - 6% CPU Increase On Websphere

    After upgrading from spring-webflow-2.0.8/spring-core-2.5.6 to spring-webflow-2.3.0/spring-core-3.0.5 we noticed a CPU increase or ~6% when running load tests. The CPU increase was a result of many short lived objects being created and GC'd.

    We would see the heap fill to from 500M to 2G and then GC'd to 500M over the course of about 2 minutes. The heap dump showed 6 different strings in memory 3 million times each along with 6 different method in memory 100,000 times each (the combined shallow heap accounted for 600M). Since the heap fills up so frequently, the GC obviously has to run more frequently and as a side effect we see a 6% increase in overall CPU use.

    The root cause for this new performance impact is a result of three things.
    1. We are running on Websphere 7 (but not limited to 7)
    2. Spring webflow 2.3.0/Spring 3.0.5 now evaluates annotation equivalence when binding form fields to the backing object
    3. Each field of our form backing object is annotated with @XmlElement

    Spring-webflow/spring-core, when binding a form field to its backing object, will now (with the fields type) ensure the annotations are equivalent - an equals check will now delegate to the underlying InvocationHandler(annotations are returned as proxys) offered by the running JVM. IBM's InvocationHandler$AnnotationInv ocationHandler seems to always invoke the annotation's getDeclaredMethods(). Note: Each time Class.getDeclaredMethods() is invoked, a new array of methods will be created and stored in memory.

    Our application will annotate each field for use in marshaling for web services. Each field is annotated with @XmlElement. Every time a field is bound to, an equivalence check is done on the backing objects annotations which will delegate to IBMs AnnotationInvocationHandler and will at the point create 6 new methods each time. As a result we saw 18 million strings and 600,000 methods all related to the XmlElement annotation.

    Here is a stack trace of what I am referring to

    WebContainer : 14
      at java.lang.Class.getDeclaredMethodsImpl()[Ljava/lang/reflect/Method; (Native Method)
      at java.lang.Class.getDeclaredMethods()[Ljava/lang/reflect/Method; ( Code))
      at$[Ljava/lang/Object;)Z ( Code))
      at$AnnotationInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; ( Code))
      at $Proxy108.equals(Ljava/lang/Object;)Z (null(Compiled Code))
      at java.util.Arrays.equals([Ljava/lang/Object;[Ljava/lang/Object;)Z ( Code))
      at org.springframework.util.ObjectUtils.nullSafeEquals(Ljava/lang/Object;Ljava/lang/Object;)Z ( Code))
      at org.springframework.core.convert.TypeDescriptor.equals(Ljava/lang/Object;)Z ( Code))
      at$ConverterCacheKey.equals(Ljava/lang/Object;)Z ( Code))
      at java.util.concurrent.ConcurrentHashMap$Segment.get(Ljava/lang/Object;I)Ljava/lang/Object; ( Code))
      at java.util.concurrent.ConcurrentHashMap.get(Ljava/lang/Object;)Ljava/lang/Object; ( Code))
      at;Lorg/springframework/core/convert/TypeDescriptor;)Lorg/springframework/core/convert/converter/GenericConverter; ( Code))
      at;Lorg/springframework/core/convert/TypeDescriptor;)Z ( Code))
      at org.springframework.webflow.mvc.view.ConvertingPropertyEditorAdapter.<init>(Lorg/springframework/binding/convert/ConversionService;Ljava/lang/String;Lorg/springframework/core/convert/TypeDescriptor;)V ( Code))
      at org.springframework.webflow.mvc.view.BindingModel.findSpringConvertingPropertyEditor(Ljava/lang/String;Ljava/lang/Class;)Ljava/beans/PropertyEditor; ( Code))
      at org.springframework.webflow.mvc.view.BindingModel.findEditor(Ljava/lang/String;Ljava/lang/Class;)Ljava/beans/PropertyEditor; ( Code))
      at;)Ljava/beans/PropertyEditor; ( Code))
      at org.springframework.web.servlet.tags.form.OptionWriter.getDisplayString(Ljava/lang/Object;)Ljava/lang/String; ( Code))
      at org.springframework.web.servlet.tags.form.OptionWriter.renderOption(Lorg/springframework/web/servlet/tags/form/TagWriter;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V ( Code))
      at org.springframework.web.servlet.tags.form.OptionWriter.doRenderFromCollection(Ljava/util/Collection;Lorg/springframework/web/servlet/tags/form/TagWriter;)V ( Code))
      at org.springframework.web.servlet.tags.form.OptionWriter.renderFromCollection(Lorg/springframework/web/servlet/tags/form/TagWriter;)V ( Code))
      at org.springframework.web.servlet.tags.form.OptionWriter.writeOptions(Lorg/springframework/web/servlet/tags/form/TagWriter;)V ( Code))
      at org.springframework.web.servlet.tags.form.OptionsTag.writeTagContent(Lorg/springframework/web/servlet/tags/form/TagWriter;)I ( Code))
      at org.springframework.web.servlet.tags.form.AbstractFormTag.doStartTagInternal()I ( Code))
      at org.springframework.web.servlet.tags.RequestContextAwareTag.doStartTag()I ( Code))
    I looked into the source of Java's packaged InvocationHandler and they seem to be caching the methods after first invocation - my next step is to look into the IBM InvocationHandler and see if that too is occurring.

    I hacked into the ObjectUtils.class to prevent the comparison if the annotation type is XmlElement - and sure enough the heap and CPU utilization went back to normal (not as a resolution but as a proof of concept).

    So I am wondering if anyone has seen this type of behavior before. Or if there is a way I can circumvent the specific annotation comparisons. We have reached out to IBM as well, just looking for insight on both ends.

    Any suggestions or questions are welcome!

    Last edited by johnvint; Mar 6th, 2012, 09:19 AM.