Announcement Announcement Module
Collapse
No announcement yet.
Paged Results Collection Failing Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Paged Results Collection Failing

    Hi,
    I am trying to collect paged results (eventually ~300,000) from an LDAP server, which will then be stored in a database. The search runs fine a number of times but around the 15th time (this varies), the LDAP server (ITDS 6.0) returns LDAP: error code 53 - Unwilling To Perform.
    Delving further by setting "-Dcom.sun.jndi.ldap.connect.pool.debug=all" I can see the message:
    weak reference cleanup: Closing Connections:com.sun.jndi.ldap.pool.Connections<at> 7f727f72
    which seems to suggest that my connection (and my cookie?) has expired.
    In the debug from the previous search to this I see:
    com.sun.jndi.ldap.pool.Pool<at>3a2e3a2e {}.get(): ~address~:~port~:::cn=root
    where the debug from the previous pages had information between the braces.

    Looking in the LDAP log I can see an unbind operation happening before the request that fails.

    My code looks like this:
    public HashMap getItems(DataMapping dataMapping) {
    LDAPDataMapping ldapMap =(LDAPDataMapping)dataMapping;
    String searchBase = (String)ldapMap.getSearchBase();
    String searchString = (String)ldapMap.getSearchString();
    log.debug("Getting all items with searchstring:"+searchString+" and searchbase "+searchBase);
    List items = ldapTemplate.search(searchBase, searchString, this.searchControls, new ItemAttributesMapper(), this.requestControl);
    byte[] cookie = requestControl.getCookie().getCookie();
    HashMap itemsAndCookie = new HashMap();
    itemsAndCookie.put("items", items);
    itemsAndCookie.put("cookie", cookie);
    return itemsAndCookie;
    }
    private class ItemAttributesMapper implements ContextMapper {
    public Object mapFromContext(Object ctx) {
    HashMap itemattrs = new HashMap();
    String attName;
    DirContextAdapter context = (DirContextAdapter)ctx;
    Attributes attrs = context.getAttributes();
    Enumeration en = attrs.getIDs();
    itemattrs.put("dn", context.getNameInNamespace());
    while (en.hasMoreElements()){
    attName = (String)en.nextElement();
    itemattrs.put(attName.replaceAll("'", "\\'").toLowerCase(), LDAPItemDAO.getAttribute(attrs,attName.replaceAll( "'", "\\'")));
    }
    return new ItemVO(itemattrs);
    }
    }
    The rest of the code just calls getItems until the cookie is null
    Spring.xml:
    <bean id="contextSource"
    class="org.springframework.ldap.core.support.LdapC ontextSource">
    <property name="url" value="~ldapaddress~" />
    <property name="base" value="c=US" />
    <property name="userName" value="cn=root" />
    <property name="password" value="${testldappwd}" />
    <property name="pooled" value="true" />
    <property name="dirObjectFactory" value="org.springframework.ldap.core.support.Defau ltDirObjectFactory" />
    </bean>
    <bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate" >
    <constructor-arg ref="contextSource" />
    </bean>
    <bean id="ldapItem"
    class="com.something.DAO.LDAPItemDAO">
    <property name="ldapTemplate" ref="ldapTemplate" />
    </bean>

    Does anyone have any suggestions what this could be or how I might get round it? Is the connection getting cleaned up by the garbage collector because its only stored as a weak reference?
    Any pointers would be much appreciated.
    Many Thanks
    Dave

  • #2
    Have you tried without pooling?

    Comment


    • #3
      Hi. Thanks for helping.
      When I turn off pooling (<property name="pooled" value="false" />), the 'unwilling to perform operation' message is returned when attempting to get the second page. I get much further when using pooling, even when setting com.sun.jndi.ldap.connect.pool.maxsize=1

      Comment


      • #4
        Yes, it seems it's not enough to send a valid cookie; the actual connection has to be re-used as well. I get a similar behavior with OpenLDAP 2.3.27.

        We try really hard in Spring LDAP to get a stateless behavior, where you never worry about the underlying connection. We might need to rethink how we handle scenarios like this.

        Thanks for bringing this up.

        Comment


        • #5
          oh dear, sounds like there's not going to be an easy way to fix this. I was considering using a modified com.sun.jndi.ldap.pool that uses a normal HashMap rather than a weak one to store the connections (its a batch process so i'm not too fussed about the possible memory leak). From your experience do you think this might work? Do you have any other suggestions of how I can accomplish my task? I guess if I re-write it using 'traditional' ldap programming libraries I shouldn't get this problem?

          Comment


          • #6
            I can confirm that by modifying com.sun.jndi.ldap.pool.Pool.java to use a HashMap instead of a WeakHashMap fixes my problem. Obviously i'd prefer not to have to do this as it reduces portability but as this needs to be working asap I think thats the route i'm going to go down

            Comment


            • #7
              In a batch scenario you could use the pooling functionality shipped with Spring LDAP. If you use a PoolingContextSource with a pool size of 1 you should always get the same DirContext instance back. It's kind of a hack, but at least it's better than fiddling with the Map implementation in the built-in Java LDAP pooling.

              This doesn't solve the general problem however; using a pool with only one connection in a multi-user environment is probably not what you would want (then again, this pool could be local to the actual operations that require paged results).

              Comment


              • #8
                Unfortunately, using the PoolingContextSource to solve this requires some effort. First of all, the DelegatingLdapContext throws an exception when the request controls are being set:

                Code:
                public class DelegatingLdapContext extends DelegatingDirContext implements LdapContext {
                   ...
                   public void setRequestControls(Control[] requestControls) throws NamingException {
                      throw new UnsupportedOperationException(
                         "Cannot call setRequestControls on a pooled context");
                   }
                The immediate impulse is to subclass and override this method. I created a custom DelegatingLdapContext subclass that allows controls to be set:

                Code:
                public class CustomDelegatingLdapContext extends DelegatingLdapContext {
                
                   public CustomDelegatingLdapContext(KeyedObjectPool keyedObjectPool, LdapContext delegateLdapContext,
                         DirContextType dirContextType) {
                      super(keyedObjectPool, delegateLdapContext, dirContextType);
                   }
                
                   public void setRequestControls(Control[] requestControls) throws NamingException {
                      assertOpen();
                      getDelegateLdapContext().setRequestControls(requestControls);
                   }
                }
                However, the creation of these classes is currently hard-coded in PoolingContextSource:

                Code:
                   protected DirContext getContext(DirContextType dirContextType) {
                      ...
                      if (dirContext instanceof LdapContext) {
                         return new DelegatingLdapContext(this.keyedObjectPool, (LdapContext)dirContext, dirContextType);
                      }
                
                      return new DelegatingDirContext(this.keyedObjectPool, dirContext, dirContextType);
                   }
                The immediate impulse is, again, to subclass and override, but the problem is that keyedObjectPool is private and there is no getter. I was forced to make a change in PoolingContextSource. I chose to make keyedObjectPool protected. Then I could subclass PoolingContextSource and make sure that my CustomDelegatingLdapContext is created instead:

                Code:
                public class CustomPoolingContextSource extends PoolingContextSource {
                    protected DirContext getContext(DirContextType dirContextType) {
                        final DirContext dirContext;
                        try {
                            dirContext = (DirContext)this.keyedObjectPool.borrowObject(dirContextType);
                        }
                        catch (Exception e) {
                            throw new DataAccessResourceFailureException("Failed to borrow DirContext from pool.", e);
                        }
                        
                        if (dirContext instanceof LdapContext) {
                            return new CustomDelegatingLdapContext(this.keyedObjectPool, (LdapContext)dirContext, dirContextType);
                        }
                
                        return new DelegatingDirContext(this.keyedObjectPool, dirContext, dirContextType);
                    }
                }
                The next problem was that preProcess in AbstractRequestControlDirContextProcessor doesn't check for duplicate request controls; it happily adds a new instance of PagedResultsControl for every search. This is not allowed, as the following exception illustrates:

                Code:
                javax.naming.CommunicationException: 
                   [LDAP: error code 2 - paged results control specified multiple times]; 
                   remaining name '/'
                However, the new control must be added, since that contains the new cookie. The solution was to check for an existing control of the same class and replace it:

                Code:
                public abstract class AbstractRequestControlDirContextProcessor implements
                      DirContextProcessor {
                   private boolean replaceSameControlEnabled = true;
                   ...
                   public void preProcess(DirContext ctx) throws NamingException {
                      ...
                      Control[] newControls = new Control[requestControls.length + 1];
                      for (int i = 0; i < requestControls.length; i++) {
                         if (replaceSameControlEnabled && requestControls[i].getClass() == newControl.getClass()) {
                            log.debug("Replacing already existing control in context: " + newControl);
                            requestControls[i] = newControl;
                            ldapContext.setRequestControls(requestControls);
                            return;
                         }
                         newControls[i] = requestControls[i];
                      }
                Summary of changes:
                1. Make PoolingContextSource.keyedObjectPool protected.
                2. Replace existing control in AbstractRequestControlDirContextProcessor.preProce ss.
                3. Subclass DelegatingLdapContext and override setRequestControls.
                4. Subclass PoolingContextSource and override getContext.

                With these changes, it's possible to use a Spring LDAP pool of size 1 to avoid the somewhat random reclaim of the pooled context when using the JNDI pool. I've tested with 100,000 entries and various page sizes and it works fine.

                If Mattias has no objections, I'll check the first two changes in, so you can at least use a snapshot build instead of hacking the JNDI pool.

                Comment


                • #9
                  Thanks for such a quick response. Apparently the version with a modified Sun Pool still failed when running against large numbers of LDAP entries so it would be very helpful to be able to get a snapshot build if possible. In the meantime I have made the changes you mentioned and it works for me on smaller amounts of LDAP data. I will let you know how it goes after testing with larger volumes of data.
                  Again, many thanks for such a speedy fix

                  Comment


                  • #10
                    OK, the changes are now committed. You should see them in the next snapshot build. Let me know if it works.

                    Comment


                    • #11
                      Hi, one more thing with ldap pooling. For me fails further when i try to validate the connection, so no validator can be used, as the default validator tries a search against the context, when it expects basically the new page reuquest....Same for you?

                      And what happens if the connections should be built using different users? You have to add at least the username to the pool key. This means the DirContextKey should be changed and the key used to read the connection from the pool should be set in the thread local from the dao for example and reused in the DirContextKey.

                      The later one is working for me, the question is how can i validate the connection?
                      Any ideas?

                      Comment


                      • #12
                        I validate without problems:

                        Code:
                        <bean id="poolingContextSource" class="org.springframework.ldap.control.CustomPoolingContextSource">
                           <property name="contextSource" ref="contextSource" />
                           <property name="dirContextValidator">
                              <bean class="org.springframework.ldap.pool.validation.DefaultDirContextValidator" />
                           </property>
                           <property name="maxTotal" value="1" />
                           <property name="testOnBorrow" value="true"/>
                           <property name="testWhileIdle" value="true"/>
                        </bean>
                        Here's the debug log from a run with page size 300, mixed with the printout of the current list size (300, 600, ...). As you can see, the connection passes validation before each paged search:

                        Code:
                        DEBUG [...core.support.AbstractContextSource] - <Not using LDAP pooling>
                        DEBUG [...core.support.AbstractContextSource] - <Trying provider Urls: ldap://127.0.0.1/dc=jayway,dc=se>
                        DEBUG [...pool.factory.DirContextPoolableObjectFactory] - <Creating a new READ_ONLY DirContext>
                        DEBUG [...core.support.AbstractContextSource] - <Got Ldap context on server 'ldap://127.0.0.1/dc=jayway,dc=se'>
                        DEBUG [...pool.factory.DirContextPoolableObjectFactory] - <Created new READ_ONLY DirContext='javax.naming.ldap.InitialLdapContext@bc081f'>
                        DEBUG [...pool.validation.DefaultDirContextValidator] - <DirContext 'javax.naming.ldap.InitialLdapContext@bc081f' passed validation.>
                        300
                        DEBUG [...pool.validation.DefaultDirContextValidator] - <DirContext 'javax.naming.ldap.InitialLdapContext@bc081f' passed validation.>
                        DEBUG [...control.AbstractRequestControlDirContextProcessor] - <Replacing already existing control in context: com.sun.jndi.ldap.ctl.PagedResultsControl@97507c>
                        600
                        ...
                        Yes, you're right, this is a poor man's solution. If the connections should be built using different users, this is clearly not sufficient. We're discussing a more robust and generic solution based on a NeverClosingDirContext.

                        Comment


                        • #13
                          OK thanks it's working. The problem was that the default search used for validation was throwing unwilling to perform, because no base was specified by default. With a none empty base it's working.

                          It would be great to have a more robust solution for this and multi-user stuff, you are right. It is just really needed, so im looking forward to have it as soon as possible

                          Comment


                          • #14
                            Well today we tested the LDAP export against 300,000 records and it worked absolutely great. Low memory usage and quick processing and not a single error from the LDAP server. I can't thank you guys enough for releasing a fix so quickly.

                            Comment


                            • #15
                              I'm facing the same problem with Paged Results and OpenLDAP.

                              There are no nightly builds available for Spring-LDAP.
                              I've checked out trunk sources and manually built spring-ldap-core but the problem is still there : java.lang.UnsupportedOperationException: Cannot call setRequestControls on a pooled context

                              Furthermore, now PagedResultRequestControl class is deprecated.

                              I don't understand how anybody has already achieved paged results with OpenLDAP.

                              Here is my code :

                              Code:
                              public class FacultyLdapServiceSpring implements FacultyLdapService {
                              
                              	private LdapTemplate ldapTemplate;
                              
                              	private Filter allUsersFilter;
                              	private String rdnAttribute;
                              	private int pagedResultSize = 500;
                              
                              	public void setLdapTemplate(LdapTemplate ldapTemplate) {
                              		this.ldapTemplate = ldapTemplate;
                              	}
                              
                              	public void setAllUsersFilter(Filter allUsersFilter) {
                              		this.allUsersFilter = allUsersFilter;
                              	}
                              
                              	public void setRdnAttribute(String rdnAttribute) {
                              		this.rdnAttribute = rdnAttribute;
                              	}
                              
                              	public PagedResult getAllUsers(PagedResultsCookie cookie) {
                              		
                              		String [] attrs = {rdnAttribute};
                              		
                              		SearchControls searchControls = new SearchControls();
                              		searchControls.setReturningAttributes(attrs);
                              		searchControls.setSearchScope(SearchControls.SUBTREE_SCOPE);
                              		
                              		PagedResultsRequestControl pageControl = new PagedResultsRequestControl(pagedResultSize, cookie);
                              		
                              		List results = ldapTemplate.search("", allUsersFilter.encode(), searchControls, new UserMapper(), pageControl);
                              		return new PagedResult(results, pageControl.getCookie());
                              	}
                              
                              	private class UserMapper implements ContextMapper {
                              
                              		public Object mapFromContext(Object ctx) {
                              			DirContextAdapter context = (DirContextAdapter)ctx;
                              			LdapInformation ldapInformation = new LdapInformation();
                              			ldapInformation.setLogin(context.getStringAttribute(rdnAttribute));
                              			return ldapInformation;
                              		}
                              	}
                              }
                              Code:
                              public class ExpirationServiceDao implements ExpirationService {
                              
                              	protected static Log log = LogFactory.getLog(ExpirationService.class);
                              	
                              	private FacultyLdapService facultyLdapService;
                              	
                              	public void setFacultyLdapService(FacultyLdapService facultyLdapService) {
                              		this.facultyLdapService = facultyLdapService;
                              	}
                              
                              	public void testGetAllAccounts() {
                              		int nb = 0;
                              		PagedResultsCookie cookie = null;
                              		do {
                              			if(cookie != null) {
                              				log.debug("Cookie [" + cookie.getCookie().toString() + "]");
                              			}
                              			PagedResult result = facultyLdapService.getAllUsers(cookie);
                              			cookie = result.getCookie();
                              			nb+= result.getResultList().size();
                              			log.debug("Fetched " + result.getResultList().size() + " users. " + nb + " already fetched");
                              		}
                              		while(cookie != null && cookie.getCookie() != null);
                              	}
                              }
                              Here is the Spring configuration :
                              Code:
                              	<bean id="dsNancy2" class="org.springframework.ldap.core.support.LdapContextSource">
                              		<property name="url" value="${ldap.nancy2.url}/${ldap.nancy2.base}" />
                              		<property name="userDn" value="${ldap.nancy2.user}" />
                              		<property name="password" value="${ldap.nancy2.password}" />
                              		<!--<property name="pooled" value="false" />-->
                              	</bean>
                              
                              	<bean id="ldapNancy2" class="fr.univnancy2.web.sesame.dao.utils.SimpleLdapTemplate">
                              		<property name="contextSource" ref="dsNancy2" />
                              	</bean>
                              
                              	<bean id="nancy2Service" class="fr.univnancy2.web.sesame.dao.FacultyLdapServiceSpring">
                              		<property name="rdnAttribute" value="uid" />
                              		<property name="allUsersFilter" value="(objectclass=n2classpersonnel)" />
                              		<property name="ldapTemplate" ref="ldapNancy2" />
                              	</bean>
                              
                              	<bean id="expirationService" class="fr.univnancy2.web.sesame.domain.ExpirationServiceDao">
                              		<property name="facultyLdapService" ref="nancy2Service" />
                              	</bean>
                              Here is my test :
                              Code:
                              public class ExpirationTestCase extends TestCase {
                              
                              	private ExpirationService expirationService;
                              	
                              	public ExpirationTestCase() {
                              		ApplicationContext context = new ClassPathXmlApplicationContext("test-expiration.xml");
                              		expirationService = (ExpirationService)context.getBean("expirationService");
                              	}
                              	
                              	public void testExpiration() throws Exception {
                              		expirationService.testGetAllAccounts();
                              	}
                              }
                              Here are the results :
                              Code:
                              Running fr.univnancy2.web.sesame.test.domain.constraint.ExpirationTestCase
                              web-sesame DEBUG [main] dao.FacultyLdapService.[] janv./16 10:52:39 - FacultyLdapService::getAllUsers()
                              web-sesame  INFO [main] core.LdapTemplate.[] janv./16 10:52:39 - The returnObjFlag of supplied SearchControls is not set but a ContextMapper is used - setting flag to true
                              web-sesame DEBUG [main] domain.ExpirationService.[] janv./16 10:52:40 - Fetched 500 users. 500 already fetched
                              web-sesame DEBUG [main] domain.ExpirationService.[] janv./16 10:52:40 - Cookie [[B@19ea173]
                              web-sesame DEBUG [main] dao.FacultyLdapService.[] janv./16 10:52:40 - FacultyLdapService::getAllUsers()
                              web-sesame  INFO [main] core.LdapTemplate.[] janv./16 10:52:40 - The returnObjFlag of supplied SearchControls is not set but a ContextMapper is used - setting flag to true
                              web-sesame FATAL [main] control.AbstractRequestControlDirContextProcessor.[] janv./16 10:52:40 - No matching response control found for paged results - looking for 'class javax.naming.ldap.PagedResultsResponseControl
                              Code:
                              -------------------------------------------------------------------------------
                              Test set: fr.univnancy2.web.sesame.test.domain.constraint.ExpirationTestCase
                              -------------------------------------------------------------------------------
                              Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.692 sec <<< FAILURE!
                              testExpiration(fr.univnancy2.web.sesame.test.domain.constraint.ExpirationTestCase)  Time elapsed: 0.687 sec  <<< ERROR!
                              org.springframework.ldap.CommunicationException: [LDAP: error code 2 - paged results cookie is invalid]; nested exception is javax.naming.CommunicationException: [LDAP: error code 2 - paged results cookie is invalid]; remaining name ''
                              	at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:98)
                              	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:319)
                              	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:259)
                              	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:606)
                              	at fr.univnancy2.web.sesame.dao.utils.SimpleLdapTemplate.search(SimpleLdapTemplate.java:96)
                              	at fr.univnancy2.web.sesame.dao.FacultyLdapServiceSpring.getAllUsers(FacultyLdapServiceSpring.java:141)
                              	at fr.univnancy2.web.sesame.domain.ExpirationServiceDao.testGetAllAccounts(ExpirationServiceDao.java:28)
                              	at fr.univnancy2.web.sesame.test.domain.constraint.ExpirationTestCase.testExpiration(ExpirationTestCase.java:20)
                              	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                              	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                              	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                              	at java.lang.reflect.Method.invoke(Method.java:597)
                              	at junit.framework.TestCase.runTest(TestCase.java:168)
                              	at junit.framework.TestCase.runBare(TestCase.java:134)
                              	at junit.framework.TestResult$1.protect(TestResult.java:110)
                              	at junit.framework.TestResult.runProtected(TestResult.java:128)
                              	at junit.framework.TestResult.run(TestResult.java:113)
                              	at junit.framework.TestCase.run(TestCase.java:124)
                              	at junit.framework.TestSuite.runTest(TestSuite.java:232)
                              	at junit.framework.TestSuite.run(TestSuite.java:227)
                              	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
                              	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62)
                              	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140)
                              	at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:127)
                              	at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
                              	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                              	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                              	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                              	at java.lang.reflect.Method.invoke(Method.java:597)
                              	at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:338)
                              	at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:997)
                              Caused by: javax.naming.CommunicationException: [LDAP: error code 2 - paged results cookie is invalid]; remaining name ''
                              	at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3053)
                              	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2951)
                              	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2758)
                              	at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1812)
                              	at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1735)
                              	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
                              	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
                              	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:321)
                              	at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:248)
                              	at org.springframework.ldap.core.LdapTemplate$4.executeSearch(LdapTemplate.java:253)
                              	at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:293)
                              	... 29 more

                              Comment

                              Working...
                              X