Announcement Announcement Module
Collapse
No announcement yet.
OtpIOException: failed to connect from 'rabbit-spring-monitor' to peer node Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • OtpIOException: failed to connect from 'rabbit-spring-monitor' to peer node

    I'm working on some code to send heartbeat messages from consumers. Getting the following exception:

    Code:
    org.springframework.erlang.OtpIOException: failed to connect from 'rabbit-spring-monitor' to peer node 'rabbit@localhost'
    	at org.springframework.erlang.connection.SimpleConnectionFactory.createConnection(SimpleConnectionFactory.java:65)
    	at org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangAccessor.java:43)
    	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:114)
    	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:46)
    	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:98)
    	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getQueues(RabbitBrokerAdmin.java:148)
    	at spg.poly.consumer.monitoring.HeartbeatPublisher.sendHeartbeat(HeartbeatPublisher.java:33)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    	at java.lang.reflect.Method.invoke(Method.java:597)
    	at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
    	at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)
    	at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)
    	at java.util.TimerThread.mainLoop(Timer.java:512)
    	at java.util.TimerThread.run(Timer.java:462)
    Caused by: java.io.IOException: Cannot connect to peer node
    	at com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.java:998)
    	at com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:205)
    	at com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
    	at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
    	at org.springframework.erlang.connection.SimpleConnectionFactory.createConnection(SimpleConnectionFactory.java:62)
    	... 15 more
    Here's the code causing the exception:

    Code:
    public void sendHeartbeat() {                                 
        log.info("sending heartbeat...");
    
        ConnectionFactory connFactory = config.connectionFactory();
        RabbitBrokerAdmin admin = new RabbitBrokerAdmin(connFactory);
    
        StringBuilder msg = new StringBuilder();
        msg.append("[Heartbeat] ");
    
        List<QueueInfo> queues = admin.getQueues();
        for (QueueInfo qi : queues) {
    
          msg.append("consumers=").append(qi.getConsumers());
    
        }
    
        config.rabbitTemplate().convertAndSend("heartbeat-ex", null, msg);
      }
    The broker is version 2.0 and has the default settings.
    Any ideas?

  • #2
    Hi,

    The issue is that JInterface requires that the string for the host name matches what the rabbit broker reports as its node name. Look at the console output when starting rabbit for 'node ='.

    JInterface actually hides the underlying exception related to this handshake protocol, I'll see how to file an issue so the stack trace and underlying exception can better describe what is happening.

    In my case the node name is called rabbit@MARK6500 and not rabbit@localhost.

    I'm guessing that you are passing 'rabbit@localhost' into the constructor of the connection factory (SingleConnectionFactory or CachingConnectionFactory). If you use the zero arg constructor it will call InetAddress.getLocalHost().getHostName() which will typically match what the rabbit broker is using for its node name.

    The node name for rabbit itself can be configured in rabbitmq.conf should you want to use something else.

    You can experiment with this yourself with the following lower level code that demonstrates the problem.

    Code:
    OtpSelf self = new OtpSelf("rabbit-monitor");
    OtpPeer peer = new OtpPeer("rabbit@localhost");
    self.connect(peer);
    will fail while

    Code:
    OtpSelf self = new OtpSelf("rabbit-monitor");
    OtpPeer peer = new OtpPeer("rabbit@" + InetAddress.getLocalHost().getHostName());
    self.connect(peer);
    will pass.

    Cheers,
    Mark

    Comment


    • #3
      Mark, thanks for the reply. I did as you suggested and passed a zero arg constructor to the connection factory. I also confirmed that the hostname matches what my local broker reports. However, I'm still seeing the same exception (with the changed hostname):

      Code:
      1:40:16 [ConsumerTimerFactory] ERROR
      timer.MethodInvokingTimerTaskFactoryBean  - Invocation of method
      'sendHeartbeat' on target class [class
      spg.poly.consumer.monitoring.HeartbeatPublisher] failed
      org.springframework.erlang.OtpIOException: failed to connect from
      'rabbit-spring-monitor' to peer node 'rabbit@dbjaxwslkhgzmy'
             at
      org.springframework.erlang.connection.SimpleConnectionFactory.createConnec
      tion(SimpleConnectionFactory.java:65)
             at
      org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangA
      ccessor.java:43)
             at
      org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java
      :114)
             at
      org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemp
      late.java:46)
             at
      org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(Erlang
      Template.java:98)
             at
      org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getQueues(RabbitBr
      okerAdmin.java:148)
             at
      spg.poly.consumer.monitoring.HeartbeatPublisher.sendHeartbeat(HeartbeatPub
      lisher.java:33)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
      39)
             at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
      pl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at
      org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
             at
      org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodIn
      vokingRunnable.java:65)
             at
      org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTim
      erTask.java:70)
             at java.util.TimerThread.mainLoop(Timer.java:512)
             at java.util.TimerThread.run(Timer.java:462)
      Caused by: java.io.IOException: Cannot connect to peer node
             at
      com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.ja
      va:998)
             at
      com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:
      205)
             at
      com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
             at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
             at
      org.springframework.erlang.connection.SimpleConnectionFactory.createConnec
      tion(SimpleConnectionFactory.java:62)
             ... 15 more

      Comment


      • #4
        In Rabbit server the node name does not necessarily have the form of 'rabbit@yourhostname'. To find out yours, run

        Code:
        cat /var/log/rabbitmq/startup_log
        In the output you will see the node name. Providing correct name fixes 'Cannot connect to peer node' error, but you also need to provide correct cookie to pass authentication. Just to prove the concept I made quick hack (see attachment), after which 'admin.getQueues()' successfully passed.

        Now we need to think how to implement it properly.

        Comment


        • #5
          I was never able to get this working on windows but I did get it working on mac. Is it possible this is a windows specific bug?

          I'm setting host to 'DBJAXWSLKHGZMY' which leads to the exception below. Setting it to 'dbjaxwslkhgzmy' results in the previous exception. Rabbit reports node: 'rabbit@DBJAXWSLKHGZMY' on startup.

          Code:
          Exception in thread "main" org.springframework.erlang.UncategorizedOtpException: com.ericsson.otp.erlang.OtpAuthException: Peer authentication error.
          	at org.springframework.erlang.support.ErlangUtils.convertOtpAccessException(ErlangUtils.java:51)
          	at org.springframework.erlang.core.ErlangTemplate.convertOtpAccessException(ErlangTemplate.java:142)
          	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:121)
          	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:46)
          	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:98)
          	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:292)

          Comment


          • #6
            At the prompting of Dave Syer, I tried again using a recent snapshot (1.0.0.BUILD-20101101.013057). The following is the result:

            Code:
            Exception in thread "main" org.springframework.amqp.rabbit.admin.RabbitAdminAuthException: Could not authorise connection to Erlang process. This can happen if the broker is running, but as root or rabbitmq and the current user is not authorised to connect. Try starting the broker again as a different user.
            	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:358)
            	at spg.dexc.integ.BrokerAdminTest.stopApp(BrokerAdminTest.java:32)
            	at spg.dexc.integ.BrokerAdminTest.main(BrokerAdminTest.java:22)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:585)
            	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
            Caused by: com.ericsson.otp.erlang.OtpAuthException: Peer authentication error.
            	at com.ericsson.otp.erlang.AbstractConnection.recvChallengeAck(AbstractConnection.java:1303)
            	at com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.java:990)
            	at com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:205)
            	at com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
            	at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
            	at org.springframework.erlang.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:211)
            	at org.springframework.erlang.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:150)
            	at org.springframework.erlang.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:133)
            	at org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangAccessor.java:43)
            	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:115)
            	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:47)
            	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:99)
            	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:352)
            	... 7 more

            Comment


            • #7
              I decided to try the same code on another windows machine and it worked. Will report back when I find something more conclusive.

              Comment


              • #8
                The problem seems to come up when running on Windows XP. Tried it on 3 different XP machines and always fails. Works ok on Windows 7.

                Comment


                • #9
                  Correction. Seems to have more to do with case sensitivity around computer name and/or hostname. Windows version doesn't seem to matter.

                  Comment

                  Working...
                  X