Announcement Announcement Module
Collapse
No announcement yet.
tcp-gateway, how to end tcp-communication Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • tcp-gateway, how to end tcp-communication

    I am trying to use the tcp-inbound-gateway implementation to implement a request/response with an external, TCP-based client, and our spring integration based messaging system.

    The requirements call for the client to make a request, the messaging system to handle the request, and for the response to be sent back to the client on the same TCP socket.

    I have scoured through the documentation, and the source code for many hours to try and figure out how best to solve the issue I'm seeing, but I'm reaching out to those that developed it to help me understand something I may be missing.

    Everything happens correctly in that the tcp-inbound-gateway properly receives the message on the declared port, processes it, and sends back a response, however it seems the port does not get closed properly after the response is sent; it eventually times out, but the client receives a SocketException as a result of the timeout instead of the response.

    I've stepped through the sample tcp client/server project, but that seems to implement telnet where the connection properly does stay open while the client and server communicate back and forth, but I have not seen any examples of where the socket gets closed after a response, nor can i discern which is the appropriate way to go given my analysis of the source.

    Can someone help here?
    I can provide more details (config, etc) as necessary, but I am specifically using the regular socket implementation (not NIO) and setting single-use= true.

    Thanks in advance.

  • #2
    Sorry for the delay, I was on a cross-country flight.

    So here's the deal...

    ...it seems the port does not get closed properly after the response is sent...
    The server can't close the socket immediately after sending the response, because the FIN packet (close) might get propagated back to the client before the data packet(s).

    In this type of environment, it is normal for the client to close the socket after it receives the response. The server expects this, and will not complain when it detects the socket close.

    If, however, the client does not close the socket, we do what you have observed, when we get a read timeout, we close the socket. This is to protect the server from mis-behaved clients that might leave "single-use" sockets open.

    For single-use sockets, if no timeout is provided, we default to 10 seconds; this should be plenty to ensure the FIN doesn't beat the response to the client and, again, we're protecting the server because the normal default timeout is infinity.


    I am a little confused by your comments...

    ...the tcp-inbound-gateway properly receives the message on the declared port, processes it, and sends back a response...

    ...but the client receives a SocketException as a result of the timeout instead of the response....
    It seems you are saying we send the response OK, but you are implying the client doesn't get it; instead he later detects the socket closure after the timeout.

    That doesn't make sense to me, unless there is a problem with the protocol you are using for the response - and the client has read the response but is still expecting more data for some reason (because there is a mismatch in the wire protocol for the response compared to what he is expecting).

    You can verify what going on by using wireshark or similar to get a tcp trace (you will need to run the client on a different machine, though; localhost won't work).

    What format is the client expecting the response to have?
    Last edited by Gary Russell; Dec 17th, 2010, 08:44 PM.

    Comment


    • #3
      BTW note that, in the sample, single-use=true is set on the client side and each test is run over a separate socket...

      Code:
      ...
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Closing single use socket after inbound message localhost:11111:1653578290
      ...
      DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Closing single use socket after inbound message localhost:11111:803660549
      ...
      These logs are coming from the client side, of course.

      In this sample, we don't set single-use=true on the server side because the other part of the sample demonstrates a telnet server where we want the connection to remain open.

      single-use=true doesn't do much on the server side (for a gateway); it just suppresses the error when the socket times out, and logs it as debug instead.

      For an inbound channel adapter, where no reply is sent, single-use=true causes the server to close the socket after a message is received.

      I hope that further clarifies things.

      Comment


      • #4
        Hi Gary,

        Thanks for your detailed responses.

        It seems you are saying we send the response OK, but you are implying the client doesn't get it; instead he later detects the socket closure after the timeout.

        That doesn't make sense to me, unless there is a problem with the protocol you are using for the response - and the client has read the response but is still expecting more data for some reason (because there is a mismatch in the wire protocol for the response compared to what he is expecting).
        Yes, the implication you picked up on is correct; I should have been more clear. The TCP gateway (and associated connection classes, etc) does log that the response was sent, but the client doesn't see a response. Your suggestion of using wireshark to review what actually gets passed is a good idea.

        The type of data sent through the TCP port is a COBOL copybook string. We have created appropriate serializers/deserializers to handle this format. In our legacy java code that does this TCP handling, I noticed it closes the socket after the response is sent. I'm not sure if this is appropriate behavior as you've pointed out the FIN packet could arrive before the data packets.

        We've always used jmeter, or some home-grown socket testing tool to test the TCP handler we have, and I was hoping to use that to also test this implementation with SI and the TCP gateway. Effectively, the client apps we use to test (jmeter, etc) view the data being passed to and from the TCP handler as 'strings' with no particular protocol or format.

        Comment


        • #5
          I noticed it closes the socket after the response is sent. I'm not sure if this is appropriate behavior as you've pointed out the FIN packet could arrive before the data packets.
          Yes; for the most part, it's never going to be an issue but once in a great while it *will* happen and you'll spend a long time figuring out what happened. Typically, you only have to be burned by this once in your lifetime and you'll never make the mistake again :-)

          ...view the data being passed to and from the TCP handler as 'strings' with no particular protocol or format.
          Tcp data is a stream so you *have* to have some mechanism that will allow the recipient to know when it has received a message, whether it's a simple fixed COBOL-like format (all message are, say, 100 bytes long), or whether you add something to the data, such as [data]<CRLF>, <STX>[data]<ETX>, <length>[data] etc, etc

          If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.

          Comment


          • #6
            If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.
            This makes perfect sense.

            I'll take a second look at the legacy code we are using here to see how it accomplished notifying the client that the stream of data can be considered 'ended.'

            BTW... thanks much for your quick and detailed responses.

            Comment


            • #7
              If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.
              Yep. That's exactly what's happening.
              Using wireshark, I'm able to see that the TcpInboundGateway classes (connections, connection factories, etc) indeed send back the response, but the client (jmeter for these tests) does not know where the end of the stream is... it then continues to wait as you suggested and then the server times out. With our legacy code, jmeter (and other clients we use for testing) will correctly realize the end of the response by the FIN message sent when the server closes the socket.

              The legacy code (which we have to keep the same at this point... we're lucky enough that we can do any refactoring, but we won't be able to change the behavior at this point) also does not delimit or demarcate the end of the TCP stream; it uses the fact that closing the socket will send the FIN message to the client as the delimiter. To integrate with our SI messaging, I may have to look closer at the TCP channel adapters or custom channel adapters instead of the TcpInboundGateway, which I was really hoping to use.

              I can try to make the case for forcing the client to know how to demarcate the stream (as I would tend to agree with you), but I cannot find much documentation detailing the possibility that the FIN message might be propagated to the client before the data. Can you help me understand better why that would happen (I realize the FIN message can get there earlier, but does the TCP transport not realize that there may be more data and to continue trying to gather all the relevant packets?)? I can't seem to find any blogs or forum threads that detail this possible issue. Maybe I'm not using the correct search terms, so any guidance in finding documentation would be helpful in presenting my findings to those that can make a decision about changing the behavior of our legacy code.
              Thanks Gary!

              Comment


              • #8
                Interesting - I just re-read the TCP RFC (it's been a few years) and it does, indeed, say that it should be ok to send a FIN after data and the remote TCP should reassemble things and take the FIN after the data, even if the FIN arrives first.

                All I can say is that I have certainly had it happen in my past; perhaps there was a bug in the TCP stack I was using; perhaps there was a buggy router - I did find some references on the 'net where routers dropped data packets after seeing a FIN.

                Let me look into it some more.

                In the meantime, I am a little confused as to why the default behavior does not help you out, though. When we get the read error (no more inbound messages), we go ahead and close the socket anyway, which will send the FIN, albeit 10 seconds (by default) after your data. Maybe JMeter is timing out before the 10 seconds?

                Perhaps you could try, say, setting so-timeout="100"; this will cause us to close the socket 100 milliseconds after the write.

                You need to be careful not to set the timeout so low that we might timeout during reads on a busy network (assuming the inbound message is large enough to get segmented). The same timeout is used for both situations.

                Before giving up on the gateway, you might also consider adding an interceptor to explicitly close the socket after the write. I realize this is an advanced feature and, although documented, there are no samples. There is, however, a HelloWorld handshaking interceptor in the test cases for the ip module.

                I can help you with the interceptor but, in essence, you would extend AbstractTcpConnectionInterceptor, overriding the send() method something like as follows...

                Code:
                	@Override
                	public void send(Message<?> message) throws Exception {
                		super.send(message);
                                this.close();
                	}
                I have not tested this, so no warranties :-)

                If my investigations point me to it being OK to do the close, I will, of course, make the change - I am not sure when it will be available in a release, though.

                Comment


                • #9
                  Here is the code for the interceptor approach - I added it to the tcp-client-server-sample and it worked fine. The only problem with it is it causes the gateway to emit an ERROR log when it detects the socket closed unexpectedly.

                  Code:
                  ERROR: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Read exception localhost:46020:1579948023 SocketException:null:Socket is closed
                  Code:
                  package org.springframework.integration.samples.tcpclientserver;
                  
                  import org.springframework.integration.Message;
                  import org.springframework.integration.ip.tcp.connection.AbstractTcpConnectionInterceptor;
                  import org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptor;
                  import org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptorFactory;
                  
                  /**
                   * @author Gary Russell
                   *
                   */
                  public class SocketClosingInterceptorFactory implements
                  		TcpConnectionInterceptorFactory {
                  
                  	public TcpConnectionInterceptor getInterceptor() {
                  		
                  		return new AbstractTcpConnectionInterceptor() {
                  			@Override
                  			public void send(Message<?> message) throws Exception {
                  				super.send(message);
                  				this.close();
                  			}			
                  		};
                  	}
                  
                  }
                  Code:
                  	<ip:tcp-connection-factory id="crLfServer"
                  		type="server"
                                  single-use="true"
                  		interceptor-factory-chain="socketClosingFactoryChain"
                  		port="11111"/>
                  			
                  	<beans:bean id="socketClosingFactoryChain" class="org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptorFactoryChain">
                  		<beans:property name="interceptors">
                  			<beans:array>
                  				<beans:bean class="org.springframework.integration.samples.tcpclientserver.SocketClosingInterceptorFactory"/>
                  			</beans:array>
                  		</beans:property>
                  	</beans:bean>
                  Last edited by Gary Russell; Dec 20th, 2010, 07:12 PM.

                  Comment


                  • #10
                    Hi Gary,

                    So here's what I have to report back

                    I tried the interceptor approach you recommended. I still wasn't getting the expected behavior (the FIN message) when sending the response back to the client. Instead I was getting the RST message upon closing the socket. Watching all of this over wireshark and jmeter, I could see that the packets were sent, but the appropriate TCP end-communication steps weren't taken, ie, the RST message was sent and jmeter would consider it a "connection reset" and drop all of the data (which is why it wasn't showing up in the response console).

                    I tried all kinds of things (editing some of the files in the Tcp module of SI and observing the behavior) to get the expected behavior, to no avail. Both trying things with the interceptor and without the interceptor. I always seemed to get the RST message and never a FIN message.

                    I found in this document (http://download.oracle.com/javase/1....n_release.html) that the 'abortive procedure' (as opposed to the orderly connection release) will always be used when Socket.setLinger(true,0) is called. In other words, a RST message will always be sent when a socket is closed.

                    So, how do Java applications perform orderly and abortive releases? Let's consider abortive releases first. A convention that has existed since the days of the original BSD sockets is that the "linger" socket option can be used to force an abortive connection release. Either application can call Socket.setLinger (true, 0) to tell the TCP stack that when this socket is closed, the abortive (RST) procedure is to be used
                    When I changed the so-linger option to a non-negative, non-zero number (5000 for my case) in the config files, in conjunction with the interceptor-close method you showed in the previous post, I noticed that the FIN message was indeed sent properly and everything worked as expected.

                    Further digging in the source, I see that if the so-linger option is omitted from the xml config, the Socket.setLinger(true,0) is still called:

                    from org.Springframework.integration.ip.tcp.connection. AbstractConnectionFactory.setSocketAttributes(Sock et socket)....

                    Code:
                    ...
                    		if (this.soLinger >= 0) {
                    			socket.setSoLinger(true, this.soLinger);
                    		}
                    ...
                    Is it a good idea to use to the abortive connection release (sending the RST message) by default? Or would it be better to only set the so-linger field on the socket object only when one is specified in the config files (ie, this.soLinger > 0 instead of greater than or equal)?

                    As you mentioned, the exceptions do show up in the log when using the interceptor to close the socket, but the behavior is as I expected, ie, the request/response streams are properly terminated with the FIN messages and the client app (jmeter in this case) receives the response.

                    Comment


                    • #11
                      @posta07

                      Thanks for doing the research; my intention was that I only wanted to set SOLinger *if* a specific value had been set in the config (even 0). I intended to initialize the variable to -1 so the initialization code wouldn't set the option by default.

                      Can you create a JIRA ticket here https://jira.springsource.org ??

                      You'll have to create an account if you don't have one. I can enter the ticket but if you do it, you can track the progress.

                      I will do my utmost to get the fix in to 2.0.2 and, if you're comfortable with nightlies, you could get the fix even faster. In the mean time, I assume the workaround is ok for now?

                      Again, many thanks for grounding this out.

                      Gary

                      Comment


                      • #12
                        FYI, the root cause was indeed fixed by setting the soLinger field in AbstractTcpConnectioFactory to -1. Of course, it still took the timeout before the FIN was sent so that, alone won't help you, but at least we're sending a FIN instead of a RST.

                        However, when I unconditionally close the socket immediately after the send, a whole slew of the existing test cases fail - I need to get to the bottom of that and figure out if those use cases are legitimate. I may have to make your desired behavior an option (and suppress the error on the read, of course).

                        BTW, because of the way NIO is processed, we get no error when the socket is "prematurely" closed because there's no thread hanging on a read. You may want to consider using NIO to suppress the error while you use the interceptor workaround.

                        Here is my test case for the FIN processing; they work when we don't set SOLinger...

                        Code:
                        <?xml version="1.0" encoding="UTF-8"?>
                        <beans xmlns="http://www.springframework.org/schema/beans"
                        	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                        	xmlns:int="http://www.springframework.org/schema/integration"
                        	xmlns:int-ip="http://www.springframework.org/schema/integration/ip"
                        	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        		http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
                        		http://www.springframework.org/schema/integration/ip http://www.springframework.org/schema/integration/ip/spring-integration-ip-2.0.xsd">
                        
                        	<bean id="tcpIpUtils" class="org.springframework.integration.ip.util.SocketTestUtils" />
                        	
                        	<int-ip:tcp-connection-factory id="inCFNet"
                        		type="server"
                        		port="#{tcpIpUtils.findAvailableServerSocket(9000)}"
                        		so-timeout="1000"
                        		single-use="true"
                        		/>
                        	
                        	<int-ip:tcp-inbound-gateway request-channel="echo"
                        		connection-factory="inCFNet" />
                        	
                        	<int-ip:tcp-connection-factory id="inCFNio"
                        		type="server"
                        		port="#{tcpIpUtils.findAvailableServerSocket(9100)}"
                        		so-timeout="1000"
                        		single-use="true"
                        		using-nio="true"
                        		/>
                        	
                        	<int-ip:tcp-inbound-gateway request-channel="echo"
                        		connection-factory="inCFNio" />
                        	
                        	<int:service-activator input-channel="echo" ref="testService"/>
                        	
                        	<bean id="testService" class="org.springframework.integration.ip.tcp.TestService"/>
                        
                        </beans>
                        Code:
                        package org.springframework.integration.ip.tcp.connection;
                        
                        import static org.junit.Assert.assertEquals;
                        import static org.junit.Assert.fail;
                        
                        import java.io.IOException;
                        import java.io.InputStream;
                        import java.net.Socket;
                        
                        import javax.net.SocketFactory;
                        
                        import org.junit.Test;
                        import org.junit.runner.RunWith;
                        import org.springframework.beans.factory.annotation.Autowired;
                        import org.springframework.test.context.ContextConfiguration;
                        import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
                        
                        /**
                         * @author Gary Russell
                         * @since 2.0.2
                         *
                         */
                        @RunWith(SpringJUnit4ClassRunner.class)
                        @ContextConfiguration
                        public class SOLingerTests {
                        
                        	@Autowired
                        	private AbstractServerConnectionFactory inCFNet;
                        	
                        	@Autowired
                        	private AbstractServerConnectionFactory inCFNio;
                        	
                        	@Test
                        	public void configOk() {}
                        
                        	@Test
                        	public void finReceivedNet() {
                        		finReceived(inCFNet);
                        	}
                        
                        	@Test
                        	public void finReceivedNio() {
                        		finReceived(inCFNio);
                        	}
                        	
                        	private void finReceived(AbstractServerConnectionFactory inCF) {
                        		int port = inCF.getPort();
                        		int n = 0;
                        		while (!inCF.isListening()) {
                        			try {
                        				Thread.sleep(100);
                        			} catch (InterruptedException e) {
                        				Thread.currentThread().interrupt();
                        				fail("Interrupted");
                        			}
                        			if (n++ > 100) {
                        				fail("Failed to start");
                        			}
                        		}
                        		try {
                        			Socket socket = SocketFactory.getDefault().createSocket("localhost", port);
                        			String test = "Test\r\n";
                        			socket.getOutputStream().write(test.getBytes());
                        			byte[] buff = new byte[test.length() + 5];
                        			readFully(socket.getInputStream(), buff);
                        			assertEquals("echo:" + test, new String(buff));
                        			n = socket.getInputStream().read();
                        			// we expect an orderly close
                        			assertEquals(-1, n);
                        		} catch (Exception e) {
                        			e.printStackTrace();
                        			fail("Unexpected Exception  " + e.getMessage());
                        		}
                        	
                        	}
                        	
                        	private void readFully(InputStream is, byte[] buff) throws IOException {
                        		for (int i = 0; i < buff.length; i++) {
                        			buff[i] = (byte) is.read();
                        		}
                        	}
                        	
                        }

                        Comment


                        • #13
                          Yep, using NIO does not produce the error messages. This is more desirable, so I will go with that for now.

                          Thanks again for your help and suggestions.

                          BTW, this is off topic as it's not directly related to my problem, but I had a question about the interceptors.

                          I was looking at the code, and I'm not clear about how multiple interceptors would get run.

                          From AbstractConnectionFactory, in the wrapConnection method it looks like you loop through the interceptors, assign the interceptor as a listener of a connection, then replace the connection object with the interceptor. From there the loop will continue with the regsiterListener now being called on the the previous wrapper (which sets the original connection's listener and so forth). What I noticed is that a connection can have only one listener. So if the wrapper keeps overwriting the connection's listener, won't there be only one interceptor associated with the connection? I'm probably missing something.

                          Thanks!

                          Comment


                          • #14
                            No; the interceptor only becomes the 'listener' in wrapConnection if there is no real listener for this connection (it is a send-only connection).

                            For connections that can receive messages, we register a listener with the connection - this is usually a receiving channel adapter, or a gateway.

                            We can also register a sender with a connection, this is the object that sends messages (either an outbound adapter or a gateway).

                            Now, TcpConnectionInterceptor extends TcpConnection, TcpListener, and TcpSender.

                            In the wrapConnection method, for each interceptor in the chain, the interceptor (variable 'wrapper') is given a reference to the connection.

                            In the case of a gateway, there is always a listener and sender. So, for each interceptor in the chain, the interceptor is given a reference to the connection and this interceptor becomes the new "connection"; if there are two interceptors, the first gets a reference to the the real connection, the second gets a reference to the first interceptor. We finally return the second interceptor as "the" connection.

                            Finally, in initializeConnection(), we register the listener (gateway) with "the" connection (which is our second interceptor in this case).

                            registerListener() in AbstractTcpConnectionInterceptor looks like this...

                            Code:
                            	public void registerListener(TcpListener listener) {
                            		this.theConnection.registerListener(this);
                            		this.tcpListener = listener;
                            	}
                            This effectively chains the interceptors - interceptor 2's listener is the gateway, interceptor 1's listener is interceptor 2, the real connection's listener is interceptor 1. Now, when a new message is assembled, the message is passed up through the interceptor chain.

                            If you want to see it in action, check out the source from git and take a look at InterceptedSharedConnectionTests where I nested two hello world interceptors. These interceptors do some handshaking during the initial send.

                            Code:
                            	<bean id="helloWorldInterceptors" class="org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptorFactoryChain">
                            		<property name="interceptors">
                            			<array>
                            				<bean class="org.springframework.integration.ip.tcp.connection.HelloWorldInterceptorFactory"/>
                            				<bean class="org.springframework.integration.ip.tcp.connection.HelloWorldInterceptorFactory">
                            					<constructor-arg value="Hi"/>
                            					<constructor-arg value="planet!"/>
                            				</bean>
                            			</array>
                            		</property>
                            	</bean>
                            On the client side, when we send a message, the second (outer) interceptor sees negotiation is needed so holds on to the original messge and sends 'Hi' which is intercepted by the inner interceptor who also needs negotiation, so he holds on to the 'Hi' sends 'Hello' and . On the server side, the inner interceptor sees the Hello and sends 'world!' which is received by the inner client interceptor and because negotiation is complete, sends on the 'Hi'. On the server side, the inner interceptor is fully negotiated so he passes the Hi up the chain; the outer interceptor completes his negotiation and, eventually, the original payload is sent and from that point on, both interceptors are pass-throughs.

                            Code:
                            2010-12-22 11:29:06,564 DEBUG [main] DirectChannel: preSend on channel 'input', message: [Payload=Test][Headers={timestamp=1293035346564, id=bd9cb8e6-39b6-4486-a1ae-132dba8840f2}]
                            ...
                            2010-12-22 11:29:06,576 DEBUG [main] HelloWorldInterceptor: Sending Hi
                            2010-12-22 11:29:06,576 DEBUG [main] HelloWorldInterceptor: Sending Hello
                            ...
                            2010-12-22 11:29:06,585 DEBUG [pool-1-thread-3] HelloWorldInterceptor: sending world!
                            ...
                            2010-12-22 11:29:06,587 DEBUG [pool-2-thread-3] HelloWorldInterceptor: received world!
                            ...
                            2010-12-22 11:29:06,619 DEBUG [pool-1-thread-2] HelloWorldInterceptor: sending planet!
                            ...
                            2010-12-22 11:29:06,661 DEBUG [pool-2-thread-5] HelloWorldInterceptor: received planet!
                            ...
                            2010-12-22 11:29:06,662 DEBUG [main] DirectChannel: postSend (sent=true) on channel 'input', message: [Payload=Test][Headers={timestamp=1293035346564, id=bd9cb8e6-39b6-4486-a1ae-132dba8840f2}]
                            
                            ...
                            
                            2010-12-22 11:29:06,702 DEBUG [main] QueueChannel: postReceive on channel 'replies', message: [Payload=Test][Headers={timestamp=1293035346702, id=cad2f313-16ba-48ee-9173-beef6dce86c5, ip_address=127.0.0.1, ip_connection_seq=3, ip_hostname=localhost, ip_tcp_remote_port=10100, ip_connection_id=localhost:10100:1022146175}]
                            BTW, the interceptor chain was conceived for doing handshaking/negotiation such as this, but we did foresee that it might be used for other things, so I am pleased we were able to use it as a work-around for your problem.

                            Comment


                            • #15
                              Hi Christian,

                              I managed to get the fix into tonight's nightly build.

                              It would be great if you have an opportunity to test it in your environment before we release 2.0.2.

                              If you have time, you can change your pom to pull down Spring Integration 2.0.2.BUILD-SNAPSHOT.


                              Thanks again for finding this and your help with tracking it down.

                              Gary

                              Comment

                              Working...
                              X