Announcement Announcement Module
Collapse
No announcement yet.
Quartz or JMS or ?? Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Quartz or JMS or ??

    Hi,

    I need some opinions on how to design a software like this:

    job one is to get Data from the Web on every trading day, transform this data into objects and store them into a Database.
    after this there are some calculations made in job two.
    and with the result of this calculations are thirdly job three sending emails with the results.

    a other thing is a web frontent for managing the objects (Data).

    my question is, how to desing this application best.

    par example I can create one appliaction whit a webfrontend and quartz triggerd beans.
    or I use two applications or four applications with communicate over JMS even there are on the same tomcat server or on the same hardware.
    The Problem is that if first job crashes after a defined number of trys, two und three to must not be executed. A admin have to be informed to solve this problem, if it is done the job one have to be executed again with the same state as befor the crash and than two and three have to be executed.

    because of using HTTP to get the data job one needs 5 to 10 min (at time in the old no more useable Version).

    what will be the best JMS or Quartz to solve this Problem? or there are any other solutions?

    Thanks,

    mfg Gideon

  • #2
    Re: Quartz or JMS or ??

    Originally posted by Gideon
    Hi,

    I need some opinions on how to design a software like this:

    job one is to get Data from the Web on every trading day, transform this data into objects and store them into a Database.
    after this there are some calculations made in job two.
    and with the result of this calculations are thirdly job three sending emails with the results.

    a other thing is a web frontent for managing the objects (Data).

    my question is, how to desing this application best.

    par example I can create one appliaction whit a webfrontend and quartz triggerd beans.
    or I use two applications or four applications with communicate over JMS even there are on the same tomcat server or on the same hardware.
    The Problem is that if first job crashes after a defined number of trys, two und three to must not be executed. A admin have to be informed to solve this problem, if it is done the job one have to be executed again with the same state as befor the crash and than two and three have to be executed.

    because of using HTTP to get the data job one needs 5 to 10 min (at time in the old no more useable Version).

    what will be the best JMS or Quartz to solve this Problem? or there are any other solutions?

    Thanks,

    mfg Gideon
    You could use Quartz for the triggering, but you could use also use a lighter alternative (like the java.util.Timer and if you use jdk 5.0 you can use the scheduledexecutor from the new concurrency library.. there is also a backport of the concurrency library for jdk 1.4). I prefer using a lighter alternative.. the only thing I use quartz for is the ability to use expression for triggers.. but for the rest I find it too heavy.

    The choice for the trigger technology should not have an impact in the design of your system. I can change the trigger technology I use in my application without writing a single line of code, I only have to change the Spring configuration.

    JMS could be used to implement the factoryline, but I think this solution is overkill if you don`t need those tasks being distributed on different machines. If you use the concurrency library from jdk 5.0, you`ll find excelent tools for creating such a factory line. In the last half year I have created a few factory lines:
    -a complex system for mailing to be send
    -analysis system for Lucene
    They all are build on a line of Blockingqueues with ExecutorServices (ThreadPoolExecutorServices) that can be build/configured perfectly from Spring. You even can use DirectExecutor instead of ThreadPoolExecutorService to test the components


    So if you don`t need those tasks being remoted, I wouldn`t recommend JMS. And you can use ordinairy transactions to make it atomic (store the job data in the db).

    Comment


    • #3
      HI,

      at time i will choose JMS because:

      I can start the process when I will (Quartz or Trigger are configured in Spring)
      Process 2 will only be startet if Process 1 send them a messeage to start
      Process 1 can start Process 2 when a subset of the work is done so many things can be done at the same time.
      Proccess 1 can rollback its changes whenn Process 3 have Erros, and then try again, and thats a verry good thing only the subset of the work must be rollback.

      if this is also possible with concurrency can you give me a example please ? (never heard abot it)

      the criticial points are:

      the Website where to get the Data my be not available so the process must automacitly try again later

      the data is corrupt, then the complete process must stop immediatly and rollback the changes if there are some.

      the time, the process ca only start at 22.00 o'clock the e-mails must be delivered until 22.15 o'clock if there is no error.

      are there solutions to do this with your tool?

      thanks.

      mfg Gideon

      Comment


      • #4
        An example:
        Code:
        class Job1Processor{
        	private ExecutorService _executorService;
        	private Job2Processor _nextProcessor;
        	
        	//constructor for the fields.
        	
        	public void check(){
        		List job1DataList = retrieveDataFromSomewhere();
        		for(Iterator it = jobDataList.iterator();it.hasNext();()){
        			Job1 job = new Job1((Job1Data)it.next());
        			_executorService.submit(job);
        		}
        	}
        	
        	private class Job1 implements Runnable{
        		private Job1Data _data;
        		
        		//contructor for fields
        		
        		public void run(){
        			.. do something difficult with your data
        			Job2Data job2Data = new Job2Data(...);
        			_nextProcessor.execut(job2Data);
        		}
        	}
        }
        
        class Job2Processor{
        	private ExecutorService _executorService;
        	private Job3Processor _nextProcessor;
        	
        	public void execute(Job2Data data){
        		Job2 job = new Job2(data);
        		_executor.submit(job);
        	}
        	
        	.. job2 implementation
        }
        If you create an extra initialize method in every processor, and let spring execute that initialize method if the bean is created, you can let them check the database if there were unfinished jobs from a previous run.

        And this is how it could be configured in Spring:
        Code:
        <bean id="job1Processor"
        		class="Job1Processor">
        
        		<constructor-arg index="0">
        			<bean
        				class="java.util.concurrent.ThreadPoolExecutor"
        				destroy-method="shutdown">
        
        				<!-- core pool size -->
        				<constructor-arg index="0">
        					<value>3</value>
        				</constructor-arg>
        
        				<!-- max pool size -->
        				<constructor-arg index="1">
        					<value>3</value>
        				</constructor-arg>
        
        				<!-- die-time -->
        				<constructor-arg index="2">
        					<value>5000</value>
        				</constructor-arg>
        
        				<!-- de tijdeenheid waarmee deze executorservice gaat werken -->
        				<constructor-arg index="3">
        					<ref bean="java.util.concurrent.TimeUnit.MILLISECONDS"/>
        				</constructor-arg>
        
        				<!-- de queue die gebruikt wordt om taken in op te slaan -->
        				<constructor-arg index="4">
        					<bean class="java.util.concurrent.LinkedBlockingQueue">
        						<constructor-arg index="0">
        							<!-- maximale size -->
        							<value>5000</value>
        						</constructor-arg>
        					</bean>
        				</constructor-arg>
        
        				<!--
        					- de threadfactory die deze executorservice gebruikt om threads voor de threadpool
        					- aan te maken.
        					 -->
        				<constructor-arg index="5">
        					<bean class="com.jph.concurrent.StdThreadFactory">
        						<constructor-arg index="0">
        							<!--
        								- Geef de analyzers een lage priority zodat het systeem voor
        								   - gebruikers wel vlot blijft aanvoelen. Zorg er verder voor
        								- dat de prioriteit lager is dan die van de indexwriters, zodat
        								- je eerder een index weg gaat schrijven, dan analyze taken
        								- gaat uitvoeren.
        								 -->
        							<value>1</value>
        						</constructor-arg>
        
        						<!-- De naam van ThreadGroup waar de Threads onder vallen -->
        						<constructor-arg index="1">
        							<value>analyzers</value>
        						</constructor-arg>
        					</bean>
        				</constructor-arg>
        			</bean>
        		</constructor-arg>
        
        		<constructor-arg index="1">
        			<ref bean="job2Processor"/>
        		</constructor-arg>
        	</bean>

        Comment


        • #5
          Hi,

          thank you for your example.

          if i understood right,

          the following architecture should be right for me:

          using a scheduler for executing a JobProcessor every 15 min or so on.

          this JobProcessor look sinto a database for jobs that should be executet.

          a another scheduler calls a JMS template with put the jobs for getting the URL Data into the Jobsdatabase.

          the getURLdata jobs also puts a job into the database for calculating.
          and the calculating job puts the sendMail jobs into the JobDatabase.

          I think with this solution I get the following:

          everey job will be 100% executed and at the right time in the right order.
          over JMS i can execute a job manual.
          the jobs will be executet parallel for saving time.

          Are there still any suggestions ?

          Thanks.

          mfg Gideon

          Comment


          • #6
            Originally posted by Gideon
            Hi,

            thank you for your example.

            if i understood right,

            the following architecture should be right for me:

            using a scheduler for executing a JobProcessor every 15 min or so on.
            Yes... some kind of scheduler (quartz,something else) needs to trigger the first processor.

            processor1->processor2->.... ->processorn

            After that the other processors wil be activated by the previous processor. So prosessor1 activates processor2 if processor1 is finished with a job.. processor n-1 wil activate processor n if processorn-1 is finished with a job.

            this JobProcessor look sinto a database for jobs that should be executet.
            Yes... but all the processors have to look in the database as soon as they start up if there are jobs they should proces. After they have checked the database, they won`t have to check for new jobs again (only the first processor needs to check the db once and a while). All the processor (accept the first) wil be activated by the previous processor. The previous processor should store all the data for the next processor in the db, and has to tell the next processor: he.. there is a new job for you in the db.. this is the key.. or could give him the job-data directly (save a db trip). But the JobQueues (every processor has one) has to be kept synchronized with the db.

            a another scheduler calls a JMS template with put the jobs for getting the URL Data into the Jobsdatabase.
            Kick JMS

            the jobs will be executet parallel for saving time.
            Making a system multithreaded doesn`t have to make it faster. If there is a lot of IO (so a lot of blocking) a multithreaded version could be faster. But there are scenarios where a multithreaded app is not faster (could even be slower because of the overhead of concurrency: synchronisation/context switching etc etc.

            Are there still any suggestions ?
            Kick JMS if you don`t need remoting.

            Comment


            • #7
              Originally posted by Alarmnummer
              Kick JMS if you don`t need remoting.
              In fact, in general a job executor doesn't necessarily have to run from the same JVM where the job came from, since all the job data can be read from the DB. And there can be multiple job executors work distributedly, with a DB level locking mechanism.

              Comment

              Working...
              X