Wednesday, September 23, 2009

Straight talk with ActiveMQ

I found that filling an ActiveMQ queue through a cfml gateway was extremely inefficient when sending a large number of messages at once. Here is the code to send messages directly to the queue via java. This was a little difficult to find this documented anywhere. I had to pull in bits an pieces from different code examples.
<cfset mess = structnew() />

<cfquery name="rs" datasource="dsn">
select id from mytable
</cfquery>

<cfscript>
connectionFactory = createobject(
'java'
,'org.apache.activemq.ActiveMQConnectionFactory'
).init('','', 'tcp://#amqServer#:#amqPort#?jms.useAsyncSend=true');

connection = connectionFactory.createConnection();
connection.start();
// createSession(transacted, session.auto_acknowledge)
javasession = connection.createSession(false, 1);
destination = javasession.createQueue("queuename");
producer = javasession.createProducer(destination);
</cfscript>

<cfloop query="rs">
<cfset mess.agentid = rs.id />
<cfwddx action="cfml2wddx" input="#mess#" output="strMess">

<!--- Send message to activeMQ through our opened channel --->
<cfscript>
message = javasession.createTextMessage(strMess);
producer.send(destination,message);
</cfscript>
</cfloop>

<!--- close up connections with activeMQ now that we are done with the channel --->
<cfscript>
producer.close();
javasession.close();
connection.close();
</cfscript>
Blessings,
Terry

6 comments:

Bas van der Graaf said...

Hi Terry, Thanks for this post!!

Had done some testing with the activeMQ gateway and discovered the same thing. The coldfusion way (sendGatewayMessage) it took me 5 minutes to create 607 messages. Using Java it only took 953ms to create 1000 messages ;-)

Do you use this script in production? How does it perform?

Teeps (Terry Palmer) said...

I've had great success with this in prod. The other day I was able to fill 1.5 million messages in around 10 minutes. If you are doing a large number of messages, make sure that your MemoryLimit is set high enough on the queue so that your producer can push all it needs too. If its too low, it will have to keep waiting for another message to get consumed so that it can put another in. With the direct connection, however, I had an issue with the connection being blamed for being inactive for too long. I found a solution that I will post soon.

Bas van der Graaf said...

Sounds great. Thanks for blogging about this topic ;) Have read your other posts as well.

Do you always send TextMessages with wddx data in it? And do you use the standard coldfusion activeMQ event gateway to read messages from the queue?

Teeps (Terry Palmer) said...

In this case, we did it primarily so that we could read the messages via rss and display to the end user what was left to be run. In this way it was somewhat self documenting and we didn't need to hit the database for additional info. We could have easily just done a numeric id and been ok.

As far as the consumer goes, we do use the AMQ event gateway.

Unknown said...

Hi Terry,

Great Post. Any way to cofigure a activeMQ consumer that relays the inbound messages to a cfc without using Coldfusion gateway. I ask because I am running on Railo and don't have access to the CF Gateways.

Thanks,

Adam

Teeps (Terry Palmer) said...

Yes Adam, there are ways of doing it much the same way I'm using it to publish messages to the queue. However I have not tried it myself. I don't know that you can duplicate the listener behavior of the gateway, but you should be able to do it in a delayed fashion. I would think you could schedule a cf task to run every, say, 5 mins to check for messages in AMQ. Once it finds messages, you could have it process the queue until it is empty. There may be other ways, but that's what comes to mind.