ActiveMQ: Channel your energy

I started noticing an error when trying to fill my queue with a large number of messages. The error was the following: Channel was inactive for too long. This would happen anywhere from one minute after the producer and channel were created to as long as ten minutes. It wasn't all that consistent. After doing a little research, I found a solution to turn off the inactivity check.

<cfscript>
connectionFactory = createobject(
'java','org.apache.activemq.ActiveMQConnectionFactory'
).init(
  '',
  '',
  'tcp://{host}:{port}
         ?jms.useAsyncSend=true
         &wireFormat.maxInactivityDuration=0'
  );
connection = connectionFactory.createConnection();
connection.start();

....

</cfscript>

By appending &wireFormat.maxInactivityDuration=0, it will turn off the inactivity check for the channel. You should also do this on the AMQ server side by doing the following in your ActiveMQ.xml:

<transportConnectors>
<transportConnector name="openwire"
uri="tcp://{host}:{port}/?wireFormat.maxInactivityDuration=0"
discoveryUri="multicast://default"/>

...

</transportConnectors>

If you are experiencing issues like this with your CF ActiveMQ gateways, you could likely add this to your gateway config file. However, I have yet to see this error in connection with the CF ActiveMQ gateway. For more info on connecting directly to ActiveMQ, see my post Straight talk with ActiveMQ.

ActiveMQ: Ghost consumers just in time for Halloween

I have been noticing some issues with the communication between ActiveMQ and Coldfusion the higher our load gets. We've got a large process that can cause a java heap space error in the environment. If that happens when there is a coldfusion gateway consuming messages, it can cause the current consumer to go into ghost mode and a new consumer is created. If this is allowed to persist, the communication between ActiveMQ and Coldfusion sometimes comes to a screeching halt. The only way we have found to remedy this battle of digital wills is to restart the Coldfusion service. This seems to properly reset the connections and things return to normal.

So, what is causing this behavior? It's a question I wrestled with for a few weeks. I could not replicate this in my local environment at all. It helped to get jconsole fired up so that I could see a little bit of what was going on behind the scenes beyond what the ActiveMQ web console shows. One thing I noticed with the actual filling of the queue is that if the MemoryLimit is not set high enough, the MemoryPercentUsage will be maxed at 100% and you can only put so many messages in the queue. A new message can only get added if one in the queue got consumed. This can be a really long Coldfusion task if you are max'd at 4000 messages and you have to fill it with 500,000. With jconsole you can edit some of the limits on the fly for the queue.



The queue size at the time was 5m. I started bumping it up to see what effect it would have. I went to 10m and the queue doubled in size, but still wasn't where I wanted it. I bumped it to 20m and wooosh, the queue filled with all 500,000 messages. There seems to be some threshold that needs to be hit in order for it to hold that many messages. With 500,000 messages in the queue, the MemoryPercentUsage is only at 43% or so. I'm sure an ActiveMQ guru could explain what is going on there. If you know, please leave a comment. Now rather than the Coldfusion process taking 12 hrs to fill the queue, it fills in 10 minutes.

So, let's make this MemoryLimit change permanent. Open up your activemx.xml in your /conf directory and adjust as needed:

<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="20mb"/>

I think the default memory limit for ActiveMQ off the shelf is 5m. If you are doing a higher quantity of messages over 5000, it seems that you will have to bump this number up.

Now the cool thing about this is that our hanging consumer issue has disappeared. This was an unexpected benefit. I decided to test it on my local environment by dropping the MemoryLimit down to 48k. This allowed a max of 42 messages in the queue. I then wrote the following code to create a java heap space error:

<cffunction name="javaheapspace" output="false" access="public" returntype="void">
   <cfset var myxml =  ''/>
<cfset var parsedxml =  ''/>
<cfset var i = 0 />
<cfset var j = 0 />  

<cfsavecontent variable="myxml">
   <rootnode>
   <cfloop from="1" to ="299999" index="i">
        <subnode> 
             <textinfo>with text in here</textinfo>
                  <mynumber>#i#</mynumber>
        </subnode>
   </cfloop>
   </rootnode>
</cfsavecontent>
<cfset parsedxml = xmlparse(myxml) /> 
</cffunction>

The loop count may have to be moved up or down depending on the amount of memory you have allocated to your JVM. I think mine is up around 512m. I wanted it just small enough to be able to finish the myxml variable creation and just big enough to die in the parse.

So in order to replicate this issue, I put in motion the process that would begin filling the queue and make sure the consumption begins. With jconsole open, I watched the MemoryPercentageUsage and waited until it hit 100%. Then I fired off the javaheapspace() call in another script. Once it bombed, I refreshed my jconsole view and, sure enough, the consumer count went from 1 to 2.



It doesn't work every time so you may have to try a few times. The more time it sits at 100% usage, the better your chances. I went into CFAdmin and shut off the gateway that was my consumer and the count went down to one. So it left a connection that cannot be managed. In fact if you try to kill that connection through jconsole, it will create another one it its place. The only way to kill it that I have found, is to restart the CF instance.

So the moral of the story is to keep your queue's MemoryLimit high enough so that your MemoryPercentUsage does not reach 100%. Otherwise you may be haunted by the "ghost of consumer past"

Blessings,
Terry

JConsole has consoled me

A long and hard battle was fought tonight and finally victory is mine! I set out to get a little further down the road of my understanding of ActiveMQ. I have read all over the place that you can monitor AMQ with jconsole. It should be so easy right?
Type jconsole at the command prompt and slap service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi in the remote process box and whamo.

Well "no soup for me". In the words of the Pirates who don't do anything, "nothing, zilch, nodda". At first I was getting a connection timeout and would get absolutely nowhere. After a flurry of changes, it would actually connect but I didn't see the org.apache.activeMQ in the list on the left.

I was looking through my activemq.xml in my conf directory and noticed this:

<managementcontext>
<managementcontext createconnector="false">
</managementcontext>

Ok... lets turn that puppy on. Now I start seeing this in the startup output of ActiveMQ:

WARNING: Failed to start jmx connector:
Cannot bind to URL[rmi://localhost:1099/jmxrmi]
:javax.naming.NameAlreadyBoundException:
jmxrmi [Rootexception is
 java.rmi.AlreadyBoundException: jmxrmi]   

So... at least I saw something remotely familiar to me that is supposed to go in the Remote Process box. But it seems that something else is on that port. I started thinking about what else on my system might be using jmx. I am running Apache Solr. Let's shut that down and see if it starts up. Nope. Nodda. Oh wait, how about Jrun. There was coldfusion and jrun stuff in the left pane of jconsole. After I shut down jrun and restarted ActiveMQ, finally it magically appeared. A thing of beauty.

INFO  ManagementContext - JMX consoles can connect
to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

Now we still have an issue. Obviously if I'm going to be using coldfusion and ActiveMQ together, they need to learn to get along. All we have to do is add an attribute to the managementContext node in conf/activemq.xml:

<managementcontext>
<managementcontext createconnector="true" connectorport="1199">
</managementcontext>

Now restart ActiveMQ and start coldfusion back up and you should be able to view ActiveMQ behind the scenes through jconsole. Now were cookin with gas.

Ultimately I'd like to be able to connect to JMX via java in CF to get things like queue count and connection status. We've been seeing some strangeness when CF dies due to java heap space errors that CF and AMQ will just stop talking to each other after the instance comes back up. If I cycle the CF instance, then consumer counts go back to normal and they start talking again. Communication is key to any good relationship, especially when clients are riding on it.

Speaking of communication, how has your's been with Jesus? Your eternity is riding on it.

Blessings,
Terry

CFCatch me if you can

One thing that really annoys me with the cfcatch struct/object (or whatever its pretending to be) is trying to use one of the keys and have it not exist. There have been a few times where I simply want to log cfcatch.detail and it's undefined, throwing a hard error. To me, if it's documented that it is going to be available, it should at least be available as an empty string. I set out this morning looking for a clean way of param'ing some of the keys in cfcatch. Since it doesn't seem to be a true struct, you can't simply do:

<cfparam name="cfcatch.detail" default="" />

That would be way too easy. I found a couple places that suggested using duplicate() to make it act like a struct. I tried that too, and you still can't do a cfparam on it. The key simply won't exist.

After much googling, I came across this post by Isaac Dealey. So using that idea, I created the following function:

<cffunction name="structifyCatch" returntype="struct">
   <cfargument name="catch" type="any" required="true"/>

   <cfset var err = structnew() /> 
   <cfset structappend(err,arguments.catch,true) />

   <cfreturn err />

  </cffunction>

then to implement it, you call the following:

<cftry>
   ... some code ...
   <cfcatch>
     <cfset cfcatch = structifyCatch(cfcatch) />
     <cfparam name="cfcatch.message" default=""/>
     <cfparam name="cfcatch.detail" default=""/>
     <cflog text="There was an error in somecode().
        Message: #cfcatch.message# Detail: #cfcatch.Detail#"
       application="yes">
   </cfcatch>
 </cftry>

Hopefully this will save you a little bit of angst.
There are some interesting behaviors documented here too:
cfcatch was born a struct, then it wasn’t but now it’s back

Blessings,
Terry

Straight talk with ActiveMQ

I found that filling an ActiveMQ queue through a cfml gateway was extremely inefficient when sending a large number of messages at once. Here is the code to send messages directly to the queue via java. This was a little difficult to find this documented anywhere. I had to pull in bits an pieces from different code examples.

<cfset mess = structnew() />

<cfquery name="rs" datasource="dsn">
select id from mytable
</cfquery>

<cfscript>
 connectionFactory = createobject(
           'java'
           ,'org.apache.activemq.ActiveMQConnectionFactory'
 ).init('','', 'tcp://#amqServer#:#amqPort#?jms.useAsyncSend=true');

 connection = connectionFactory.createConnection();
 connection.start();
 // createSession(transacted, session.auto_acknowledge)
 javasession = connection.createSession(false, 1);
 destination = javasession.createQueue("queuename");
 producer = javasession.createProducer(destination);
</cfscript>

<cfloop query="rs">
<cfset mess.agentid = rs.id />
 <cfwddx action="cfml2wddx" input="#mess#" output="strMess">

 <!--- Send message to activeMQ through our opened channel --->
 <cfscript>
    message = javasession.createTextMessage(strMess);
    producer.send(destination,message);
 </cfscript>
</cfloop>

<!--- close up connections with activeMQ now that we are done with the channel --->
<cfscript>
 producer.close(); 
 javasession.close();
 connection.close();
</cfscript>

Blessings,
Terry

Lost in translation: The Encoding Experience

I had written an undelivr folder viewer about a month back. We started testing some special chars through our mailing app recently and found that Programów was becoming Programów. We figured it had something to do with encoding. I checked the cfadmin setting for the default mail encoding and it was UTF-8. That should have been ok. I tried hardcoding charset="UTF-8" in the cfmail tag to see if that would help. Same results.

So then I switched gears to my local dev server and wrote up this quick script.

<cfset str ="Programów"/>
<cfmail from="a@b.com" to="a@c.com" subject="1.0" charset="utf-8">
#str#
</cfmail>

This works perfectly fine in my local dev. When I threw this script on the server it still got mangled. So now we're dealing with an environment issue. I'm running Ubuntu locally and we are running windows on the server. So I started digging into the settings in cfadmin. I took the settings summary from both servers and threw it into Kompare. One of the differences I saw right away was the Java File Encoding. On the server it was set to cp1252 and on my Linux it was UTF-8. After much searching, I was able to track down how to change the JVM's default file encoding here: Java's file.encoding property on Windows platform. After adding -Dfile.encoding=UTF-8 to the jvm's server.args (typically found in jrun4/bin/jvm.conf), I restarted the instance and re-ran the test. Bingo, the special chars were preserved.

There must be an issue with some special chars moving from UTF-8 to cp1252 or vice versa. I just ran a test and this does affect all emails if you are spooling to disk. My assumption is that under the covers, java is writing the file using the default JVM file encoding, not the default coding set for the cfmail tag. This makes the cfmail default setting a little misleading. At any rate, setting the JVM encoding to UTF-8 did the trick.

Blessings,
Terry

Hot under the hotfix..

We ran into an error today on one of our dev servers.

coldfusion.runtime.J2eeSessionScopeStub cannot
be cast to coldfusion.runtime.J2eeSessionScopeStub

We had recently patched that server with "Cumulative Hot Fix 3 for ColdFusion 8.0.1" That was really the only thing that changed with the server before getting the error. I went into JRun4\servers\{instancename}\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\updates and removed chf8010003.jar. After restarting the instance, things came up normal. I thought that maybe the file we downloaded was corrupt so I grabbed a fresh copy. That file didn't work either. The head scratcher in this is that we installed the patch on other instances with no problems. Not sure what is so unique with this instance, but it just doesn't jive with the new hotfix.

here is the stack:
18840 java.lang.ClassCastException: coldfusion.runtime.J2eeSessionScopeStub cannot be cast to coldfusion.runtime.J2eeSessionScopeStub
18841 at coldfusion.runtime.SessionTracker.getSession(SessionTracker.java:76)
18842 at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:528)
18843 at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:592)
18844 at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:235)
18845 at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
18846 at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
18847 at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
18848 at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
18849 at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
18850 at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
18851 at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
18852 at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
18853 at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
18854 at coldfusion.CfmServlet.service(CfmServlet.java:175)
18855 at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
18856 at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
18857 at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
18858 at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
18859 at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
18860 at jrun.servlet.FilterChain.service(FilterChain.java:101)
18861 at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
18862 at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
18863 at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
18864 at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
18865 at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
18866 at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
18867 at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Keepass Conundrum

I'm running Keepass on Kubuntu 8.10. Somewhere along the way Keepass stopped working for me. I could open the program and log in successfully. I could even see the groups on the left hand side of the window. What I could NOT see were the entries with their logins and passwords on the right. All I would get was a blank gray pane on the right. I tried a bunch of things including upgrading to KeepassX 0.4.0. Nothing worked. Fortunately there is a very simple solution to this. If you are experiencing the same thing, look very closely at the bottom of the right hand pane. You will see three little dots. Click that and drag upwards. Voila! There are all of your entries that you missed so dearly. Not sure if this was a bug with KDE and Keepass or what, but I sure was glad to find the fix on a blog somewhere. Thought I'd share the love here too.

Blessings,
Teeps

Eclipse Annoyance

I truely appreciate eclipse as the ide for CF development. I've had a slight annoyance with it though since recently moving to a linux dev environment. In short, everytime I did a text search.... I would get an annoying little error box popping up saying:



Looking at the details of the error... I see the following:
"resource is out of sync with the file system"

After digging around a bit on google I found that this generally occurs when a file is changed/added outside of eclipse and it has, just as the error states, become out of sync with the file system. The solution is to refresh the project so that it picks up the changed/new files. If outside processes frequently change files outside of your eclipse, you can have eclipse automatically refresh the project by going to Window > Preferences and in General > Workspace. Check the box that says "Refresh Automatically". We use the fusebox framework which compiles the code into the parsed folder. So in development mode, the parsed folder has changed/new files all the time.

Well... I made this change and it still didn't work for me. I don't know if its something unique with linux or just a bug or what, but refreshing the parsed folder and setting refresh automatically did absolutely nothing to resolve the issue.

Then I had a duh moment. For my text search, I have it search in a working set. Why don't I just exclude the parsed folder from the working set. I don't really need to look in the parsed folder anyway. Sheesh, sometimes we can make things way harder than they need to be. So... Search > File... (or Cntrl-H for me) Then on the "Working set:" line, click "Choose...". I'm working with "Selected Working Sets". I edited my working set and under the "Working set contents" area I drilled down to the parsed folder in the relavent resource and unchecked it. Bingo... no more errors.

Hope this helps...
God Bless