|
Replies:
2
-
Last Post:
Apr 3, 2008 4:58 AM
by: Kshitiz Saxena
|
|
|
|
|
|
|
connection leak detection causes deadlock (buildb58g-fcs)
Posted:
Apr 2, 2008 11:41 PM
|
|
|
We are facing problems with deadlocks in glassfish server 9.1 (build b58g-fcs).
Our application leaks connections, so we enabled connection leak detection. But a timer thread of the leak detection collides with a work thread. Clean shutdown was impossible.
Our parameters:
pool size 64 or 80 Leak Timeout = 300s Leak Reclaim enabled Lazy Association enabled Lazy Connection Enlistment enabled Associate with Thread enabled Match Connections enabled
<quotation from log>
Found one Java-level deadlock: ============================= "RMI TCP Connection(37)-127.0.0.1": waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool), which is held by "httpSSLWorkerThread-39080-1" "httpSSLWorkerThread-39080-1": waiting to lock monitor 0x08ec27f0 (object 0x6289e308, a java.lang.Object), which is held by "Timer-6" "Timer-6": waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool), which is held by "httpSSLWorkerThread-39080-1"
Java stack information for the threads listed above: =================================================== "RMI TCP Connection(37)-127.0.0.1": at com.sun.enterprise.resource.AbstractResourcePool.cancelResizerTask(AbstractResourcePool.java:1782) - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) at com.sun.enterprise.resource.PoolManagerImpl.killPool(PoolManagerImpl.java:573) at com.sun.enterprise.resource.PoolManagerImpl.killAllPools(PoolManagerImpl.java:867) at com.sun.enterprise.server.ApplicationServer.onTermination(ApplicationServer.java:574) at com.sun.enterprise.server.Shutdown.startShutdown(Shutdown.java:76) at com.sun.enterprise.admin.event.AdminEventMulticaster.invokeShutdownEventListener(AdminEventMulticaster.java:1501) at com.sun.enterprise.admin.event.AdminEventMulticaster.handleShutdownEvent(AdminEventMulticaster.java:1490) at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:480) at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176) at com.sun.enterprise.admin.server.core.channel.AdminChannelServer.sendNotification(AdminChannelServer.java:121) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) "httpSSLWorkerThread-39080-1": at com.sun.enterprise.resource.AbstractResourcePool.startConnectionLeakTracing(AbstractResourcePool.java:309) - waiting to lock <0x6289e308> (a java.lang.Object) at com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301) at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778) - locked <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652) at com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136) at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594) at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443) at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248) at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176) at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158) at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108) at de.denic.guiservices.ejb.AuthServiceManagerBean.getRoles(AuthServiceManagerBean.java:60) at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.enterprise.security.SecurityUtil$2.run(SecurityUtil.java:182) at java.security.AccessController.doPrivileged(Native Method) at com.sun.enterprise.security.application.EJBSecurityManager.doAsPrivileged(EJBSecurityManager.java:1041) at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:187) at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986) at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203) at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:77) at $Proxy45.getRoles(Unknown Source) at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:233) at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225) at de.denic.guiservices.__AuthServiceManager_Remote_DynamicStub.getRoles(de/denic/guiservices/__AuthServiceManager_Remote_DynamicStub.java) at de.denic.guiservices._AuthServiceManager_Wrapper.getRoles(de/denic/guiservices/_AuthServiceManager_Wrapper.java) at de.denic.mir.controller.Authorisation.getRegAccIdsWhereMemberIsDenied(Authorisation.java:165) - locked <0x63f4eb38> (a de.denic.mir.controller.Authorisation) at de.denic.mir.controller.MirRegAccController.loadRegAccListTest(MirRegAccController.java:484) at de.denic.mir.controller.MirRegAccController.loadRegAccList(MirRegAccController.java:417) at de.denic.mir.controller.MirRegAccController.init(MirRegAccController.java:153) at org.apache.jsp.de.jsf.mirRegAcc.Init_jsp._jspService(Init_jsp.java from :86) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474) at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:107) at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:122) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:352) at com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:408) at com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:442) at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:115) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212) at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) "Timer-6": at com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:1060) - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344) - locked <0x6289e308> (a java.lang.Object) at com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62) at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1543) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462)
Found 1 deadlock.
|
|
|
|
|
|
|
Re: connection leak detection causes deadlock (buildb58g-fcs)
Posted:
Apr 3, 2008 2:01 AM
in response to: wjp
|
|
|
Looks like this is related to https://glassfish.dev.java.net/issues/show_bug.cgi?id=3960 This will be fixed in 9.1.1.
glassfish@javadesktop.org wrote: > We are facing problems with deadlocks in glassfish server 9.1 (build b58g-fcs). > > Our application leaks connections, so we enabled connection leak detection. But a timer thread of the leak detection collides with a work thread. Clean shutdown was impossible. > > Our parameters: > > pool size 64 or 80 > Leak Timeout = 300s > Leak Reclaim enabled > Lazy Association enabled > Lazy Connection Enlistment enabled > Associate with Thread enabled > Match Connections enabled > > <quotation from log> > > Found one Java-level deadlock: > ============================= > "RMI TCP Connection(37)-127.0.0.1": > waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool), > which is held by "httpSSLWorkerThread-39080-1" > "httpSSLWorkerThread-39080-1": > waiting to lock monitor 0x08ec27f0 (object 0x6289e308, a java.lang.Object), > which is held by "Timer-6" > "Timer-6": > waiting to lock monitor 0x0830c1e4 (object 0x62895970, a com.sun.enterprise.resource.AssocWithThreadResourcePool), > which is held by "httpSSLWorkerThread-39080-1" > > Java stack information for the threads listed above: > =================================================== > "RMI TCP Connection(37)-127.0.0.1": > at com.sun.enterprise.resource.AbstractResourcePool.cancelResizerTask(AbstractResourcePool.java:1782) > - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) > at com.sun.enterprise.resource.PoolManagerImpl.killPool(PoolManagerImpl.java:573) > at com.sun.enterprise.resource.PoolManagerImpl.killAllPools(PoolManagerImpl.java:867) > at com.sun.enterprise.server.ApplicationServer.onTermination(ApplicationServer.java:574) > at com.sun.enterprise.server.Shutdown.startShutdown(Shutdown.java:76) > at com.sun.enterprise.admin.event.AdminEventMulticaster.invokeShutdownEventListener(AdminEventMulticaster.java:1501) > at com.sun.enterprise.admin.event.AdminEventMulticaster.handleShutdownEvent(AdminEventMulticaster.java:1490) > at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:480) > at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176) > at com.sun.enterprise.admin.server.core.channel.AdminChannelServer.sendNotification(AdminChannelServer.java:121) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > "httpSSLWorkerThread-39080-1": > at com.sun.enterprise.resource.AbstractResourcePool.startConnectionLeakTracing(AbstractResourcePool.java:309) > - waiting to lock <0x6289e308> (a java.lang.Object) > at com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301) > at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778) > - locked <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) > at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652) > at com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136) > at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594) > at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443) > at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248) > at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176) > at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327) > at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189) > at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165) > at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158) > at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108) > at de.denic.guiservices.ejb.AuthServiceManagerBean.getRoles(AuthServiceManagerBean.java:60) > at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at com.sun.enterprise.security.SecurityUtil$2.run(SecurityUtil.java:182) > at java.security.AccessController.doPrivileged(Native Method) > at com.sun.enterprise.security.application.EJBSecurityManager.doAsPrivileged(EJBSecurityManager.java:1041) > at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:187) > at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895) > at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986) > at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203) > at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:77) > at $Proxy45.getRoles(Unknown Source) > at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:233) > at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) > at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225) > at de.denic.guiservices.__AuthServiceManager_Remote_DynamicStub.getRoles(de/denic/guiservices/__AuthServiceManager_Remote_DynamicStub.java) > at de.denic.guiservices._AuthServiceManager_Wrapper.getRoles(de/denic/guiservices/_AuthServiceManager_Wrapper.java) > at de.denic.mir.controller.Authorisation.getRegAccIdsWhereMemberIsDenied(Authorisation.java:165) > - locked <0x63f4eb38> (a de.denic.mir.controller.Authorisation) > at de.denic.mir.controller.MirRegAccController.loadRegAccListTest(MirRegAccController.java:484) > at de.denic.mir.controller.MirRegAccController.loadRegAccList(MirRegAccController.java:417) > at de.denic.mir.controller.MirRegAccController.init(MirRegAccController.java:153) > at org.apache.jsp.de.jsf.mirRegAcc.Init_jsp._jspService(Init_jsp.java from :86) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) > at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) > at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470) > at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) > at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) > at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) > at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855) > at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703) > at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542) > at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474) > at org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:107) > at org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:122) > at java.security.AccessController.doPrivileged(Native Method) > at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:352) > at com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:408) > at com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:442) > at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:115) > at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) > at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) > at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) > at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) > at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) > at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) > at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290) > at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) > at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212) > at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361) > at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) > at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) > "Timer-6": > at com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:1060) > - waiting to lock <0x62895970> (a com.sun.enterprise.resource.AssocWithThreadResourcePool) > at com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344) > - locked <0x6289e308> (a java.lang.Object) > at com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62) > at com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1543) > at java.util.TimerThread.mainLoop(Timer.java:512) > at java.util.TimerThread.run(Timer.java:462) > > Found 1 deadlock. > [Message sent by forum member 'wjp' (wjp)] > > http://forums.java.net/jive/thread.jspa?messageID=267464 > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net > For additional commands, e-mail: users-help@glassfish.dev.java.net > >
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
Re: connection leak detection causes deadlock (buildb58g-fcs)
Posted:
Apr 3, 2008 4:58 AM
in response to: Shalini Muthukr...
|
|
|
Please disable connection reclaim and this deadlock will not happen. So it will display all connection leaks, however will not reclaim any leaking connection. In such a case, connection may get exhausted due to connection leaks. Fix for bug 3960 is not available right now. Meanwhile you can fix your application to avoid connection leaks.
Thanks, Kshitiz
Shalini Muthukrishnan wrote: > Looks like this is related to > https://glassfish.dev.java.net/issues/show_bug.cgi?id=3960 > This will be fixed in 9.1.1. > > glassfish@javadesktop.org wrote: >> We are facing problems with deadlocks in glassfish server 9.1 >> (build b58g-fcs). >> >> Our application leaks connections, so we enabled connection leak >> detection. But a timer thread of the leak detection collides with a >> work thread. Clean shutdown was impossible. >> >> Our parameters: >> >> pool size 64 or 80 >> Leak Timeout = 300s >> Leak Reclaim enabled >> Lazy Association enabled >> Lazy Connection Enlistment enabled >> Associate with Thread enabled >> Match Connections enabled >> >> <quotation from log> >> >> Found one Java-level deadlock: >> ============================= >> "RMI TCP Connection(37)-127.0.0.1": >> waiting to lock monitor 0x0830c1e4 (object 0x62895970, a >> com.sun.enterprise.resource.AssocWithThreadResourcePool), >> which is held by "httpSSLWorkerThread-39080-1" >> "httpSSLWorkerThread-39080-1": >> waiting to lock monitor 0x08ec27f0 (object 0x6289e308, a >> java.lang.Object), >> which is held by "Timer-6" >> "Timer-6": >> waiting to lock monitor 0x0830c1e4 (object 0x62895970, a >> com.sun.enterprise.resource.AssocWithThreadResourcePool), >> which is held by "httpSSLWorkerThread-39080-1" >> >> Java stack information for the threads listed above: >> =================================================== >> "RMI TCP Connection(37)-127.0.0.1": >> at >> com.sun.enterprise.resource.AbstractResourcePool.cancelResizerTask(AbstractResourcePool.java:1782) >> >> - waiting to lock <0x62895970> (a >> com.sun.enterprise.resource.AssocWithThreadResourcePool) >> at >> com.sun.enterprise.resource.PoolManagerImpl.killPool(PoolManagerImpl.java:573) >> >> at >> com.sun.enterprise.resource.PoolManagerImpl.killAllPools(PoolManagerImpl.java:867) >> >> at >> com.sun.enterprise.server.ApplicationServer.onTermination(ApplicationServer.java:574) >> >> at >> com.sun.enterprise.server.Shutdown.startShutdown(Shutdown.java:76) >> at >> com.sun.enterprise.admin.event.AdminEventMulticaster.invokeShutdownEventListener(AdminEventMulticaster.java:1501) >> >> at >> com.sun.enterprise.admin.event.AdminEventMulticaster.handleShutdownEvent(AdminEventMulticaster.java:1490) >> >> at >> com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:480) >> >> at >> com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:176) >> >> at >> com.sun.enterprise.admin.server.core.channel.AdminChannelServer.sendNotification(AdminChannelServer.java:121) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> >> at java.lang.Thread.run(Thread.java:619) >> "httpSSLWorkerThread-39080-1": >> at >> com.sun.enterprise.resource.AbstractResourcePool.startConnectionLeakTracing(AbstractResourcePool.java:309) >> >> - waiting to lock <0x6289e308> (a java.lang.Object) >> at >> com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301) >> >> at >> com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778) >> >> - locked <0x62895970> (a >> com.sun.enterprise.resource.AssocWithThreadResourcePool) >> at >> com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652) >> >> at >> com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136) >> >> at >> com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594) >> >> at >> com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443) >> >> at >> com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248) >> >> at >> com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176) >> >> at >> com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327) >> >> at >> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189) >> >> at >> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165) >> >> at >> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158) >> >> at >> com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108) >> at >> de.denic.guiservices.ejb.AuthServiceManagerBean.getRoles(AuthServiceManagerBean.java:60) >> >> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> com.sun.enterprise.security.SecurityUtil$2.run(SecurityUtil.java:182) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> com.sun.enterprise.security.application.EJBSecurityManager.doAsPrivileged(EJBSecurityManager.java:1041) >> >> at >> com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:187) >> at >> com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895) >> >> at >> com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986) >> at >> com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:203) >> >> at >> com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:77) >> >> at $Proxy45.getRoles(Unknown Source) >> at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:233) >> >> at >> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) >> >> at >> com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225) >> >> at >> de.denic.guiservices.__AuthServiceManager_Remote_DynamicStub.getRoles(de/denic/guiservices/__AuthServiceManager_Remote_DynamicStub.java) >> >> at >> de.denic.guiservices._AuthServiceManager_Wrapper.getRoles(de/denic/guiservices/_AuthServiceManager_Wrapper.java) >> >> at >> de.denic.mir.controller.Authorisation.getRegAccIdsWhereMemberIsDenied(Authorisation.java:165) >> >> - locked <0x63f4eb38> (a de.denic.mir.controller.Authorisation) >> at >> de.denic.mir.controller.MirRegAccController.loadRegAccListTest(MirRegAccController.java:484) >> >> at >> de.denic.mir.controller.MirRegAccController.loadRegAccList(MirRegAccController.java:417) >> >> at >> de.denic.mir.controller.MirRegAccController.init(MirRegAccController.java:153) >> >> at >> org.apache.jsp.de.jsf.mirRegAcc.Init_jsp._jspService(Init_jsp.java >> from :86) >> at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) >> at >> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) >> >> at >> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470) >> at >> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) >> at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) >> at >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) >> >> at >> org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) >> >> at >> org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855) >> >> at >> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703) >> >> at >> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542) >> >> at >> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474) >> >> at >> org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:107) >> >> at >> org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:122) >> >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:352) >> >> at >> com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:408) >> >> at >> com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(ViewHandlerImpl.java:442) >> >> at >> com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:115) >> >> at >> com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) >> >> at >> com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) >> at >> com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) >> at >> javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) >> at sun.reflect.GeneratedMethodAccessor358.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:276) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) >> at >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:192) >> >> at >> org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:404) >> >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290) >> >> at >> org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) >> >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) >> >> at >> com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) >> >> at >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) >> >> at >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) >> >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) >> >> at >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) >> >> at >> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) >> at >> org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270) >> at >> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) >> >> at >> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) >> >> at >> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) >> >> at >> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339) >> >> at >> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261) >> >> at >> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212) >> >> at >> com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361) >> >> at >> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) >> at >> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) >> >> "Timer-6": >> at >> com.sun.enterprise.resource.AbstractResourcePool.freeResource(AbstractResourcePool.java:1060) >> >> - waiting to lock <0x62895970> (a >> com.sun.enterprise.resource.AssocWithThreadResourcePool) >> at >> com.sun.enterprise.resource.AbstractResourcePool.potentialConnectionLeakFound(AbstractResourcePool.java:344) >> >> - locked <0x6289e308> (a java.lang.Object) >> at >> com.sun.enterprise.resource.AbstractResourcePool.access$000(AbstractResourcePool.java:62) >> >> at >> com.sun.enterprise.resource.AbstractResourcePool$ConnectionLeakTask.run(AbstractResourcePool.java:1543) >> >> at java.util.TimerThread.mainLoop(Timer.java:512) >> at java.util.TimerThread.run(Timer.java:462) >> >> Found 1 deadlock. >> [Message sent by forum member 'wjp' (wjp)] >> >> http://forums.java.net/jive/thread.jspa?messageID=267464 >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net >> For additional commands, e-mail: users-help@glassfish.dev.java.net >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net > For additional commands, e-mail: users-help@glassfish.dev.java.net >
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|