|
Replies:
17
-
Last Post:
Jun 26, 2009 8:00 AM
by: nigeldeakin
|
|
|
|
|
|
|
Delivery of JMS message in case of distributed transaction
Posted:
Jun 18, 2009 8:58 AM
|
|
|
I have a general question about distributed transactions, and how other people deal with them in the real world.
We use in our of our project an Oracle database plus a few JMS queues. We have SLSB in charge to insert some data into the database then send a JMS message for asynchronous processing.
This works fine, but we have realized that sometimes the message is delivered before the changes in database were committed. This means that we sometimes read stale data, which is a serious issue. Our workaround is to lock the row before reading (or use "select ... for update") so that the message processing will wait until the original transaction is completed.
This seems to me like a basic scenario and I guess that other people also faced a similar problem. I'm wondering how other people solved this issue?
Message was edited by: ewernli
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 18, 2009 10:37 AM
in response to: ewernli
|
|
|
Are your database updates and JMS sends performed within a single global transaction? By default, JMS sends within EE components running in a global transaction have transactional semantics. That means that the even though control returns from the JMS API the message should not actually be forwarded to the target destination until the transaction commits. That
also means any other transactional work in the same transaction, like database updates, should
have committed before a consumer could actually receive the message.
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 18, 2009 5:43 PM
in response to: ewernli
|
|
|
Conceptually I understand your problem is caused because the JMS update
is faster than the database update, but I don't know of anyone in the
real world who is having a similar issue.
Actually, I'm not 100% sure why you are ever seeing this issue ..
Are you *always* performing the database operation first ??? (you have
to if you want to guarantee the right behavior)
Are you running in a transaction ??? (not required, although it may help
to pinpoint an issue)
Here is my confusion .... I can see two possible ways to do this ...
- Run outside of a transaction:
- update the table (should happen before the request returns
since its not transacted)
- send the JMS messages [which will arrive after the table is
updated since the update was done before it was sent]
- Run inside of a transaction:
- updating a database table
- sending a JMS message
- prepare for database then JMS message run (XA only)
- commit for database run (database updated)
- commit for JMS runs (and message is then sent, this occurs
AFTER the database has committed)
Obviously, in both cases the messages are processed in order. (unless
somehow the order for the commits does not match what I'd expect)
Can you give me more information about the order you are doing
processing and transaction type ??
-- Linda
glassfish@javadesktop.org wrote:
> I have a general question about distributed transactions, and how other people deal with them in the real world.
>
> We use in our of our project an Oracle database plus a few JMS queues. We have SLSB in charge to insert some data into the database then send a JMS for asynchronous processing.
>
> This works fine, but we have realized that sometimes the message is delivered before the changes in database were committed. This means that we sometimes read stale data, which is a serious issue. Our workaround is to lock the row before reading (or use "select ... for update") so that the message processing will wait until the original transaction is completed.
>
> This seems to me like a basic scenario and I guess that other people also faced a similar problem. I'm wondering how other people solved this issue?
> [Message sent by forum member 'ewernli' (ewernli)]
>
> http://forums.java.net/jive/thread.jspa?messageID=351867
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 19, 2009 1:16 AM
in response to: Linda Schneider
|
|
|
Thanks for your feedback.
The EJB that updates the database and sends the message is running within a distributed transaction. From the bean point of view the transactional semantics is ok: the database update and the message sending are committed or rolled back together when the transaction completes. No message is ever delivered if transaction is rolled back.
We first update the database then send the JMS message. We are using JPA to load and update the entity in database, so it's hard to say when the database changes are actually flushed, and if it impacts the order of events in the 2 phase commit protocol.
At a point in time, there are two participants enlisted in the transaction manager (the database connection and the JMS queue, both XAResource). I don't know how the transaction manager decides the order of the prepare/commit sequence. The transaction manager doesn't know the nature of the participants and doesn't know what operations were performed on each participant. From my understanding the order shouldn't matter and it could be either database-JMS or JMS-database.
Unfortunately, if it turns out that the transaction manager commits first the JMS then the database, the message may be delivered before the database changes are done.
The problem is reproducible, but not systematic. This could be explained either because of the delivery delay, or because the order in the 2 phase commit is not deterministic.
|
|
|
|
|
|
|
|
|
RE: Delivery of JMS message in case of distributed transaction
Posted:
Jun 19, 2009 3:48 AM
in response to: ewernli
|
|
|
|
|
In that case I would suggest installing and configuring a X/Open Distributed Transaction Processor
such as Oracle's Transaction Process Manager to quote
"<the TPM> uses Oracle XA library subroutines to tell Oracle Database how to process
the transaction, based on its knowledge of all RMs in the transaction."
Oracle handles the coordination from multiple registering requesting entities as Resource Managers to TPM
the TPM Transaction Manager coordinates request activity to all RMs by either committing or rolling back all
Transaction requests from RM participants
http://stanford.edu/dept/itss/docs/oracle/10g/appdev.101/b10795/adfns_xa.htm#1015699
HTH
Martin
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Fri, 19 Jun 2009 01:16:46 -0700
> From: glassfish@javadesktop.org
> To: users@glassfish.dev.java.net
> Subject: Re: Delivery of JMS message in case of distributed transaction
>
> Thanks for your feedback.
>
> The EJB that updates the database and sends the message is running within a distributed transaction. From the bean point of view the transactional semantics is ok: the database update and the message sending are committed or rolled back together when the transaction completes. No message is ever delivered if transaction is rolled back.
>
> We first update the database then send the JMS message. We are using JPA to load and update the entity in database, so it's hard to say when the database changes are actually flushed, and if it impacts the order of events in the 2 phase commit protocol.
>
> At a point in time, there are two participants enlisted in the transaction manager (the database connection and the JMS queue, both XAResource). I don't know how the transaction manager decides the order of the prepare/commit sequence. The transaction manager doesn't know the nature of the participants and doesn't know what operations were performed on each participant. From my understanding the order shouldn't matter and it could be either database-JMS or JMS-database.
>
> Unfortunately, if it turns out that the transaction manager commits first the JMS then the database, the message may be delivered before the database changes are done.
>
> The problem is reproducible, but not systematic. This could be explained either because of the delivery delay, or because the order in the 2 phase commit is not deterministic.
> [Message sent by forum member 'ewernli' (ewernli)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352031
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage_062009
[att1.html]
|
|
|
|
|
|
|
|
|
Re: RE: Delivery of JMS message in case of distributed transaction
Posted:
Jun 19, 2009 9:52 AM
in response to: Martin Gainty
|
|
|
You mean to switch the built-in Transaction Manager of Glassfish for another one? I don't know if this is possible.
My problem boils down to the fact that there is no way to make a distributed transaction absolutely atomic. At a point in time, the transaction manager will call commit on resource A, then on resource B. The interval between both calls can be extremely short, but there will always be a moment where A is committed and B isn't.
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 19, 2009 10:44 AM
in response to: ewernli
|
|
|
Can it be a database bug that the data is not locked on flush in
beforeCompletion? What are you using for JPA?
You can also try to use a non-XA resource for JPA (if there is only 1 database
involved) - this is supported as "last-agent-optimization". In which case the
database commit will happen before JMS.
Regards,
-marina
glassfish@javadesktop.org wrote:
> Thanks for your feedback.
>
> The EJB that updates the database and sends the message is running within a distributed transaction. From the bean point of view the transactional semantics is ok: the database update and the message sending are committed or rolled back together when the transaction completes. No message is ever delivered if transaction is rolled back.
>
> We first update the database then send the JMS message. We are using JPA to load and update the entity in database, so it's hard to say when the database changes are actually flushed, and if it impacts the order of events in the 2 phase commit protocol.
>
> At a point in time, there are two participants enlisted in the transaction manager (the database connection and the JMS queue, both XAResource). I don't know how the transaction manager decides the order of the prepare/commit sequence. The transaction manager doesn't know the nature of the participants and doesn't know what operations were performed on each participant. From my understanding the order shouldn't matter and it could be either database-JMS or JMS-database.
>
> Unfortunately, if it turns out that the transaction manager commits first the JMS then the database, the message may be delivered before the database changes are done.
>
> The problem is reproducible, but not systematic. This could be explained either because of the delivery delay, or because the order in the 2 phase commit is not deterministic.
> [Message sent by forum member 'ewernli' (ewernli)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352031
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 19, 2009 3:52 PM
in response to: Marina Vatkina
|
|
|
Our JPA implementation is Hibernate 3. The synchronization works and the dirty entites are flushed before the 2PC starts - I suppose that it's using JTA's synchronization facility under the hood.
I didn't thought about the last-agent-optimization. However, based on my understanding, JMS should be the one to be non-XA. In this case JMS will always be the last participant and this forces the database commit to happen first.
I checked again the JTA, JCA and JMS specs briefly, and I couldn't find any information about how the transaction manager is supposed to order/prioritize the RM enlisted in the distributed transaction.
* Does the TM prepare and commit the RM in the same order they were enlisted?
* Is there some settings somewhere where we can assign a priority to the RM ?
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 22, 2009 1:45 PM
in response to: ewernli
|
|
|
LAO (last-agent-optimization).commit() happens at the end of prepare() phase, so
that the result of it can be used to commit or rollback the XA resources.
Regards,
-marina
glassfish@javadesktop.org wrote:
> Our JPA implementation is Hibernate 3. The synchronization works and the dirty entites are flushed before the 2PC starts - I suppose that it's using JTA's synchronization facility under the hood.
>
> I didn't thought about the last-agent-optimization. However, based on my understanding, JMS should be the one to be non-XA. In this case JMS will always be the last participant and this forces the database commit to happen first.
>
> I checked again the JTA, JCA and JMS specs briefly, and I couldn't find any information about how the transaction manager is supposed to order/prioritize the RM enlisted in the distributed transaction.
> * Does the TM prepare and commit the RM in the same order they were enlisted?
> * Is there some settings somewhere where we can assign a priority to the RM ?
> [Message sent by forum member 'ewernli' (ewernli)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352172
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 23, 2009 5:43 AM
in response to: Marina Vatkina
|
|
|
LAO will cause the non-XA resource to be committed first so for this to work the DB will need to be the non-XA one.
The XA protocol doesn't dictate order of prepare's or commit's of RMs, so any ability to do so would have to be defined in the JTA spec or be app.server specific (I have no idea what JTA or GF provides), but this might be a good way to go if GF supports it.
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 23, 2009 4:11 PM
in response to: matterbury
|
|
|
In GF it's a List.
-marina
glassfish@javadesktop.org wrote:
> LAO will cause the non-XA resource to be committed first so for this to work the DB will need to be the non-XA one.
>
> The XA protocol doesn't dictate order of prepare's or commit's of RMs, so any ability to do so would have to be defined in the JTA spec or be app.server specific (I have no idea what JTA or GF provides), but this might be a good way to go if GF supports it.
> [Message sent by forum member 'matterbury' (matterbury)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352497
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 23, 2009 1:36 AM
in response to: ewernli
|
|
|
Hi there, I am experiencing the same issue. I'm Using JPA [hibernate], Glassfish, Spring and i have an MDB.
1] I create a persistent object @Entity, call create [and call entitymanager.flush() even...]
2] I then retrieve the DB generated primary key and put it in the message and put it on the Queue.
3] The async MDB fires and receives the message
4] The ID is retrieved from the message and uses hibernate to load the object by id.
5] Once i have the object i do some updates on it and call merge.
All seems ok until i exit onMessage(). 100% of the time i get an optimisitc lock exception on the object, the first time the message is processed. The message is redelivered after an interval and its process correctly without error the 2nd time. I can confirm that i have put @version on an Integer, not a date that loses precision etc...
Any suggestions would be apreciated. This is proving very difficult to resolve.
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 23, 2009 8:59 AM
in response to: gregjh
|
|
|
We had also the problem when we insert a new entity. What I don't understand then is how come that when the message is delivered you "see" the row not yet committed in database.
1) new row is inserted (new entity created + flush)
2) async msg MDB fires and receives the message
3) because message delivery happens in a concurrent transaction, the changes of the original transaction are not visible, and entity can not be loaded
4) insert is comitted and becomes visible to the other transactions.
Are you using some kind of second-level cache?
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 23, 2009 4:04 PM
in response to: gregjh
|
|
|
Can it be that there is caching involved, and the cache is not yet updated by
the time the transaction commits?
Does MDB share the same PU as the component that creates an entity?
thanks,
-marina
glassfish@javadesktop.org wrote:
> Hi there, I am experiencing the same issue. I'm Using JPA [hibernate], Glassfish, Spring and i have an MDB.
>
> 1] I create a persistent object @Entity, call create [and call entitymanager.flush() even...]
> 2] I then retrieve the DB generated primary key and put it in the message and put it on the Queue.
> 3] The async MDB fires and receives the message
> 4] The ID is retrieved from the message and uses hibernate to load the object by id.
> 5] Once i have the object i do some updates on it and call merge.
>
> All seems ok until i exit onMessage(). 100% of the time i get an optimisitc lock exception on the object, the first time the message is processed. The message is redelivered after an interval and its process correctly without error the 2nd time. I can confirm that i have put @version on an Integer, not a date that loses precision etc...
>
> Any suggestions would be apreciated. This is proving very difficult to resolve.
> [Message sent by forum member 'gregjh' (gregjh)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352466
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 26, 2009 7:06 AM
in response to: ewernli
|
|
|
|
|
Dear all,
I've created a sample EAR to illustrate the problem. Could anyone let me know if the problem is reproducible somewhere else?
1) Create the table
CREATE TABLE COUNTER
(
COUNTERID NUMBER,
COUNTER NUMBER
);
2) Create the sequence "CREATE SEQUENCE COUNTER_SEQ"
3) Create a JMS Connection factory named "jms/ConnectionFactory"
4) Create a JMS Queue named "jms/TestQueue"
5) Adapt the persitence.xml to point to your database
6) The EAR contains 2 beans.
- A SLSB named "ejb/DataStore"
- A MDB that binds to "jms/TestQueue"
7) Set the log level "com.imtf.atlas.ulysse.sandbox" to FINEST
8) Then run the test "com.imtf.atlas.ulysse.sandbox.test.jms.JmsDeliveryBeforeDb".The test will
call 100 times the SLSB. The test uses the key "test-20" to display messages in the server.log
but you can change it easily. Sometimes the 100 calls are properly processed, however if I run it
a couple of time, I always get a few errors after some time.
My environment is Oracle 10 Enterprise, Glassfish V2UR2 with Hibernate 3.
The SLSB will insert one row in the table Counter and set the counter value to 0. It then sends a JMS message
with the counter primary key. When the message is delivered, the value of the counter is updated to 1. If entity is not
found I get a NPE, because manager.find(...) returns null.
After I have run the test, I get for instance:
1) grep "test-020" server.log | wc -l
100
This shows that the 100 messages were correctly deliverey
2) SELECT counter, COUNT(*) FROM counter GROUP BY counter;
COUNTER|COUNT
1 |98
0 |2
This shows that 2 messages were not properly processed.
3) grep "test-020" * | grep Error
server.log:[#|2009-06-26T15:26:55.268+0200|SEVERE|sun-appserver9.1|com.imtf.atlas.ulysse.sandbox.mdb.DataStoreMDB|_ThreadID=40;_ThreadName=p: thread-pool-1; w: 37;_RequestID=d908eb68-4cc0-4546-976e-ebb72d537cb3;|Error while processing message [batch=test-020,id=2735]|#]
server.log:[#|2009-06-26T15:26:58.624+0200|SEVERE|sun-appserver9.1|com.imtf.atlas.ulysse.sandbox.mdb.DataStoreMDB|_ThreadID=39;_ThreadName=p: thread-pool-1; w: 38;_RequestID=d22420b3-a2a2-4f59-9c3d-7bab242e9975;|Error while processing message [batch=test-020,id=2792]|#]
This shows the two message delivery that failed and the corresponding counter PK.
4) If I check in the sever.log, I see the NPE which indicates that the entity was not yet committed in database at the time the message was processed.
12808 [#|2009-06-26T15:26:58.624+0200|SEVERE|sun-appserver9.1|com.imtf.atlas.ulysse.sandbox.mdb.DataStoreMDB|_ThreadID=39;_ThreadName=p: thread-pool-1; w: 38;_RequestID=d22420b3-a2a2-4f59-9c3d-7bab242e9975;|Error while processing message [batch=test-020,id=2792]|#]
12809
12810 [#|2009-06-26T15:26:58.625+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=39;_ThreadName=p: thread-pool-1; w: 38;_Request ID=d22420b3-a2a2-4f59-9c3d-7bab242e9975;|
12811 java.lang.NullPointerException
12812 at com.imtf.atlas.ulysse.sandbox.mdb.DataStoreMDB.onMessage(DataStoreMDB.java:65)
12813 at sun.reflect.GeneratedMethodAccessor639.invoke(Unknown Source)
12814 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
12815 at java.lang.reflect.Method.invoke(Method.java:585)
12816 at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1067)
12817 at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:176)
12818 at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2895)
12819 at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:3986)
12820 at com.sun.ejb.containers.MessageBeanContainer.deliverMessage(MessageBeanContainer.java:1111)
12821 at com.sun.ejb.containers.MessageBeanListenerImpl.deliverMessage(MessageBeanListenerImpl.java:74)
12822 at com.sun.enterprise.connectors.inflow.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:179)
12823 at $Proxy299.onMessage(Unknown Source)
12824 at com.sun.messaging.jms.ra.OnMessageRunner.run(OnMessageRunner.java:258)
12825 at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:76)
12826 at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
12827 |#]
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 26, 2009 7:21 AM
in response to: ewernli
|
|
|
Could also someone confirm me that using
c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
to send a JMS message from within an SLSB is the right way to go? I've always heard that the flags had no importance if we were in a global transaction....
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 26, 2009 7:35 AM
in response to: ewernli
|
|
|
When I wrote XA JMS code I had to use the XA specific create method(s).
I have not (yet) looked at your sample code but if you are using createQueueSession then AFAIK the JMS queue will not be part of the XA transaction.
|
|
|
|
|
|
|
|
|
Re: Delivery of JMS message in case of distributed transaction
Posted:
Jun 26, 2009 8:00 AM
in response to: ewernli
|
|
|
> Could also someone confirm me that using
>
> c.createQueueSession(false,
> Session.AUTO_ACKNOWLEDGE);
>
> to send a JMS message from within an SLSB is the
> right way to go? I've always heard that the flags had
> no importance if we were in a global transaction....
If you've looked up the connection from Glassfish's JNDI in the standard way, then yes, that's correct. The arguments to createSession() are not used when you're running in JavaEE environment: the session will join the XA transaction (whether contrainer-manager or bean-managed).
Nigel
|
|
|
|
|
java.net RSS
