How to take part in discussions on other sites? (e.g. the LiveJournal)
Solution to UPDATE...INSERT problem
Hello Guest
  
  • Login
• Register…
• Start blog
  • Who, Where, When
• What can I do?
• What to Read?
  • Polls
• Avatars
• Interests
  • Cities and Countries
• Random blog
• Users search
  • Search
• Games
• Tests
• QAIX
  • Сообщества
• Talxy Chat
• Horoscope
• Online
 
Зарегистрируйся!

QAIX > PostgreSQL database development > Solution to UPDATE...INSERT problem 28 March 2003 08:37:36

  Recent blog posts: 
  They have birthday today: 
  Forums:   
  Discuss: 
  Recent forum topics: 
  Recent forum comments:
  Moderators:

Solution to UPDATE...INSERT problem

Christopher Kings-Lynne 28 March 2003 08:37:36
 Hi Guys,

I just thought I'd share with you guys a very clever solution to the old
'update row. if no rows affected, then insert the row' race condition
problem. A guy at my work came up with it.

We were discussing this earlier on -hackers, but no-one could find a
solution that didn't involve locking the entire table around the
update...insert commands.

The problem is that sometimes the row will be inserted by another process
between your update and insert, causing your insert to fail with a unique
constraint violation.

So, say this is the insert:

INSERT INTO table VALUES (1, 'foo'); // 1 is in the primary key column

Rewrite it like this:

INSERT INTO table SELECT 1, 'foo' EXCEPT SELECT 1, 'foo' FROM table WHERE
pkcol=1;

See? So now that INSERT statement will insert the row if it doesn't exist,
or insert zero rows if it does. You are then guaranteed that your
transaction will not fail and rollback, so you can repeat your update, or do
the insert first and then the update, etc.

Hope that's handy for people,

Chris


-------------------­--------(end of broadcast)---------­------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgres­ql.org/docs/faqs/FAQ­.html

Add comment
Tom Lane 27 March 2003 09:41:29 permanent link ]
 "Christopher Kings-Lynne" <chriskl@familyheal­th.com.au> writes:> INSERT INTO table SELECT 1, 'foo' EXCEPT SELECT 1, 'foo' FROM table WHERE> pkcol=1;
See? So now that INSERT statement will insert the row if it doesn't exist,> or insert zero rows if it does. You are then guaranteed that your> transaction will not fail and rollback, so you can repeat your update, or do> the insert first and then the update, etc.

Uh, why exactly do you think this is race-free?

It looks fancy, but AFAICS the SELECT will return info that is correct
as of its starting timestamp; which is not enough to guarantee that the
INSERT won't conflict with another transaction doing the same thing
concurrently.

regards, tom lane


-------------------­--------(end of broadcast)---------­------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresq­l.org

Add comment
Christopher Kings-Lynne 27 March 2003 09:55:53 permanent link ]
 
Uh, why exactly do you think this is race-free?>
It looks fancy, but AFAICS the SELECT will return info that is correct> as of its starting timestamp; which is not enough to guarantee that the> INSERT won't conflict with another transaction doing the same thing> concurrently.

How about:

INSERT INTO table SELECT 1, 'foo' WHERE NOT EXISTS (SELECT TRUE FROM table
WHERE pkcol=1 FOR UPDATE);

It's a lot more straightforward and has a FOR UPDATE. Can this still cause
unique constraint failures?

Chris


-------------------­--------(end of broadcast)---------­------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHer­e" to majordomo@postgresq­l.org)

Add comment
Tom Lane 27 March 2003 10:13:28 permanent link ]
 "Christopher Kings-Lynne" <chriskl@familyheal­th.com.au> writes:>> Uh, why exactly do you think this is race-free?
How about:
INSERT INTO table SELECT 1, 'foo' WHERE NOT EXISTS (SELECT TRUE FROM table> WHERE pkcol=1 FOR UPDATE);
It's a lot more straightforward and has a FOR UPDATE. Can this still cause> unique constraint failures?

Certainly. FOR UPDATE locks an existing row; it cannot lock the
condition of non-existence of a row.

regards, tom lane


-------------------­--------(end of broadcast)---------­------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresq­l.org

Add comment
Lincoln Yeoh 27 March 2003 12:19:18 permanent link ]
 AFAIK the "except" select won't see other inserts in uncommitted
transactions. If those transactions are committed you will end up with the
same problem. You can try it yourself, by manually doing two separate
transactions in psql.

You either have to lock the whole table, or lock at the application layer.
Some time back I suggested a "lock on arbitrary string" feature for
postgresql for this and various other purposes, but that feature probably
wouldn't scale in terms of management (it requires 100% cooperation amongst
all apps/clients involved).

There's no "select * from table where pkey=x for insert;" which would block
on uncommitted inserts/updates of pkey=x and other selects for insert/update.

In contrast "select ... for update" blocks on committed stuff.

Regards,
Link.

At 09:55 AM 3/27/03 +0800, Christopher Kings-Lynne wrote:
Hi Guys,>
I just thought I'd share with you guys a very clever solution to the old>'update row. if no rows affected, then insert the row' race condition>problem. A guy at my work came up with it.>
We were discussing this earlier on -hackers, but no-one could find a>solution that didn't involve locking the entire table around the>update...insert­ commands.>
The problem is that sometimes the row will be inserted by another process>between your update and insert, causing your insert to fail with a unique>constraint violation.>
So, say this is the insert:>
INSERT INTO table VALUES (1, 'foo'); // 1 is in the primary key column>
Rewrite it like this:>
INSERT INTO table SELECT 1, 'foo' EXCEPT SELECT 1, 'foo' FROM table WHERE>pkcol=1;>
See? So now that INSERT statement will insert the row if it doesn't exist,>or insert zero rows if it does. You are then guaranteed that your>transaction will not fail and rollback, so you can repeat your update, or do>the insert first and then the update, etc.>
Hope that's handy for people,>
Chris>
------------------­---------(end of broadcast)---------­------------------>T­IP 1: subscribe and unsubscribe commands go to majordomo@postgresq­l.org


-------------------­--------(end of broadcast)---------­------------------
TIP 6: Have you searched our list archives?

http://archives.pos­tgresql.org

Add comment
Christopher Kings-Lynne 27 March 2003 12:28:42 permanent link ]
 
AFAIK the "except" select won't see other inserts in uncommitted> transactions. If those transactions are committed you will end up with the> same problem. You can try it yourself, by manually doing two separate> transactions in psql.

Yeah, I see that now.
You either have to lock the whole table, or lock at the application layer.> Some time back I suggested a "lock on arbitrary string" feature for> postgresql for this and various other purposes, but that feature probably> wouldn't scale in terms of management (it requires 100% cooperation
amongst> all apps/clients involved).>
There's no "select * from table where pkey=x for insert;" which would
block> on uncommitted inserts/updates of pkey=x and other selects for
insert/update.

How about user locks? Isn't there something in contrib/ for that??? I
could do a userlock on the primary key, whether it existed or not?

Chris


-------------------­--------(end of broadcast)---------­------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHer­e" to majordomo@postgresq­l.org)

Add comment
Dennis Gearon 27 March 2003 19:55:26 permanent link ]
 so the only real solution to this now is in application code outside of a
transatction, i.e. PHP,Perl,VB,C,Pytho­n, etc, right?

Lincoln Yeoh wrote:> AFAIK the "except" select won't see other inserts in uncommitted > transactions. If those transactions are committed you will end up with > the same problem. You can try it yourself, by manually doing two > separate transactions in psql.>
You either have to lock the whole table, or lock at the application > layer. Some time back I suggested a "lock on arbitrary string" feature > for postgresql for this and various other purposes, but that feature > probably wouldn't scale in terms of management (it requires 100% > cooperation amongst all apps/clients involved).>
There's no "select * from table where pkey=x for insert;" which would > block on uncommitted inserts/updates of pkey=x and other selects for > insert/update.>
In contrast "select ... for update" blocks on committed stuff.>
Regards,> Link.>
At 09:55 AM 3/27/03 +0800, Christopher Kings-Lynne wrote:>
Hi Guys,>>
I just thought I'd share with you guys a very clever solution to the old>> 'update row. if no rows affected, then insert the row' race condition>> problem. A guy at my work came up with it.>>
We were discussing this earlier on -hackers, but no-one could find a>> solution that didn't involve locking the entire table around the>> update...insert commands.>>
The problem is that sometimes the row will be inserted by another process>> between your update and insert, causing your insert to fail with a unique>> constraint violation.>>
So, say this is the insert:>>
INSERT INTO table VALUES (1, 'foo'); // 1 is in the primary key column>>
Rewrite it like this:>>
INSERT INTO table SELECT 1, 'foo' EXCEPT SELECT 1, 'foo' FROM table WHERE>> pkcol=1;>>
See? So now that INSERT statement will insert the row if it doesn't >> exist,>> or insert zero rows if it does. You are then guaranteed that your>> transaction will not fail and rollback, so you can repeat your update, >> or do>> the insert first and then the update, etc.>>
Hope that's handy for people,>>
Chris>>
-------------------­--------(end of broadcast)---------­------------------>>­ TIP 1: subscribe and unsubscribe commands go to majordomo@postgresq­l.org>
-------------------­--------(end of broadcast)---------­------------------> TIP 6: Have you searched our list archives?>


-------------------­--------(end of broadcast)---------­------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgres­ql.org/docs/faqs/FAQ­.html

Add comment
Haroldo Stenger 28 March 2003 00:15:27 permanent link ]
 El jue, 27-03-2003 a las 03:41, Tom Lane escribiГі:> "Christopher Kings-Lynne" <chriskl@familyheal­th.com.au> writes:> > INSERT INTO table SELECT 1, 'foo' EXCEPT SELECT 1, 'foo' FROM table WHERE> > pkcol=1;>
See? So now that INSERT statement will insert the row if it doesn't exist,> > or insert zero rows if it does. You are then guaranteed that your> > transaction will not fail and rollback, so you can repeat your update, or do> > the insert first and then the update, etc.>
Uh, why exactly do you think this is race-free?>
It looks fancy, but AFAICS the SELECT will return info that is correct> as of its starting timestamp; which is not enough to guarantee that the> INSERT won't conflict with another transaction doing the same thing> concurrently.

This approach certainly reduces significantly the time span within which
a race could occur, compared to, say, using two separate statements, or
worse, two statements in two consecutive transactions. But race
conditions either exist or they don't, so you're right.

Now, up to my knowledge this problem was only a an intractable one in
PostgreSQL because of transactions going into abort state once a unique
restriction violation happened. If savepoints/nested transactions were
there, one would simply put the insert within a protected area, and
retry as much as needed.

That's my bet why other databases don't seem to have a problem with this
one. Am I right? Or do they have some magic solution other than locking
the whole table?

Regards
Haroldo


-------------------­--------(end of broadcast)---------­------------------
TIP 4: Don't 'kill -9' the postmaster

Add comment
Lincoln Yeoh 28 March 2003 08:37:36 permanent link ]
 At 05:28 PM 3/27/03 +0800, Christopher Kings-Lynne wrote:> > There's no "select * from table where pkey=x for insert;" which would>block> > on uncommitted inserts/updates of pkey=x and other selects for>insert/update.>­
How about user locks? Isn't there something in contrib/ for that??? I>could do a userlock on the primary key, whether it existed or not?

Depends on your case, whether you can correctly convert your potential
primary keys into integers to be locked on.

It still requires full cooperation by all relevant apps/clients.

Actually select ... for updates also require cooperation, but it's a
standard way of doing things, so apps that don't cooperate can be said to
be broken :)­.

Is there a standard for "select ... for insert"? Or lock table for insert
where pkey=x?

Regards,
Link.


-------------------­--------(end of broadcast)---------­------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresq­l.org so that your
message can get through to the mailing list cleanly

Add comment
 

Add new comment

As:
Login:  Password:  
 
 
  
 
Пожалуйста, относитесь к собеседникам уважительно, не используйте нецензурные слова, не злоупотребляйте заглавными буквами, не публикуйте рекламу и объявления о купле/продаже, а также материалы нарушающие сетевой этикет или УК РФ.


QAIX > PostgreSQL database development > Solution to UPDATE...INSERT problem 28 March 2003 08:37:36

see also:
[JBoss jBPM] - problem with the…
[JBoss Portal] - Portal install…
[JBossWS] - JBossWS RPC-Style SOAP with…
пройди тесты:
see also:
All in one Video Converter Platinum
About
Hello

  Copyright © 2001—2008 QAIX
Idea: Miсhael Monashev
Помощь и задать вопросы можно в сообществе support.qaix.com.
Сообщения об ошибках оставляем в сообществе bugs.qaix.com.
Предложения и комментарии пишем в сообществе suggest.qaix.com.
Информация для родителей.
Write us at:
If you would like to report an abuse of our service, such as a spam message, please .