safe usage within before_update/before_insert events

[Kent]

You've mentioned multiple times (to me and others) that some operations,
such as reaching across relationships or loading relationships from within
a before_update Mapper event is not safe.


- I understand this is safe from within Session event before_flush(),
correct?
- We mentioned at some point adding a Mapper level "before_flush()"
event. To avoid duplicate work, has that been done?


- I presume that other queries, particularly those with
populate_existing() are also unsafe from within before_update? Are such
safe from before_flush()?

[Kent]

Posted before I was done, sorry...


- Is it safe to do all these things from Mapper event "after_update"?
- Is it harmful to invoke session.flush() from within "after_update"?

Post by Kent
You've mentioned multiple times (to me and others) that some operations,
such as reaching across relationships or loading relationships from within
a before_update Mapper event is not safe.
- I understand this is safe from within Session event before_flush(),
correct?
- We mentioned at some point adding a Mapper level "before_flush()"
event. To avoid duplicate work, has that been done?
- I presume that other queries, particularly those with
populate_existing() are also unsafe from within before_update? Are such
safe from before_flush()?

...

[Michael Bayer]

Post by Kent
Posted before I was done, sorry...
Is it safe to do all these things from Mapper event "after_update"?

relationships that are being persisted in that flush are not necessarily done being persisted, so loading a relationship is not a great idea in there, just not sure if results are predictable

Post by Kent
Is it harmful to invoke session.flush() from within "after_update"?

it's not possible because you are already inside the flush.

Post by Kent
You've mentioned multiple times (to me and others) that some operations, such as reaching across relationships or loading relationships from within a before_update Mapper event is not safe.
I understand this is safe from within Session event before_flush(), correct?

session is fully usable, except for flush() of course, inside of before_flush()

Post by Kent
We mentioned at some point adding a Mapper level "before_flush()" event. To avoid duplicate work, has that been done?

this feature would just be a filter function, and the hard part here is coming up with a clean API that works for all cases and isn't terribly non-performant

Post by Kent
I presume that other queries, particularly those with populate_existing() are also unsafe from within before_update? Are such safe from before_flush()?

i dont know that a populate_existing is "unsafe", just consider that when you're in before_update, the session has worked out the full list of every single object its going to "save" or "delete", as well as all the inter-object relationships that are going to be synchronized. It is now working through that list, and halfway through, a query with populate_existing() blows through and resets any number of those changes from having taken place, except some portion of them have already been flushed to the DB, some portion of them have not, and I really can't say what the effect would be. it certainly is extremely difficult to be deterministic, as any change in the structure of your mappings or data can change what the "flush plan" will be, meaning the effects of your flush-plan-invalidating populate_existing() can't really be determined.

Another aspect of flush is that, it updates the attributes of in-memory objects that it knows about. if your query(), even without populate_existing, loads some new collections in, and those collections are then impacted by something like "ON DELETE CASCADE", the flush plan might not to know to mark those objects as deleted, depending on when they got loaded in. In this case as well, I don't offhand have some surprise failure conditions to show you, but the fact that I've been compelled to put all those warnings in the docs is almost certainly because people came to me with "bugs" where they were expecting the session to act completely normally inside of before_update() or after_update(), which it plainly cannot.

As I'm sure I've said before, the flush is where the session is just taking everything that's pending, and flushing it out. you should not have to do any ORM queries in the middle of the flush plan. If you have logic that needs to run through all the objects pending to determine something else that might have to happen, you should probably iterate through those objects yourself, rather than relying upon the convenience of before_update() to provide this iteration for you.

[Kent]

I should step back and present my current issue:

When updates to instances of a certain class (table) are to be issued,
there are sometimes other related updates that need to be made also. But I
need to be able to flush() these changes in between processing the updating
instances. I can't do this inside before_ (or after_) _update (or _flush),
because I need to issue a flush(), but will already be within a flush().

It's almost like I need an event that happens *truly after *an
update/flush; after I'm out of the flush.

I considered recording the fact that I need to make updates once flush() is
finished and then monkey patch the session.flush() function so I can
iterate over the recently updated items after I'm outside the flush(). I
don't like that solution one bit. Do you have any ideas how you might
approach this?

Thanks, as always.

[Michael Bayer]

When updates to instances of a certain class (table) are to be issued, there are sometimes other related updates that need to be made also. But I need to be able to flush() these changes in between processing the updating instances. I can't do this inside before_ (or after_) _update (or _flush), because I need to issue a flush(), but will already be within a flush().

there is a flag I don't like to encourage called "batch=False" for mapper(), which causes the mapper's "save everything of class X" step to run just one object at a time, emitting the before/after_insert/update events fully for that one object, before going onto the next. it is a much more inefficient system, but we need it for the "nested sets" example - since "nested sets" requires that this giant UPDATE across the entire table occur after every row inserted, changing the steps to be taken for the next row. not sure if this is relevant to your situation.

Usually, the job of the flush is, take this whole graph of pending changes, and turn it into SQL. If you need "object Y to change when object X does", the idea is, all those changes can be done pre-flush, or within before_flush(). you try not to consider the changes to objects in terms of SQL, the SQL is something that will happen later when you have the whole thing set up. The *only* time you truly need the flushes to go through in order to determine next steps is when you need to fire off defaults, triggers, or something like that, which then produce new state you need to get at.

It's almost like I need an event that happens truly after an update/flush; after I'm out of the flush.

yeah there's an exceedingly slim series of scenarios where multiple flushes are needed. The scenario of "deleting a row and replacing it with another inserted row, where both rows share a UNIQUE" value comes to mind. I gave someone (was it you?) an example of how to work around that.

I considered recording the fact that I need to make updates once flush() is finished and then monkey patch the session.flush() function so I can iterate over the recently updated items after I'm outside the flush(). I don't like that solution one bit. Do you have any ideas how you might approach this?

I would:

a. establish that there is absolutely no other way to do something other than two flushes, then

b. if it is only *two*flushes, then I'd just do some stuff inside of after_flush_postexec(), and just ignore the second flush. it will happen on its own later

c. no, I really need two flushes, and i really need all the flush things to happen right now without just waiting for it. That's never happened to me, if I really knew of that situation, a new event could be proposed, though I could just as well use a Session subclass (no monkeypatching needed). But I cannot add a new event without a rock solid, definitely no-other-way-to-do-it use case, which would then become the example placed in the documentation for that event as the rationale for when to use it.