dmu_tx: break tx assign/wait when pool suspends

This adjusts dmu_tx_assign/dmu_tx_wait to be interruptable if the pool
suspends while they're waiting, rather than just on the initial check
before falling back into a wait.

Since that's not always wanted, add a DMU_TX_SUSPEND flag to ignore
suspend entirely, effectively returning to the previous behaviour.

With that, it shouldn't be possible for anything with a standard
dmu_tx_assign/wait/abort loop to block under failmode=continue.

Also should be a bit tighter than the old behaviour, where a
VERIFY0(dmu_tx_assign(DMU_TX_WAIT)) could technically fail if the pool
is already suspended and failmode=continue.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Paul Dagnelie <paul.dagnelie@klarasystems.com>
Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
Closes #17355

include/sys/dmu.h

@@ -291,13 +291,19 @@ typedef enum {
 	/*
 	 * Assign the tx to the open transaction. If the open transaction is
 	 * full, or the write throttle is active, block until the next
-	 * transaction and try again. If the pool suspends while waiting,
-	 * return an error.
+	 * transaction and try again. If the pool suspends while waiting
+	 * and failmode=continue, return an error.
 	 */
 	DMU_TX_WAIT		= (1 << 0),
 
 	/* If the write throttle would prevent the assignment, ignore it. */
 	DMU_TX_NOTHROTTLE	= (1 << 1),
+
+	/*
+	 * With DMU_TX_WAIT, always block if the pool suspends during
+	 * assignment, regardless of the value of the failmode= property.
+	 */
+	DMU_TX_SUSPEND		= (1 << 2),
 } dmu_tx_flag_t;
 
 void byteswap_uint64_array(void *buf, size_t size);